1,700 RL tasks lifted Kimi K2.7 on every coding benchmark tested

Broad expert task portfolios, not benchmark targeting, are now the reliable RL training recipe. On September 11, 2026, Surge AI, the expert data vendor, published a study in which 1,700 agentic coding tasks lifted Kimi K2.7, Moonshot AI’s open-weight model, on all five external coding benchmarks tested, by 12.5 percentage points on average, with gains from +4.7 to +20.0 depending on the benchmark. The training was reinforcement learning only, with no supervised warmup (no phase of copying example solutions first), on 1,000 repository-style fixes and 700 terminal jobs. The tasks came from Surge AI’s existing catalogue, were not selected to target any of the five benchmarks, and were collected before three of the five existed. That independence is the finding, and it matches the 397B (397 billion parameter) result Mercor published on September 1 and the office-work transfer in August: buy a broad portfolio of expert tasks with calibrated grading, train against it, and the gains show up on evaluations nobody optimized for.

Key Takeaways

• RL-only post-training on 1,700 catalogue tasks improved all five external coding benchmarks tested.
• The tasks were not selected to target any of the five benchmarks and predate three of them, which is what makes the transfer meaningful rather than contamination.
• The trained agent also got faster: median solution trajectories (the sequence of steps an agent takes on one task) shortened by a quarter to a third on two suites.

What did Surge train, and on which tasks?

Kimi K2.7, Moonshot AI’s open-weight model, was post-trained with reinforcement learning alone: eight attempts per task per training step, scored by graders, no supervised examples. The 1,700 tasks split into 1,000 repository-style tasks (fix or extend real code until hidden tests pass) and 700 terminal-style tasks (finish a job in a command-line shell). Both came from Surge AI’s standing dataset catalogue rather than from any benchmark’s distribution.

“1,700 agentic coding tasks improved Kimi K2.7 on all five coding benchmarks”

Did the gains transfer, or did one number move?

All five moved up, and none of them were in the training set’s distribution. The five suites were:

• Terminal-Bench 2.1
• Terminal-Bench 3
• SWE-Marathon
• DeepSWE
• SWE-Bench Pro

Breadth matters because a single benchmark rising can mean leakage, luck, or targeting, while five rising together, on suites built by different teams in different formats, means the capability transferred. The study’s cleanest control is temporal: the task catalogue predates three of the five benchmarks, so for those three the training data could not have been shaped around them even by accident. For the other two, Surge states the tasks were not selected to improve them.

Why does faster matter as much as stronger?

The trained agent passed more tasks and took fewer steps to pass them. Median trajectory length fell from 150 steps to 98 on one evaluation suite and from 102 to 78 on another. Shorter trajectories cost less to serve and leave fewer steps in which an agent can fail. They also suggest the model learned the work rather than learning to search longer.

The reward design explains some of this. Each task scored as a hard gate multiplied by a dense signal: the gate requires that every previously passing test still passes, and the dense part pays out the fraction of target checks completed. That structure (no credit if anything breaks, partial credit for progress) is the same grader design benchmark maintainers are converging on, because it rewards finishing work instead of gaming the check.

What this means

Two vendors have now published this recipe three times in six weeks with matching results: broad expert task portfolios plus calibrated grading produce transferable gains, in Surge’s August office-work run, in Mercor’s 35 billion and 397 billion parameter runs, and now on an open-weight agent model. For buyers, the diligence question is shifting from whether task-based RL works to whether a vendor’s catalogue predates and stays independent of the benchmarks it will be judged on. The result is Surge AI’s own measurement of its own catalogue; rl.engineering research found no independent replication as of September 12, 2026.

FAQ

What does hill-climbing mean in RL post-training?

Hill-climbing is iteratively improving a model against a fixed set of tasks: attempt, score, update, repeat. Surge’s Kimi K2.7 study climbed on its own catalogue and measured the result on five external benchmarks it never trained against.

Why does it matter that the tasks predate the benchmarks?

The date matters because the strongest objection to any benchmark gain is contamination, training data that resembles the test. A catalogue collected before three of the five benchmarks existed cannot have been sampled from those three, which makes improvement on them evidence of capability rather than memorization.

What is a pass-to-pass test?

A pass-to-pass test is a test that passed before the agent’s change and must still pass after it. Scoring them as an all-or-nothing gate means an agent cannot earn reward by fixing one thing while breaking another.