Blogs on RL Engineering

Who defines ground truth for AI agents?

Mon, 24 Aug 2026 00:00:00 +0000

The quality bar separating environments labs trust from environments labs return is this: separated roles, published calibration thresholds, and models that never approve truth. The pattern emerging across expert-built environments is three roles with hard access boundaries. Authors create cases and private truth. Performers complete work in a clean VM seeing only public assets. Reviewers adjudicate correctness but cannot change a score without a reason code. Models draft, generate variants, and red-team; approval and release promotion stay human. This is where Mercor’s 27% margin lives. Verification is the product.

Running AI agents in microVMs with zero inbound ports

Mon, 17 Aug 2026 00:00:00 +0000

Long-horizon episodes need real operating systems, real applications, and hard isolation. The emerging pattern is microVMs with zero inbound ports. Firecracker microVMs run Linux sessions; Cloud Hypervisor runs Windows Server for enterprise work that is Windows-native. All traffic flows over outbound tunnels. Gateway URLs are single-session and expiring. Capacity is reserved in a database before any VM exists. Two problems remain unsolved across the industry: per-episode application-database reset for deterministic verifiers, and an action-level GUI event schema for trajectory labeling.

How AI agents cheat their training environments

Mon, 10 Aug 2026 00:00:00 +0000

Under RL pressure, agents game environments in predictable ways. The formal name is reward hacking: optimizing a proxy for the objective in ways that degrade the objective itself (Skalse et al., 2022). Across the environment designs we’ve reviewed, the gaming moves fall into a taxonomy, from format masquerading as competence to quiet memorization of leaked test structure, and the counters are becoming standard. The failures start with reward design: a multi-component weighted rubric leaks reward through prose formatting and keyword compliance. The industry fix is one verifiable outcome bit, backed by integrity gates that run before scoring and carry zero reward weight.

Training AI on office work made it better at coding

Mon, 03 Aug 2026 00:00:00 +0000

Environment training transfers across domains. On August 3, Surge reported that a model post-trained on non-coding office tasks gained 5.8 points on SWE-Bench Pro, a coding benchmark the model was never trained on. The gain is transferable goal-directed execution, not domain knowledge. This changes what labs are buying: environments aren’t domain tutors, they train execution. The result rhymes with AfterQuery’s expert-data uplifts on other benchmarks, and it raises the stakes on contamination controls, the topic for next week.

What Mercor's APEX-Accounting benchmark measures, and what it misses

Fri, 31 Jul 2026 00:00:00 +0000

Mercor and Ramp released APEX-Accounting on July 31: a benchmark for reconciliations, variance analysis, and closing the books. It’s a serious benchmark. Real workflows, expert-authored pass/fail criteria, open release. It’s also a benchmark that cannot carry evidence integrity, which is what audit work is actually graded on. A reference audit case we examined makes the distinction concrete: a 12,840-row inventory subledger totaling $8.305 million, seeded with planted discrepancies whose combined effect exceeds the engagement’s materiality threshold. Accounting tasks were among the 96.5% without definitive benchmark coverage in June. That gap is now closing. The question is which domains remain open.

What a production-grade RL environment spec looks like

Mon, 20 Jul 2026 00:00:00 +0000

Most of the roughly 99 tracked environments publish no verifier calibration and no frozen corpus. The ones labs pay for look different. A reference environment contract we examined runs 572 lines and fixes state machine, tool schemas, reward, curriculum, and splits before any simulator code exists, backed by 79 checksum-frozen case assets, down to a 12,840-row inventory subledger and an oracle script per case. A spec of this kind is not a running system. It’s the contract that must exist before a running system is worth building. The contrast with the market is direct: Surge’s CoreCraft argues for chaotic, large-scale enterprise worlds, but chaos that can’t replay byte-identically can’t calibrate a verifier.

Which jobs can become RL environments?

Fri, 17 Jul 2026 00:00:00 +0000

A screening framework answers the question every vendor and lab faces: can this job be simulated with a verifiable reward, or not? The framework evaluates candidate occupations across ten dimensions, from computer-work share to output verifiability, with one hard rule: a critical physical dependency vetoes construction outright, no score adjustment. A job that requires hands-on equipment operation, physical inventory observation, or in-person client interaction can’t become a sandbox environment, no matter how well it scores everywhere else. The framework explains the gap between the 99 catalogued environments and the 66 tracked occupations that have none.

Why Mercor bought Deeptune

Fri, 10 Jul 2026 00:00:00 +0000

Mercor’s Deeptune acquisition says the constraint has shifted from expert networks to the environments themselves. On July 9, Mercor announced the acquisition of Deeptune, a vendor that had raised $43M from a16z in March 2026 and spent two years recreating hundreds of enterprise applications, from spreadsheets to Salesforce. Mercor said it outright: the bottleneck is no longer the experts, it’s the software layer they work in. Our vendor data confirms the whitespace the deal targets. 0 of 66 tracked occupations have a corresponding RL environment. The market is crowded where it is easy and empty where it matters.

3,029 AI benchmarks, 99 RL environments: the field is over-indexed on evaluation

Fri, 03 Jul 2026 00:00:00 +0000

The benchmark pile keeps growing while catalogued RL environments stay rare and static. Our catalogue counts 3,029 unique benchmarks and 24,810 model score rows, against just 99 catalogued RL environments. That’s a 31-to-1 ratio. The leverage sits not in adding more benchmarks but in expert verification: this week, Surge trained a 4B model on 1,000 expert-written rubrics to parity with a model 60 times larger, and AfterQuery helped NVIDIA hill-climb GDPval. The pile of benchmarks grows. Expert verification moves the numbers.

AI benchmarks cover only 3.5% of real work

Fri, 26 Jun 2026 00:00:00 +0000

Our study joining 202 O*NET occupational tasks against 223 candidate benchmarks with an LLM judge yields definitive coverage of 3.5%. Only 7 of 202 tasks are definitively covered by an existing benchmark. The study cost $13.79 across 441 judge calls. A robustness check, a recall audit of 4,935 task-benchmark pairs against every unlinked benchmark, found zero missed matches for $8.82, bringing the combined cost to $22.61. Benchmark success materially overstates workflow competence: existing benchmarks test isolated calculations or document operations, not the multi-step occupational workflows they claim to represent.

What AI expert job postings reveal about RL environment demand

Mon, 15 Jun 2026 00:00:00 +0000

Expert job listings on Mercor, AfterQuery, and micro1 are a leading indicator of where labs are investing in environment development. We track active expert postings across all three platforms daily, deduplicated so cross-posted roles count once. The pattern is consistent: labs recruit domain experts months before the environments those experts build reach a training run. Demand concentrates in a few domains (coding, finance, legal, medical) while the vendor census shows 21 of 38 tracked companies sell coding environments. Where recruiting volume diverges from what vendors currently sell is where the next environment categories will appear.

The RL environment industry is 38 companies, mostly under 50 people

Tue, 09 Jun 2026 00:00:00 +0000

Our census of 38 tracked RL environment vendors finds 31 with 50 or fewer employees. The companies supplying the most capitalized labs on earth are a cottage industry. Scale AI and Turing are the only firms above 200 staff. Everyone else, including the vendors whose benchmarks appear in frontier system cards, runs on a small team. And small doesn’t mean marginal: this same week, AfterQuery, a 51–200-person YC W25 lab, published a +21.4% net win-loss gain on GDPval from on-policy distillation. Small shops move frontier-model numbers.

The economics of selling RL environments to AI labs

Sun, 07 Jun 2026 00:00:00 +0000

Frontier labs are shifting spend from labeled data to executable environments, and the vendor margins tell you where the leverage sits. Mercor runs roughly $2B in annualized gross payment volume at a 27% gross margin and is valued at $10B after a $350M Series C. AfterQuery closed a $30M Series A at a $300M valuation and says it has since passed a $100M revenue run rate. Neither margin profile looks anything like software. The category winner sells expert labor organized around verification, and the labs buying it now cite vendor evals in their own system cards. The money is moving from annotation budgets to environment budgets, and it’s flowing to the companies that can grade what a model does, not just label what it sees.