AI benchmarks cover only 3.5% of real work

Our study joining 202 O*NET occupational tasks against 223 candidate benchmarks with an LLM judge yields definitive coverage of 3.5%. Only 7 of 202 tasks are definitively covered by an existing benchmark. The study cost $13.79 across 441 judge calls. A robustness check, a recall audit of 4,935 task-benchmark pairs against every unlinked benchmark, found zero missed matches for $8.82, bringing the combined cost to $22.61. Benchmark success materially overstates workflow competence: existing benchmarks test isolated calculations or document operations, not the multi-step occupational workflows they claim to represent.

Key Takeaways

• Joining 202 O*NET occupational tasks against 223 benchmarks with an LLM judge found only 7 tasks (3.5%) definitively covered; the other 96.5% is the addressable market for environment builders.
• A recall audit of 4,935 task-benchmark pairs found zero missed matches, so the gap is real, not a search failure. Both studies together cost $22.61.
• Vendors confirmed the gap the same week: frontier agents scored under 25% on Mercor’s APEX-Agents and no model beat 36.2% on Surge’s HANDBOOK.md.

How was the coverage study conducted?

The study evaluated 7 occupations and 202 tasks from O*NET 30.3 against 223 unique candidate benchmarks across 255 occupation links, using 441 judge calls with every task receiving its complete candidate pool. The judge consumed 3.67M input tokens and 646K output tokens.

The approach has a research lineage. OpenAI’s GPTs are GPTs established task-level O*NET analysis as the way to measure what LLMs can touch, using human and model raters over the same taxonomy. We ran the same task-level logic against benchmarks instead of wage exposure: for each occupational task, does any benchmark actually require performing it?

The dominant result is partial coverage. Existing benchmarks often test an isolated calculation, a question-answering skill, a document operation, or a simplified decision rather than the complete occupational workflow. Of the 643 covering or partial verdicts retained, 564 used fetched task-level evidence rather than catalog descriptions alone, and 496 contained a quotation verified against the fetched source.

What does the recall audit prove?

A bounded recall audit tested the three highest-value not-yet-covered tasks in each occupation against 235 reputable agentic, multimodal, robotics, and coding benchmarks that no occupation links to. The audit covered 21 tasks and 4,935 task-benchmark pairs in 252 judge calls, costing $8.82.

The result: zero missed covering benchmarks, a 0% task-level recall-failure rate, and 46 partial matches. The partial matches reinforce the central finding. Broad agentic benchmarks exercise a component of the work but do not require the complete workflow:

• Finance analysis without examining facilities and producing a client recommendation.
• Legal research without gathering case evidence or negotiating a settlement.
• Scientific experimentation without planning the complete validation program.

The measurement literature has been circling this for years. An Oxford-led review of 445 LLM benchmarks found pervasive construct-validity failures, benchmarks that don’t measure what they claim to (Bean et al., 2025), and Raji et al. (2021) warned against reading any narrow benchmark as general capability. Even our covered 3.5% deserves a skeptical read: audits of popular agentic benchmarks show task-setup and reward flaws can distort measured performance by up to 100% relative (Zhu et al., 2025).

What did vendors report the same week?

Two vendor results corroborate the coverage gap from different directions. Mercor’s APEX-Agents benchmark reports frontier agents under 25% completion on professional tasks, and only 40% even with 8 tries. The benchmark uses 480 tasks across 33 simulated worlds covering investment banking, law, consulting, medicine, software engineering, and accounting (the setup is documented in the APEX-Agents paper).

Surge launched HANDBOOK.md on June 25: 65 tasks governed by 20-to-124-page standard operating procedures across finance, medical billing, insurance, logistics, and HR, graded by 824 fully deterministic criteria checking required and prohibited actions. No frontier model scores above 36.2%. Failure modes include firing employees without authorization and violating policies the SOPs explicitly prohibit.

Both vendor benchmarks confirm what the coverage study found from the task side. Existing benchmarks test pieces of work, not the complete workflow. The gap is not a measurement problem. It is a coverage problem.

What this means

The 96.5% of tasks without definitive benchmark coverage is the addressable market for environment builders. Every task in that gap is a workflow no existing benchmark tests end-to-end, and every such workflow is a candidate for a production-grade RL environment with a verifiable reward.

FAQ

What counts as “definitively covered”?

A task is definitively covered when a benchmark requires performing the task’s core action, not an isolated calculation, a simplified decision, or a document operation that is one step in the workflow. The judge applied a strict rubric: shared domain is not coverage.

Why does partial coverage not count?

Partial coverage means a benchmark tests a component of the task without requiring the complete multi-step workflow. A model that passes a partial benchmark has demonstrated competence at one step, not at the occupation-level task the benchmark claims to represent.

What is the combined cost of both studies?

The coverage study cost $13.79 and the recall audit cost $8.82, for a combined $22.61. Publishing study costs alongside findings is part of the methodology: a claim about benchmark coverage should state what it cost to establish.