Who defines ground truth for AI agents?

The quality bar separating environments labs trust from environments labs return is this: separated roles, published calibration thresholds, and models that never approve truth. The pattern emerging across expert-built environments is three roles with hard access boundaries. Authors create cases and private truth. Performers complete work in a clean VM seeing only public assets. Reviewers adjudicate correctness but cannot change a score without a reason code. Models draft, generate variants, and red-team; approval and release promotion stay human. This is where Mercor’s 27% margin lives. Verification is the product.

Key Takeaways

• Trusted environments separate three roles architecturally: authors who create private truth, performers who see only public assets, and reviewers who can’t change a score without a reason code.
• Models draft, generate variants, and red-team. They never approve truth, because the research shows LLM judges carry systematic biases and collapse to near-random exactly where verification matters most.
• Publish the bars or expect skepticism: verifier precision and recall at or above 0.95, hard gates at exactly 1.00, and reward error measured against adjudicated expert scores.

What is the separation-of-duties pattern?

Expert-built environments are converging on three roles with hard access boundaries:

• The author creates cases, private truth, acceptable alternatives, and reviewer guidance. The author cannot see future performer identities or model trajectories.
• The performer completes independent multi-hour work orders inside an instrumented VM, seeing only the public incident, assets, tools, policies, and empty templates. The performer cannot access private fault state, verifier code, calibration fixtures, or other completions.
• The reviewer adjudicates correctness, alternate valid approaches, ambiguity, and every verifier false positive or false negative. The reviewer cannot silently change a score without a reason code.

The separation is not organizational. It’s architectural. The walls between roles are enforced by the environment’s access controls, not by trust. And the reason reviewers adjudicate rather than rubber-stamp is well documented: human annotators genuinely disagree, and treating a single gold label as automatic truth is a fiction the annotation literature gave up on years ago (Plank, 2022). Disagreement gets adjudicated with reasons, not averaged away.

What do models do and not do?

Models and agents draft, generate bounded variants, and red-team the environment. They do not approve truth or promote releases. A model can generate candidate cases, propose fault scenarios, or attempt to exploit the verifier. A model cannot define what correct work looks like, validate the verifier’s calibration, or approve an environment for release.

The research record justifies the boundary. The paper that established LLM-as-a-judge also catalogued its systematic biases: position, verbosity, self-enhancement (Zheng et al., 2023). JudgeBench then showed strong judges scoring near random guessing on objectively checkable hard reasoning pairs, meaning judge reliability collapses exactly where expert verification matters most. And in RLVR training loops specifically, rule-based verifiers produce false negatives while model-based verifiers get hacked into false positives (Huang et al., 2025).

This boundary is the one the field is converging on. Mercor’s enterprise framework defines the verifier as an isolated program returning a score in [0, 1] with a stated basis. The open question is who calibrates the verifiers. The answer is human experts, not models.

What are the bars worth publishing?

Verifier precision and recall at or above 0.95 means the verifier correctly identifies at least 95% of correct and incorrect submissions. Hard gates have no tolerance: a gate that ever false-positives or false-negatives is a broken gate. Mean absolute reward error is measured against adjudicated expert scores.

Measuring the measurer is now its own research program: RewardBench exists because reward models were being deployed with no published accuracy at all. Environments deserve the same treatment. These are not aspirations. They’re the gates an environment must pass before it’s eligible for benchmark release or RL training. An environment that cannot publish its calibration numbers is an environment that has not been calibrated.

What does the week’s news sharpen?

Surge launched the Tuesday Index on August 20, a composite score for AI at work. Qwen 3.8 Max moved 8.6 points in one release on August 21. Composite indices concentrate trust in whoever grades the graders. A single score that moves 8.6 points in one release is a score whose definition matters as much as its value. If the index’s grading criteria are not published, the gain is not interpretable: it could reflect a model improvement, a criteria change, or both.

This is the same problem the environment market faces. A verifier that returns a score without publishing its calibration thresholds is a black box. The environments labs trust publish the bars and meet them. The environments labs return cannot.

What this means

Verification is the product. The 27% gross margin at the category winner is not a labor arbitrage. It’s the price of calibrated ground truth. The quality bar is separation of duties, published thresholds, and human approval of truth. Everything else is a benchmark, not an environment.

FAQ

What is a hard gate?

A hard gate is a zero-weight check that must achieve precision and recall of exactly 1.00. It rejects fabricated evidence, prohibited actions, and private-state access before any reward is scored. There is no tolerance for a gate that blocks valid submissions or passes invalid ones.

Why must models never approve truth?

A model that approves truth creates a self-referential loop: the model defines what correct work looks like, is evaluated against that definition, and scores well by construction. Human experts, authors and reviewers, define truth. Models draft, generate variants, and red-team. The separation prevents the model from grading itself.

What is the Tuesday Index?

The Tuesday Index is a composite score for AI at work, launched by Surge on August 20, 2026. Qwen 3.8 Max moved 8.6 points in one release the following day. Composite indices concentrate trust in whoever defines the scoring criteria, which is the same trust problem the environment market faces with verifier calibration.