An Anthropic researcher resigned over self-improving AI. Here is the loop he means

On September 8, a researcher named Jacob Coxon quit Anthropic and wrote that his employer and OpenAI, where he says he spent a combined three years, are “racing straight to self-improving superintelligence and gambling with our lives.” The post drew tens of millions of views inside a day, and wire services picked it up within hours. This post explains the machine behind his warning in plain terms. Today’s models improve through a simple loop: give the model real work with a checkable answer, score the attempt, nudge the model toward what scored well, repeat millions of times. Self-improvement means the model starts running parts of its own loop. That loop is a supply chain we track, and the race is visible in our data: after a September 9 refresh that changed 172 model scores, just 0.25 points separate the top two models, 84.36 against 84.11.

Key Takeaways

• The warning is about a loop, not a moment: models train on scored tasks, and stronger models increasingly help build the next round of tasks and scoring tools.
• The loop already works at the largest scale. A vendor published results this month showing a 397B-parameter model improving 70 percent relative on expert tasks through this kind of training.
• One stage still needs people: deciding what counts as a correct answer. That stage is the bottleneck, and it is also the control surface.

“racing straight to self-improving superintelligence and gambling with our lives”

What is the loop, in plain terms?

Think of training an AI agent like coaching with an exam that grades itself:

1. Set the exam. Build an environment: a realistic task with a hidden correct answer, like a bug to fix or a ledger to reconcile.
2. Let the model try. The model attempts the task, sometimes for hours, using tools like a real worker would.
3. Score the attempt. A verifier, which is a program that checks the work, returns a pass or a score.
4. Update the model. The model is nudged toward whatever scored well. Go to step 1.

Researchers call this reinforcement learning with verifiable rewards, documented as a recipe by Lambert et al. (2024). The “self-improving” part arrives when the model’s own output feeds the loop: stronger models draft new exam tasks, probe the graders for loopholes, and grade work where no simple checker exists. Every one of those substitutions removes a person and speeds the loop up. That is the design goal, not a side effect. The fear Coxon describes, per coverage of his full thread, is that the speedup compounds faster than anyone can audit what the graders are rewarding.

Is the loop actually working?

Yes, and in public. On September 1, Mercor published a training guide showing a 397B-parameter model trained with this loop on 1,928 expert-written work tasks. Its first-try success rate on a held-out exam rose 70 percent relative. No internet scraping involved: the capability came from purchased, expert-built tasks. There is now a whole industry selling that fuel, with real margins.

The race Coxon describes also shows up as telemetry. The two leading models are now separated by less than the movement of a single routine refresh, which our leaderboard-stability analysis measured in late August, and that means no lab can pause and keep its lead. That is his stated explanation for why his employer kept racing: stop, and someone less careful pulls ahead.

Where do humans still sit in the loop?

At one stage: deciding what counts as correct. Our research on who defines ground truth found that in environments labs trust, models may draft tasks and stress-test graders, but people write the answer key and check the graders against expert judgment. The checking has no slack in it, because agents exploit any grader with a loophole. And unlike compute, expert grading does not get cheaper as models get stronger. It gets harder, because the loopholes get subtler.

That is why the resignation’s policy asks, coordination between US labs and possibly a temporary halt on capability gains, point at the same place our data does. The answer key is where oversight is physically possible. Whoever calibrates the graders decides what the loop optimizes. Right now that is a market function, performed by vendors, not a regulated one.

What this means

Self-improvement is not a secret lab project. It is a supply chain with public prices, public results, and one stage that still requires human experts. Read the resignation as testimony that demand for that human stage is structural, and as a map: if you want to watch or govern the loop, watch the grading.

FAQ

Who is Jacob Coxon and what exactly did he claim?

A researcher who describes three years of pretraining work across OpenAI and Anthropic. In his September 8 resignation post he argued both labs are racing toward self-improving systems without adequate safeguards, and that many insiders privately consider the risk existential. Those claims about internal beliefs are his; neither company had publicly responded as of September 9.

What is an RL environment?

A packaged, repeatable work task with a hidden correct answer and a grader. It is the “exam” in the loop: the model practices on it, the grader scores it, and the score trains the model. Building good ones is expert labor, which is why a vendor market exists.

Do models train themselves today?

Not end to end. Models already draft tasks, generate variants, and hunt for grader loopholes, but in trusted setups a person still approves what counts as correct and signs off on releases. The trend is toward automating more stages, which is why the one human gate carries so much weight.