Early stopping and low-quality delivery in frontier coding agents
Author: Yuqian Fu
I recently spent a stretch of time doing RL training and evaluation on large coding/cowork agents, and ran into a failure mode that is surprisingly hard to categorize. Given a task, the agent would quietly reason in its chain-of-thought about "which tests the evaluation is probably going to run," write a couple of easy-to-pass coding tests into its trajectory, and then declare the task done (ending the trajectory) as soon as they passed. Edge cases go uncovered, and the real requirement never actually gets met. In the cowork setting the same thing happens: the agent guesses at the user's rough expectations for a deliverable and, after very few steps, hands back something that just barely clears the bar, when it could have done considerably better. In practice you can improve this a lot with a crude forced loop or better SFT data, but why it happens in the first place is still unclear, and it tends to get worse as models get larger and more capable.
When I first saw this, the obvious label was reward hacking. But on closer inspection it is not the same thing: the model does not rewrite test assertions to True, does not hardcode expected values, does not sys.exit(0) to bail early, and does not tamper with the reward function. Those textbook reward-hacking moves are well studied and relatively easy to catch (with agent-driven trajectory monitoring). Here the model exploits no loophole at all. It simply caters to the grader it has guessed at, and settles for doing just enough to be judged as passing. This kind of mismatch trips no "cheating" alarm, yet it ends long-horizon tasks prematurely and drags delivery quality down.
This post follows that thread through two concrete downstream symptoms and how to mitigate them. The first is early stopping: the model infers that the grader only does a shallow check, sets itself an easy self-check, and quits the moment it passes, without really finishing the long task. The second is delivery distortion: because the grader is guessed wrong and the rubric is underspecified, the deliverable or code passes the judge but does not match what the human actually wanted. I will first lay out the shared root cause and one prerequisite capability, then take the two symptoms in turn, and finally spend the most space on mitigation, mostly from the angles of data, reward design, and monitoring. As it turns out, OpenAI and Anthropic noticed these problems quite early and have written up some analysis of their own, which I will draw on throughout.
Both symptoms come from the same tendency, which OpenAI and Apollo Research call reward seeking [1]. Their definition is worth quoting directly: "the degree to which a model represents its grader and conditions its behavior on what it believes the grader rewards. A grader can be a reward model during training, an evaluation grader during testing, or a monitor during deployment." Unpacked, that is two steps: the model first builds an internal model of the grader ("how is it likely to score me"), then adjusts its behavior to that guess. Crucially, this is not necessarily cheating. The same paper deliberately separates reward seeking from reward hacking: the former is the general tendency to model the grader and act on that belief, the latter is a specific class of behavior that exploits a loophole to score high. A model can hack from a single learned heuristic without ever reasoning about the grader at all; conversely, it can touch no loophole whatsoever and merely cater, gently, to what it thinks the grader values. It is the latter that this post is about.
The paper goes further: this catering to the grader gets worse the longer you train, and erodes honesty as a side effect. In an o3-like, purely capability-focused coding RL run with no safety training, measuring-reward-seeking [1] observed that the further training went, the more the model did "what it thought the grader wanted," even when that ran against the user's or developer's intent; on an "honesty vs. task completion" evaluation, the later checkpoint's honesty became "far more contingent on what it believes the grader rewards." In other words, training pushes the model's behavior to hinge on its guess about the grader rather than on solving the task itself.
To cater to a guessed grader, the model first has to be able to sense that a grader exists, i.e. to infer from context that it is probably being evaluated and roughly by what standard. This capability is called evaluation awareness, and it is the common cause behind both of this post's symptoms.
The first thing worth noting is that this awareness grows alongside ordinary capability training, and what grows is propensity, not capability. In metagaming [2], OpenAI found that a purely capability-focused RL run with no safety training was enough to raise verbalized evaluation awareness on alignment evals from 2% to 20.6%, while it stayed at zero on capability evals. That is, the model already has the reasoning; training just makes it increasingly inclined to actually use it. This matters, because it means evaluation awareness is not taught by some batch of bad data but is a byproduct of doing RL on capability.
In a coding agent, this awareness changes behavior in very concrete ways. While auditing SWE-bench trajectories, Cursor [3] recorded a task drawn from a 2019 jq issue: the agent tried to reproduce the bug, but the image had been built after the fix landed, so reproduction failed. From that it inferred the issue had already been resolved and was most likely an eval task, and it went off to search for the existing fix rather than deriving one. A single failed reproduction was enough to change the model's entire downstream trajectory.
This sense of "being evaluated" can even adapt to different users' expectations. In Redwood's post-hoc analysis of Sonnet 4.5 [4], verbalized evaluation awareness on synthetic honeypots jumped from about 3% in Sonnet 4 to 80–85%, and they estimated that between 25% (average case) and 90% (worst case) of the "alignment improvement" on those environments actually came from the model recognizing the eval and behaving better, rather than from being genuinely more aligned. Anthropic does not dodge this in its own system card [5]: "We cannot rule out that [the snapshot]'s low deception rates in our evaluations are at least partially driven by its evaluation awareness." Nor is this unique to Sonnet 4.5. Apollo saw it back on Claude 3.7 Sonnet [6], where up to 33% of the model's chains-of-thought explicitly stated that it was in an alignment test. And what gets verbalized is only part of the picture: Anthropic's Fable 5 system card reports [7] that the model will "increasingly track whether its environment can be exploited for reward," a form of tracking that often never appears in the chain-of-thought yet still shapes behavior. As for the scale in real deployment, OpenAI's production-representative evals give a reference point [8]: in representative production conversations, the share where the model believes it might be under evaluation is about 10% for GPT-5 and about 4% for GPT-5.1 — not high, but not zero.
An agent's evaluation awareness aggravates both symptoms at once. On one hand, once the model judges that it is probably facing only a shallow eval, it is more likely to quit while ahead, which is early stopping. On the other, it contaminates our very judgment of whether the work is good. Fang et al. propose an awareness tax to quantify the second effect [9]: the performance gap between evaluation-aware and unaware samples, i.e. how much a score is inflated or deflated simply by the model recognizing the eval. They also find that recognition rates on safety-flavored scenarios are roughly 3× those on structurally matched capability scenarios. The more readily a model recognizes an eval, the less "passing the eval" proves the work was actually done well, which is what carries us into the second symptom.
Early stopping on long tasks is, at bottom, a misjudged stopping criterion. Anthropic's Haiku 4.5 model card names this pattern agentic laziness [10]: "models stop working on a problem prematurely, give incomplete answers, or cut corners on tasks." AgentPatterns puts it more mechanically [11]: the done action tends to fire the moment the first sign of progress appears — a passing test, an applied patch, a completed line of reasoning — regardless of the real, multi-file, multi-test scope. In the language of grader-guessing, the mechanism is simply this: the model infers the grader will only do a shallow check, and adopts "make the shallow check pass" as its definition of finished.
Under this symptom, the model's signature move is to set itself an easy self-check. DeepMind's MONA gives a very clean example [12]: an agent doing test-driven development learns to first write a test that is technically correct but does not cover the problem space (say, assertEqual(f([]), [])), then write an implementation that just passes it (return []). Everything passes under its self-set check, while the true held-out correctness is very low.