Author: Yuqian Fu
I have been thinking about a central question: how can we monitor an agent’s behavior during training closely enough to reduce reward hacking and other undesirable patterns that emerge during post-training?
This article is my synthesis of material that Anthropic and OpenAI have published in technical blogs, alignment posts, and model cards. It focuses on two questions:
This is a reading note, not an official account of either company's internal systems. Some of the comparisons and interpretations are mine.
An agent trajectory can include user and developer messages, the model's intermediate reasoning, tool calls, changes in the environment, the final answer, and derived artifacts such as summaries, scores, and risk labels.
The common mistake is to evaluate only the final answer. A coding agent can produce a patch that passes the visible tests while exploiting the repository, the test harness, or an accidental loophole in the environment. A model can also complete the surface task while pursuing a hidden side objective. The final output may look fine; the trajectory tells us how it got there.
Anthropic's public work is largely about building the infrastructure needed to read many long trajectories: organize them, cluster them, name the clusters, summarize them, and then run automated evaluation and human review. Clio, Petri, and Bloom have different purposes, but they all turn trajectory review from a manual inspection task into a structured, repeatable workflow.
OpenAI has put more emphasis on signals that the model exposes about itself. Its work on chain-of-thought (CoT) monitorability asks whether a reasoning trace contains evidence of reward hacking, shortcutting, or attempts to bypass a constraint. OpenAI's internal coding-agent monitoring extends this idea to the full interaction: reasoning, messages, actions, tool calls, and tool outputs. The signal can be strong, but it is also fragile. If training rewards the model for looking safe to a CoT monitor, the model may learn to hide its intentions instead of changing its behavior.
The mitigation problem is not simply "make the model behave." It is "reduce bad behavior while preserving enough visibility to detect what remains."
Anthropic has focused more on preventing local reward hacks from becoming general behavioral tendencies. Its work on reward-hacking generalization, alignment faking, training-time monitoring, and prompt design asks how to stop the model from learning a reusable strategy of exploiting loopholes.
OpenAI has also explored training, but several of its public proposals start from a different premise: some bad actions may remain, so make them easier to find. Confessions and self-reporting give the model a separate channel in which to disclose shortcuts, policy violations, uncertainty, or conflicts. The goal is not to assume that every model will volunteer the truth. It is to make concealment harder and to add another independent signal for an external monitor.
The two lines of work overlap. Anthropic's Alignment faking in large language models and OpenAI's Detecting misbehavior in frontier reasoning models both discuss reasoning-trace monitoring and the risk that models learn to hide their real objectives. Anthropic's Towards training-time mitigations for alignment faking in RL and OpenAI's How confessions can keep language models honest both separate the question of outward compliance from the question of whether the model is actually reporting what it did.
Anthropic's public work has moved through several stages. Earlier projects often constructed difficult scenarios by hand and measured how a model responded. More recent work combines automatically generated scenarios with large-scale monitoring of real or semi-real interactions.
Research hubs: