Multi-instance and multi-pass review

Why self-review can repeat the original mistake

A model reviewing code in the same session sees the context and assumptions used to create it. It may therefore repeat the same mistaken assumption instead of questioning it.

Two things that do not fix this:

• Instructing it to review its own work critically. The instruction doesn't remove the context.
• Extended thinking. More reasoning from the same starting assumptions is more of the same reasoning.

Extended thinking refers to a manually set thinking budget. Current models use adaptive thinking instead. In either case, additional reasoning in the same session still begins with the assumptions that produced the original work.

Generating two implementations in one session doesn't create an independent review. Both versions begin with the same session context and may repeat the same assumption. Use a separate instance that didn't see the original reasoning.

An independent instance, without the generator's context, approaches the diff without that commitment and catches subtle issues the author's session won't.

You can't proofread your own essay the day you wrote it. You read the sentence you meant, and memory supplies the word missing from the page. A roommate reading it cold, assignment sheet in hand, has nothing to supply it from.

What the independent reviewer should receive

Independence is about withholding the reasoning, not the requirements.

Give the second instance the change and the standard it is judged against: the specification, conventions, and acceptance criteria. Do not provide the generating session's transcript or its summary of assumptions. That summary may omit the exact assumption the first session never questioned.

A reviewer without the specification can check whether the code is internally consistent, but can't confirm that it meets the intended requirements.

Split large reviews into multiple passes

Large multi-file reviews suffer attention dilution: one pass carrying every file and every concern produces shallow, sometimes contradictory findings.

All the files may fit in the context window, but reviewing many files and concerns at once can reduce the attention given to each one. A larger context window lets more material fit; it doesn't guarantee a detailed review of every file.

Split it:

• Per-file local passes for issues contained within a file.
• A separate cross-file integration pass for data flow between files.

Splitting only by concern — one pass for security and another for style — still requires each pass to review all thirty files. Per-file passes reduce the amount of code each review must examine.

The integration pass is not optional. Per-file analysis structurally cannot see an inconsistency that exists only between files, no matter how careful each pass is.

Confidence for routing, not for filtering

A verification pass can have the model report confidence alongside each finding, but raw self-report is not calibrated. Compare it with labeled outcomes, set thresholds from that evidence, and continuously sample findings that take the automatic path. Route with confidence alongside ambiguity, severity, and business impact rather than treating one score as authority.

Keep those signals distinct. Severity says how much an issue would matter if it is real; confidence says how likely it is to be real. A severe finding the reviewer is unsure of and a trivial one it is certain of are different cases, and one number can't express both.

Two axes means two fields. These names are this app's own convention, not an API shape:

{
  "file": "src/billing/refund-service.ts",
  "finding": "ledger write happens before the gateway call is confirmed",
  "severity": "high",
  "confidence": 0.42
}

A calibrated policy outside the model can now route this one to a person on severity while sending a low-severity, high-confidence finding straight through. Collapse the two into a single number and that distinction is gone before the policy ever sees it.

Note the difference from 4.1: asking the model to filter by confidence doesn't improve precision. Reporting confidence as a field so an external, calibrated policy can use it is a different and useful thing.