Be precise about what shipped
I defined the direction and the evaluation framework. I am not going to write this up as “we hit 40% fewer defects”. Those are goals on the contract, not a victory lap. If you cannot tell the difference, you should not own an eval product.
The adjacent work that did ship in the same program: a 750-finding, 10-stack root-cause analysis that took open P1/P2 code-quality production issues from 250 to 10 with zero critical SLA breaches. Eight SDLC governance controls went from 32 consecutive failing weeks (195 GRC issues, 92% at worst) to a program-record 9 clean weeks. That is the operating environment. AI Peer Review is how you stop paying for that quality in human hours forever.
The actual product problem
LLM code review fails in a specific, boring way. The model is fluent. Fluency reads as authority. It leaves a comment that is locally plausible and globally wrong: a style nit dressed as a defect, a hallucinated API, a security finding that does not apply to this code path. The first week, engineers engage. The third week, they mute it. You have built a notification product.
So the PM job is not “more findings”. It is precision as a product requirement. Recall is the executive’s number. Precision is the reviewer’s number. If you optimize the executive’s number, the reviewer leaves and you get neither.
A 5% false-positive target is not a model metric I copied from a paper. It is the point at which I believe a competent engineer will still open the panel. Above that, you are training the org to ignore machines.
The eval contract
I treat the reviewer as a classifier sitting in a human workflow, not as a chatbot with extra steps.
| Term | Meaning in this product |
|---|---|
| False positive | A comment that a competent reviewer would not have left or would have dismissed as not actionable. Includes nits inflated to “must fix” and findings that are true in some codebase but not this one. |
| False negative | A defect class we claimed to cover that shipped anyway. Counted against the 40% defect goal, not excused because “the human should have caught it”. If the tool is in the path, it owns a share of the miss. |
| Cycle time | Time from PR open to merge, not time-to-first-comment. A faster first comment that starts an argument is not a win. |
| Severity honesty | Blocker vs nit must be calibrated. A model that calls everything High is a false-positive factory with extra steps. |
What I will not do
- Ship unscoped “review the whole diff” as v1. Scope by defect class (null safety, authz, injection, secrets, obvious dead code) so the eval set can exist.
- Use another LLM as the only grader of the first LLM. You need a human-labeled set for the classes you claim or you are grading vibes with vibes.
- Report “comments per PR” as a success metric. That metric is how you get spam.
- Put the model on a blocking gate while FP is unknown. Advisory until the 5% is measured, not hoped.
Failure modes I designed against
- Sycophantic severity. The model agrees with the author’s comments or the ticket title instead of the diff.
- Context truncation. Findings that are only true if you did not see the other file. If the window cannot hold the relevant module, the honest product behavior is silence, not a guess.
- Style-as-security. Naming nits and formatter noise presented as risk. This is the fastest way to burn the 5% budget.
- Non-reproducing findings. A comment that cannot be turned into a failing test or a concrete line-level fix is not a finding. It is a thought.
- Drift after a model swap. If we change the underlying model, the eval set runs before the reviewer is turned back on. No silent upgrades in a path humans trust.
Why this sits next to the P1/P2 work
The 250 → 10 reduction did not come from a chatbot. It came from a 750-finding RCA across ten stacks and then governance that actually closed. That is the unglamorous half of AI product work in an enterprise: you do not get to sprinkle a model on a process you have not understood.
AI Peer Review is the attempt to encode some of that understanding so the org is not dependent on the next heroic RCA. The eval framework is how we will know whether the encoding is real. Until the 5% is measured in review, it is a specification, not a miracle.
What I would do next
- A public-quality (internal) eval pack: 100 diffs, labeled by class, with “no comment” as a valid gold answer. Most review evals forget that silence is often correct.
- Split FP by severity. A false Low is cheap. A false Blocker is how you stop the line.
- Trace-first debugging: every comment carries the evidence span. No evidence, no comment. That single rule will do more for the 5% than prompt poetry.
- Connect offline FP to online ignore-rate. If engineers start collapsing the panel, the eval is already stale.