The problem
In a large engineering org, the story is the contract. If the contract is vague, every downstream system looks like a delivery problem: slipped sprints, rework, “the engineers didn’t understand the requirement”. Most of the time they understood it. There was not enough of it to understand.
The usual fixes are process: definition-of-ready checklists, more grooming, more templates. Those fail in the same way. They are binary, they are gamed and they do not scale to the number of stories a portfolio actually writes. Leadership still cannot answer a simple question: are we feeding the machine work that is specified well enough to build?
That is an evaluation problem, not a workflow problem. I treated it as one.
Why a score and why not a chatbot
The tempting AI product is a writing assistant: “improve this story”. That is a feature. It does not give an org a signal. A GVP cannot look at a portfolio and see whether requirement quality is getting better. A team cannot be held to a bar they cannot see.
So the product is a score with a reason. 0–5, not pass/fail. Pass/fail gets you the same theater as a DoR checkbox. A scale lets you see distribution, not just red/green. Explainable subscores are the part that makes it usable: if the model says 2, the author has to know whether that is missing acceptance criteria, an untestable outcome, a hidden dependency or a story that is actually an epic.
A score without a reason is an accusation. A reason without a score is an essay. The product is both or it does not ship into an org like this.
Behavior I was willing to own
I am not going to invent the internal rubric here. The contract I will own in public is the behavior:
- The system scores a story. It does not silently rewrite it.
- The output is a 0–5 plus subscores a human can disagree with. Disagreement is a feature. If nobody can contest the score, you will not get adoption from people whose work is being judged.
- The score is a quality signal, not an access-control gate, until the org trusts the distribution. Gating on v1 of an LLM judge is how you create a black market of prompt-stuffed stories.
- Failure mode I care about more than “the model is sometimes wrong”: confident wrongness on a well-written story that uses domain language the model does not know. That is how you lose senior PMs in week two.
- Fallback: if the input is empty, truncated or clearly not a story, do not emit a 3 because 3 is the mean. Refuse or score with an explicit low-confidence flag.
What “good” means
Eval for a story judge is not SWE-bench. There is no unit test for “this acceptance criterion is testable”. You need a rubric, a labeled set and a place where the model and a senior human systematically disagree.
| Question | Why it matters |
|---|---|
| Does the overall 0–5 match a senior PM within ±1 on a held-out set? | If not, you do not have a score. You have a vibe. |
| Do subscores point at the actual defect? | A correct 2 with the wrong reason trains authors to fix the wrong thing. |
| Does it fail closed on garbage input? | Mean-reverting scores on blank tickets will be screenshot in Slack and the product dies. |
| Is it stable under harmless paraphrase? | If rewriting the same story moves 5 → 2, authors will optimize the judge, not the work. |
I will not claim a public accuracy number I cannot show. The shipped artifact is the scoring product in use, not a blog benchmark. Internally, the only eval that eventually mattered was: do portfolio owners and GVP leadership trust the distribution enough to look at it on purpose?
Replication as the real test
Story Quality is not interesting if it only works on stories. The same failure mode exists one level up: epics that are slogans. We replicated the core into Epic Quality rather than building a second product from scratch. That is the systems test. If your scoring core is a prompt stuffed into one field, you cannot clone it. If it is a scoring contract (input schema, rubric dimensions, output contract, explainability) you can.
It now spans 48 portfolios. That number is the product. A clever judge on one team is a pilot. A judge that survives 48 different writing cultures, domain vocabularies and incentive structures is an evaluation system.
What changed
GVP leadership uses it to gauge requirement quality. That sentence is doing a lot of work. It means the score escaped the team that built it. It is no longer a writing aid. It is an org signal. Once that happens, the PM job changes: you are now responsible for score drift, for people gaming the rubric and for the politics of a number that can make a portfolio look worse than its neighbors.
I would rather own that than own a chatbot that makes stories longer.
What I would do next
- Publish an internal confusion matrix by story type (bug vs feature vs chore vs spike). Average accuracy across types is a lie.
- Track override rate: how often a human changes the score and whether overrides cluster on one subscore. That is your next rubric revision, for free.
- Separate underspecified from wrong-sized. Those are different author behaviors. Collapsing them into one “quality” dimension is how you get a 2 that nobody can act on.
- Do not auto-gate sprint admission until override rate and ±1 agreement are boring. Gating is a privilege the eval has to earn.