A prompt change is a deployment, so gate it like one
Prompt edits ship straight to production on the strength of one manual check. Here is what putting an eval suite in CI looks like, and what it honestly costs.
A support lead forwards a complaint. The assistant told a customer their refund window had closed when it had not. Someone opens the prompt file, adds a line clarifying the refund policy, tests it against the one conversation that failed, sees the right answer, and ships. A single sitting, start to finish, and the fix is real.
Weeks later a different complaint arrives. The assistant has started quoting policy at people who asked a simple question about delivery dates. Nobody connects the two events, because nothing recorded that a change was made, what it was meant to fix, or what else it touched. The prompt file has been edited repeatedly since. There is no obvious place to look.
The mistake is not the edit. The edit was correct and useful. The mistake is that a change to production behaviour went out with less scrutiny than a change to a CSS file would have received, because it did not look like code and so did not go through the gate that code goes through.
A prompt is production code that happens to be prose
The reason prompt edits skip review is mostly psychological. Editing a paragraph of English feels like editing documentation. There is no compiler, no type error, no failing unit test to stop you. The feedback loop is a single manual check against the case you were trying to fix, and that check almost always passes, because you wrote the change while looking at that case.
But the artefact you edited sits directly in the request path. It determines what every user sees. It has the blast radius of a deployment and, in most teams we see, none of the controls of one. Version pinning is the same story. A model version change is a deployment nobody on the team performed, which is a good argument for pinning versions explicitly and treating the upgrade as its own gated change.
A change to production behaviour went out with less scrutiny than a change to a CSS file would have received, because it did not look like code.
The pattern, stated plainly
What the gate actually consists of
An eval suite in this context is not an academic benchmark. It is a set of cases you care about, each with an input, and a way of deciding whether the output was acceptable. Some are exact matches, when the task has a right answer such as an extracted field or a routing decision. Some are assertions, checking that a required disclaimer is present or a forbidden phrase is absent. Some need a model as judge, scoring a free-text answer against a rubric, which is the least reliable category and should be the smallest.
The suite should be built out of things that went wrong. Every incident becomes a case. Every complaint that turned out to be legitimate becomes a case. This is the single most useful habit, because it means the suite grows in exactly the places your system is weak, rather than in the places that were easy to write tests for.
- 01On every change to a prompt, tool definition, or model versionRun a fast subset, weighted towards known failure modes and the most common request shapes. Keep it to a few minutes. This is the gate that must not be slow, because it runs constantly.
- 02Before releaseRun the full suite. This is where the long tail lives: the rare intents, the awkward locales, the adversarial inputs, the cases that are slow because they need multiple turns.
- 03On a schedule, against production trafficSample real conversations and score them the same way. Suites go stale because they encode last quarter's traffic. Sampled production tells you when reality has moved.
The pass condition matters as much as the cases. A suite that reports a score and lets a human decide will, under deadline, always be waved through. Set thresholds that fail the build. Usually two: an absolute floor on the whole suite, and a stricter rule that no case which previously passed may now fail. That second rule is the one that catches the refund-policy edit breaking delivery-date answers.
Store the results, or you are only ever measuring today
A pass or fail at the moment of merge is worth having. What is worth much more is the row written to a table: the commit, the prompt hash, the model version, the per-case outcome, the timestamp. Once that table exists you can chart a score over weeks, and the class of failure that used to be invisible becomes obvious.
That class is gradual decline. Nobody ships a change that drops quality off a cliff, because someone would notice. People ship a long run of changes that each cost a little, and the aggregate is a system that is meaningfully worse than it was in spring, which everyone can feel and nobody can date. A stored history dates it. You can point at the week it turned and read the changes that landed.
Where this runs is less important than that it runs on a trigger rather than on someone remembering. A CI job on the pull request handles the fast subset well. The scheduled production sampling belongs in whatever orchestrator already runs your data jobs, Dagster or Airflow or a cron entry, because it has retries, alerting and a history view you already trust. Building a second scheduling system for evals is a common and avoidable mistake.
The part that gets skipped over
Eval runs cost time and money. Every case is at least one model call, judged cases are two, and multi-turn cases are several. A large suite run on every commit across a busy team is a real line on the bill and a real wait on every pull request.
This is where the practice usually dies. Not through disagreement, but because the suite got slow enough to hurt and somebody added a flag to skip it when in a hurry, and before long the flag is the default. Slowness does not cause an argument. It causes quiet abandonment, and abandonment is indistinguishable from never having built it.
- Budget the fast tier in minutes and defend it If the subset creeps past a few minutes, cut cases rather than accepting the creep. Coverage you skip is worth zero.
- Prefer cheap assertions to model judges A regex for a required disclaimer costs nothing and never flakes. Reserve judged cases for genuinely open-ended output.
- Cache aggressively If the prompt and model and input are unchanged, the result is unchanged. Only re-run what the diff could plausibly affect.
- Retire cases A suite is a codebase. Cases that have passed unchanged for as long as anyone can remember, guarding behaviour nothing touches, are cost without information.
When this is the wrong thing to build
If you are early in a prototype and still changing what the product does most weeks, a formal eval suite will encode requirements you are about to discard. Keep a text file of a handful of inputs you paste in by hand. That is genuinely enough until the shape of the thing stops moving.
It is also the wrong investment where the output is not the product. A drafting assistant where a human edits everything before it leaves the building has a human in the loop already, and the useful measurement is how much editing that human does, not a pass rate on a fixed suite. Build the gate where a wrong answer reaches a customer unread. That is the line.
Everywhere else, the argument is simply the one made for automated tests long ago, applied to a file that happens to contain English. You would not let someone edit the pricing calculation and ship it on the strength of one manual check. The prompt deserves the same suspicion, and the mechanism for applying that suspicion is well understood. It just has to be fast enough that nobody wants to turn it off.