You cannot choose a model without a graded set of your own examples

A team we spoke to had narrowed their choice to three models. They had a spreadsheet of public benchmark scores, a rough sense of price per million tokens, and a launch date. What they did not have was a single example of their own documents with an agreed correct answer written next to it. The question they were actually asking was not which model is better in general. It was whether any of these will read a partially scanned remittance advice, notice that the reference number is written in a house format nobody outside the finance team uses, and match it to the right invoice. No leaderboard answers that.

This is the most common failure we see in AI work, and it is not a modelling failure. It is that nobody wrote down what good looks like before spending money on the thing that is supposed to produce it. Without a graded set of your own examples, every model comparison collapses into whoever demoed most persuasively on the day, and every later change to the prompt or the retrieval step becomes an argument between people recalling different anecdotes.

The fix is dull and it works. Assemble roughly a hundred real examples from your own data, decide with the people who know the domain what the right answer is for each one, freeze the set, and only then start comparing. What follows is how to do that without it taking a quarter, and an honest account of the part that cannot be made pleasant.

What a public benchmark is actually measuring

Benchmarks are useful for what they were built for: comparing general capability across model families on tasks with clean, agreed answers. They tell you something real about reasoning, coding and instruction-following in the abstract. What they cannot tell you is anything about the distribution of inputs your feature will see.

Your inputs are not a clean distribution. They contain a supplier who spells their own company name two ways, a product code scheme that changed a few years ago with no backfill, a category of query your support team knows is ambiguous and handles by asking a clarifying question, and a set of documents where the important number sits in a footer that OCR mangles often enough to matter. A model that scores well on general tasks may handle all of that gracefully or may fall over on the abbreviations, and the leaderboard position gives you no way to tell which.

A benchmark measures general capability. You are buying performance on your own edge cases, and those are the only thing worth measuring.

The pattern, stated plainly

Build the set from failures, not from happy paths

The instinct when assembling examples is to pick representative ones. That produces a set where nearly everything passes, the candidates all cluster at the top, and the differences between them vanish into noise. A set that cannot separate the candidates is a set that cost you weeks and told you nothing.

Weight it towards the awkward cases instead. Go to where failures are already recorded, because in most organisations they are: support tickets that got reopened, the manual-review queue, the exceptions spreadsheet somebody maintains, the Slack channel where people paste screenshots with the word odd. Those are your hard cases and they are free. Add the ones the domain experts flag from memory when you ask what usually goes wrong, because those tend to be the categories that matter commercially even when they are rare.

• Recorded failures Reopened tickets, manual overrides, correction logs. Cases the business already knows were handled wrongly.
• Known ambiguity Queries where the right answer is to ask a question rather than assert one. Models that guess confidently here are dangerous in production.
• Format drift Documents from before a schema change, from an acquired entity, from the one supplier who sends a PDF of a photograph of a page.
• Ordinary volume Enough straightforward cases, maybe a third of the set, that a model which fails the easy work is caught rather than flattered.
• Cases with a legal or financial edge Anything where a wrong answer creates an obligation. These get graded most carefully and weighted separately.

Grading is the actual work

For each example you need a recorded expected answer and, more importantly, a rule for deciding whether a given output matches it. For extraction tasks this is often straightforward: the invoice total is a number, either it is right or it is not. For anything generative it is harder, and the temptation is to skip to a vague notion of quality that nobody can apply consistently.

Write the grading rule down as if handing it to somebody who joined last week. If the task is answering a customer question from your documentation, the rule might be that the answer must cite the correct policy section, must not state a figure absent from the source, and must decline when the source does not cover the question. Three checks, each of which a person or a script can apply the same way twice. Graders who disagree with each other are a signal that the rule is underspecified, not that one of them is wrong, so run a small overlap deliberately and go back and sharpen the rule when they diverge.

Freeze it, then treat it like code

The moment you start comparing models you must stop editing the set. Otherwise every disappointing result becomes a quiet argument that the example was unfair, the set drifts towards whatever the current favourite handles well, and the comparison is worthless. Freeze it, version it, put it in Postgres or a repository, and record every result against the version it was run on.

Keep a second, growing set separately. New failures found in production go there, and it becomes the next frozen version at a planned point. This is the same discipline as holding out test data, applied to a problem where the pressure to peek is much higher because a business decision is waiting on the number.

What the set changes once you have it

Model choice is the reason people usually build one, and it is the least of what it buys you. Once a frozen graded set exists, every subsequent decision that would otherwise be a matter of opinion becomes a measurement. Whether adding a retrieval step over your documents helps, and by how much. Whether the cheaper model is genuinely worse on your work or only worse on benchmarks. Whether the prompt somebody rewrote last Thursday improved anything. Whether a new model release is worth migrating to, which is a question that recurs and which nobody can answer from a changelog.

It also changes the conversation with the business. Instead of asking whether the AI is good enough, which has no answer, you can say that on the frozen set the system handles a defined category of case correctly and fails on another defined category, and let people decide whether that trade is acceptable. Most stakeholders are far more comfortable with a known failure mode than with an unquantified one.

The awkward part

Building this set is not engineering work and it cannot be delegated to the engineers. The value is entirely in the judgement about what the right answer is, and that judgement lives with the people who do the job: the credit analyst, the support lead, the person who has been reconciling those accounts for years. Engineers can build the harness quickly. They cannot tell you that this particular reference format means the payment was made by the parent entity and should be matched differently.

So the honest cost is a real block of a senior domain person's time, spread over weeks, doing work that is repetitive and produces nothing visible. That is a genuinely difficult thing to ask for, and it is the reason most teams skip it. It is also why we put it in week one of an engagement rather than later, when the pressure to show a working demo makes it impossible to justify.

There are cases where this is the wrong advice. If you are using a model for something low-stakes and reversible, where a human reads every output before it goes anywhere, a hundred graded examples is over-engineering and you should just ship it and watch. If you cannot get domain time at all, be clear that you are choosing a model on vibes, and size the commitment accordingly rather than pretending otherwise. And if the task genuinely has no stable notion of a correct answer, an eval set will give you a false sense of rigour, which is worse than admitting the uncertainty. The discipline is worth the cost precisely where a wrong answer reaches a customer, a ledger or a regulator without a person in between.