Logging every model call is the cheapest governance you will buy

A customer forwards a screenshot. The assistant told them, in fluent and confident prose, that their claim was covered. It was not. By the time the screenshot reaches engineering it is nine days old, the customer is angry, and someone senior wants to know one thing: did the model invent that, or did the retrieval step hand it the wrong document.

What engineering can see is that a request hit the endpoint at 14:07, returned HTTP 200 in 3.1 seconds, and consumed a few thousand tokens. The prompt was assembled in memory and discarded. The response streamed to the browser and was never written down. The system prompt has been edited twice since. The retrieval index rebuilds nightly. There is no route back to that afternoon, so the investigation becomes archaeology, and the meeting ends with a decision to add a warning banner. Nobody learned anything and the same failure is still live.

The record that would have closed that meeting in ten minutes costs a few kilobytes per call. It is the cheapest control available to a team shipping AI features, cheaper than a policy document and far cheaper than a review board, because it is the only one that produces evidence rather than intent. The part that needs real thought is not whether to log. It is retention and redaction, because the moment you start writing prompts to disk you have created the most sensitive store of text your system has ever held.

What an incident actually asks

• What was sent The fully assembled prompt as the model received it: system message, tool definitions, conversation history, retrieved passages, all of it. Not the template it was rendered from.
• What came back The raw completion before your parser, formatter or guardrail touched it, including refusals, truncations, and malformed JSON that your code quietly retried.
• Which version answered Model snapshot, prompt version, sampling parameters, tool schema, index build, guardrail configuration.
• Who saw it User, tenant, session, and whether the output was displayed, edited, discarded, or acted on automatically by something downstream.

Every serious question after an incident decomposes into those four, and they get asked in that order by lawyers, auditors and insurers as well as by engineers. A team that can answer them reconstructs a bad afternoon in an afternoon. A team that cannot is reasoning from a screenshot, which is guessing with better manners.

Log the assembled prompt, not the template

The common half measure is to store a template identifier and the variables, on the theory that you can re-render the prompt later. This fails, because almost nothing that fed the prompt is stable. The retrieval index was rebuilt. The source document was updated by a policy team who did not tell you. The conversation summary was regenerated by another model call. A tool returned live data that no longer exists in that form. Re-rendering months later gives you a prompt that looks right, and looking right is not the property you need.

In an argument about what the model was told, a plausible reconstruction is worth nothing.

The pattern, stated plainly

Pin everything that can change the answer

Model behaviour drifts for reasons that have nothing to do with your code. A provider ships a new snapshot behind a moving alias. Someone tightens the system prompt on a Friday evening. The index picks up a newly published document that contradicts the old one. Each of those changes the answer and none of them leaves a mark in your application logs. Write the versions into the call record itself rather than planning to join to a deploy table later, because deploy tables get pruned and rows get backfilled.

• Model identifier including the dated snapshot, never the moving alias
• System prompt version, or a content hash if you have no version scheme yet
• Sampling parameters, including anything set per request rather than globally
• Tool and function schemas as they stood at call time
• Retrieval index build identifier, plus the document and chunk identifiers returned and their own versions
• Guardrail and content filter configuration
• The application release that assembled all of the above

You have just built the most sensitive table in the system

Your application database is probably governed. Columns are classified, access is role based, someone has thought about which fields may leave the country. A model call log defeats all of that, because free text defeats column level controls completely. People type their symptoms into support chat. They paste a bank statement to ask what a charge was. A case worker describes a family's circumstances in order to get a summary back. All of it lands in one column called prompt, in a table created by a backend engineer on a Tuesday, classified by nobody, with retention set to forever because retention was never set at all.

Sampling is tempting at volume and is usually a false economy. A logged call is a few kilobytes of text set against an inference call that cost real money and real latency, so the storage is a rounding error next to the spend it documents. Sampling earns its place in high volume classification, where the input is a short structured field and the output is one of six labels. Even there, keep the exceptions above complete. The failure mode of sampling is not cost, it is that the one call you need turns out to be the one you dropped.

The log is where your eval set comes from

Teams write their first evaluation suite from imagination. They sit in a room, invent the awkward questions a user might ask, and the suite passes, because they invented questions they already knew how to handle. Real users are stranger than that. They ask two things at once. They misspell the product name. They paste a table into a chat box. They switch language halfway through a sentence. A production log is a standing supply of genuinely awkward inputs that nobody had to think up, and each incident graduates into a regression case. Because the versions are pinned, you can test the fix against the exact configuration that produced the bad answer instead of against a tidied retelling of it.

What a log will not do

It will not stop a bad answer. It is evidence, not a control, and a team that ships logging and considers the governance question settled has bought a filing cabinet and hung a certificate above it. The controls that prevent harm sit in front of the model and around it: what the system is permitted to say, what it may do without a human in the loop, what happens when it is uncertain. Logging tells you afterwards whether those controls held, which is why it sits as the hinge between pre-deployment assurance and post-deployment governance rather than as a replacement for either. And if nobody reads it, it is worse than nothing, because you are now holding a large volume of other people's sensitive text in exchange for no benefit at all. If you are two people running an internal tool for a handful of colleagues, do not build a pipeline for this. Write structured records to a file, set an expiry on them, and spend the week you saved on the guardrails instead.