Model Risk and Audit Trails in Financial Services AI

Banks have had model risk management for years: models are inventoried, validated independently, monitored for drift and owned by a named person. Generative AI arrives into that framework rather than around it, and the friction is real — the discipline assumes a model you can specify, test against a known distribution, and explain.

Why LLMs are awkward for traditional MRM

• Non-determinism. The same input can produce different output, so "re-run the validation" is not a clean operation.
• Opaque provenance. You cannot describe the training data of a third-party frontier model, which is a standard validation input.
• Silent version changes. A provider upgrade changes behaviour without a change on your side — an unversioned dependency in a regime built on versioned ones.
• Unbounded output space. Traditional models emit a score; an LLM emits text that may contain a recommendation, a justification and a hallucination in one paragraph.

The instrumentation that makes governance possible

Almost every question a validator or auditor asks reduces to reconstructing a specific decision. Build for that:

• Pin and record the model version on every inference, including the provider's version string. Treat provider upgrades as a change requiring re-validation.
• Record the full input — system prompt, retrieved context, user input — not just the user's message.
• Record the raw output before any post-processing, plus what was shown to the user.
• Record the human decision layered on top: who reviewed, what they changed, what they approved.
• Record the retrieval set and the authorisation under which it was retrieved.

Separate the model's role from the decision

The cleanest governance position is that the model does not make the decision. It drafts, summarises, retrieves or flags; a person or a deterministic rule decides. This is not a legal fiction — it has to be true in the interface, with the human able to see the evidence and realistically able to disagree.

Where the model does influence an outcome directly, expect it to be treated as a model in the full sense: inventoried, independently validated, monitored, and owned.

Drift monitoring for text

Traditional drift monitoring watches input distributions and score stability. The analogues for an LLM feature:

• Refusal and escalation rates - a sudden fall usually means behaviour changed, not that the world got easier.
• Groundedness scores on sampled production traces.
• Output length and structure distribution, which shift noticeably after a provider upgrade.
• Rate of human override, which is the most honest quality signal you have.

Retention and reconstruction

The practical test is whether you can reconstruct, two years later, exactly what the system was shown and what it produced for a specific customer interaction. That means traces retained on the same schedule as the decision records they support, and an index that lets you find them by customer and date rather than only by trace ID.