Shipping AI Features Under HIPAA

The awkward property of LLM features in healthcare is that the useful context is exactly the regulated context. A summarisation feature is valuable because it reads the clinical note, and the clinical note is protected health information. There is no version of this where you avoid the question by being careful with the prompt.

Your model provider becomes a subprocessor

Sending PHI to a hosted model means that provider processes PHI on your behalf. That requires a business associate agreement, and not every provider offers one on every tier. Check before the architecture depends on it — this is a common late discovery that forces a rebuild.

• Confirm a BAA is available for the specific service and region you intend to use.
• Confirm the data-retention and training terms in writing: whether inputs are retained, for how long, and whether they can be used for model improvement.
• Record the decision and its date. Terms change, and you will need to show what was true when you shipped.

Logs are the trap

The observability practices that make AI systems debuggable — capturing full prompts, retrieved context and model outputs — are exactly the practices that spray PHI across your logging infrastructure. A trace containing a clinical note is a copy of a medical record in a system that was never scoped for it.

• Redact at capture, not at query. By the time it is in the log store it has been replicated and backed up.
• Keep identifiers out of prompts where the task does not need them. A summarisation task rarely needs the patient's name.
• Segregate AI traces into storage with the same controls and retention rules as the clinical record itself, if redaction is not feasible.
• Audit access to traces. Debugging access to production traces is access to PHI.

Minimum necessary applies to context windows

The minimum-necessary principle maps directly onto retrieval design. Retrieving the whole chart because it is easier than scoping the query is a design decision with a compliance consequence. Scope retrieval to the clinical question being asked, and be able to show what was retrieved for any given request.

Human oversight is a product requirement

Any output that could influence care needs a clinician in the loop, and the interface has to make review real rather than nominal. That means showing the source passages behind a summary, making the model's uncertainty visible, and recording that a human reviewed and what they changed.

What to be able to prove

• Which model version produced a given output, and when.
• What context was retrieved for that request, and under whose authorisation.
• That a qualified human reviewed anything that reached a care decision.
• That PHI in traces is controlled to the same standard as the source record.

Where on-premises models change the calculus

A smaller model running inside your own environment removes the subprocessor question entirely, which in healthcare is often worth more than the accuracy difference. For bounded tasks — extraction, classification, structured summarisation — this is frequently the architecture that clears review fastest.