Key takeaway: AI incidents have no error to catch and no obvious failure signal. Response depends on having a kill switch, sufficient logging to reconstruct what happened, and a decision path defined before the pressure arrives.
Why Existing Runbooks Do Not Apply
Conventional incident response starts with detection — an error rate, a failed health check, a latency alarm. A model that produces a harmful or badly wrong answer triggers none of those. The request succeeded, the latency was normal, and the output was well-formed.
Detection therefore comes from users, from social media, or from a journalist. By that point the output has been seen, possibly screenshotted, and the team is responding to a public account of something they cannot yet reproduce.
Reproduction is the second difficulty. Sampling means the same input may not produce the same output, and if the exact prompt, retrieved context and model version were not logged, the specific case cannot be recreated at all.
What Must Exist Beforehand
| Capability | Why it matters during an incident |
|---|---|
| Feature kill switch | Stops the harm within seconds |
| Per-request logging of prompt, context, model version | Makes reproduction possible |
| Ability to pin or roll back model version | Reverses a bad upgrade |
| A defined severity scale for AI-specific harms | Prevents debate during response |
| Named decision-maker for disabling a feature | Prevents delay |
| User-facing reporting channel | Turns discovery from luck into process |
The kill switch is the single most important item. A feature flag that disables the AI-dependent behaviour and falls back to a non-AI path — a search interface instead of a generated answer, a queue instead of automated triage — converts an unbounded incident into a degraded but safe service.
Designing that fallback is work that must happen while building the feature. Retrofitting it during an incident is not possible.
Severity for This Class of Failure
Ordinary severity scales key on availability and are the wrong axis here. AI incidents need severity based on harm and reach.
A model producing a legally or medically dangerous statement is severe at any volume. A model producing an embarrassing but harmless answer is minor even at high volume. A model exposing another user’s data through retrieval is a data breach and follows that path, not this one.
Writing these distinctions down in advance means the response starts with action rather than with an argument about how serious it is.
Investigation and Follow-Up
Reconstruct with the logs: the exact input, the retrieved context, the model and prompt version, and the output. Determine whether the cause was the retrieval layer surfacing wrong content, the prompt permitting something it should have constrained, a model version change, or genuine model error.
Each cause has a different fix, and conflating them produces changes that do not address what happened.
Then add the case to the evaluation set permanently. An AI incident that does not become a regression test will recur, because nothing else prevents it. That single practice is what converts incident response into improvement rather than repetition.
The Bottom Line
Build a kill switch with a non-AI fallback while building the feature, log prompt, context and model version for every request, define severity by harm rather than availability, name who can disable a feature, and add every incident to the evaluation set so it cannot happen twice.
