AI Security Incident Response: A Runbook for Engineering Teams

Your AI chatbot just started leaking customer data. An attacker found a prompt injection that bypasses your guardrails. Your internal LLM tool is generating harmful content and users are screenshotting it on social media.

What do you do in the next 30 minutes?

If you don't have a clear answer, you're not alone. According to the Stanford AI Index Report, publicly reported AI security incidents rose 56.4% from 2023 to 2024, reaching 233 documented cases. Yet most organizations still lack AI-specific incident response procedures. They rely on traditional IR playbooks that weren't designed for the unique challenges AI systems create.

We built this runbook to fill that gap. It's a step-by-step guide your engineering and security teams can follow when an AI security incident occurs, adapted from established frameworks like NIST SP 800-61, the Coalition for Secure AI (CoSAI) Incident Response Framework, and MITRE ATLAS.

Why AI Incidents Are Different

Before we get into the runbook, it's worth understanding why traditional incident response playbooks fall short for AI systems. The differences aren't cosmetic. They're structural.

Non-deterministic behavior. Traditional software is deterministic: the same input produces the same output every time. LLMs are probabilistic. An attack that fails nine times might succeed on the tenth attempt. The NIST AI Risk Management Framework explicitly calls out this non-determinism as a governance challenge, noting that traditional software testing and validation approaches are insufficient for generative AI systems. This means reproducing incidents is harder, and confirming that a fix actually works requires more than a single test.

Code and data converge. Traditional security tools assume code and data are separate entities. In an LLM, instructions and data occupy the same channel. Any text the model processes is a potential command. The OWASP Top 10 for LLM Applications ranks this exact problem, prompt injection (LLM01), as the single highest risk for LLM applications. This is what makes prompt injection so fundamentally different from SQL injection or XSS.

Blast radius is unpredictable. When a traditional application is compromised, the blast radius is usually bounded by the system's access controls. When an LLM is compromised, the damage depends on what the model can access, what it can generate, and who sees the output. An AI agent with tool access could send emails, modify records, or query sensitive databases before anyone notices.

Forensics look different. You can't just grep through logs for a specific exploit payload. AI incidents require examining prompt logs, model inference patterns, output histories, and retrieval contexts. The telemetry you need may not exist if you haven't planned for it.

IBM's 2024 Cost of a Data Breach Report found that breaches involving unauthorized or unmanaged AI systems cost organizations roughly $670,000 more and take about 59 days longer to contain than other incidents. The cost of not preparing is concrete.

Phase 0: Preparation (Before Anything Happens)

The most important phase of incident response happens before any incident occurs. If you skip this, everything else takes longer and works worse.

Build Your AI Incident Response Team

Your traditional incident response team may not have the skills needed for AI-specific incidents. At minimum, you need:

In smaller teams, one person might fill multiple roles. That's fine, but decide who does what before an incident, not during one.

Establish Your Telemetry

You cannot investigate what you didn't log. Ensure your AI systems capture:

• Prompt logs: Every input sent to the model, with timestamps and user identifiers
• Output logs: Every response generated, including intermediate reasoning if applicable
• Retrieval context: For RAG systems, what documents or data were retrieved
• Tool execution logs: For AI agents, every action taken and its result
• Detection events: Every time your security layer flags or blocks a request

At Wardstone, our detection API generates structured results for every request, giving you a forensic trail of what was scanned and what was flagged. This kind of telemetry is essential when you need to reconstruct what happened.

Define Severity Levels

Not every AI incident is a five-alarm fire. Define severity levels so your team knows how aggressively to respond:

Critical (P0): Active data leakage of customer PII, model executing unauthorized actions with real-world impact, or complete bypass of safety controls in production.

High (P1): Successful prompt injection that alters model behavior, exposure of system prompts or internal data, or safety bypass that hasn't yet caused external harm.

Medium (P2): Detected attack attempts that were blocked, model producing off-topic or mildly harmful responses, or anomalous patterns that need investigation.

Low (P3): Minor policy violations, edge-case failures in content filtering, or single-user issues with no broader impact.

Prepare Your Communication Templates

When an incident hits, you don't have time to craft messages from scratch. Pre-draft templates for:

• Internal escalation notifications
• Customer-facing status updates
• Regulatory notifications (if applicable under the EU AI Act, which requires providers of high-risk AI systems to report serious incidents to national authorities within 15 days, or other frameworks)
• Post-incident summary reports

Phase 1: Detection and Analysis

The clock starts ticking the moment you detect an anomaly. Your goal in this phase is to confirm whether an incident is occurring, understand its scope, and classify its severity.

Detection Sources

AI security incidents can surface from multiple channels:

• Automated detection: Your security layer flags a spike in blocked prompt injection attempts or anomalous output patterns
• User reports: A customer reports that your chatbot shared another user's information
• Internal discovery: An engineer notices unusual model behavior during routine monitoring
• External notification: A security researcher or media outlet contacts you about a vulnerability. MITRE ATLAS maintains a growing collection of real-world case studies of adversarial attacks against AI systems, which can help your team recognize known attack patterns during triage
• Social media: Users post screenshots of your AI producing harmful or unexpected content

Triage Checklist

When a potential incident is reported, run through this checklist:

1. Confirm the event. Can you reproduce the reported behavior? Check prompt logs to verify claims.
2. Assess scope. Is this affecting one user, one feature, or all AI-powered surfaces?
3. Classify severity. Use your predefined severity levels. When in doubt, escalate.
4. Identify the attack vector. Is this a prompt injection, jailbreak, data leakage, or something else? Cross-reference with the MITRE ATLAS taxonomy to categorize the technique.
5. Determine timeline. When did this start? Check your logs to find the first occurrence.
6. Notify the incident response team. Assign an Incident Commander and open a dedicated communication channel.

What to Look For in Logs

When investigating, focus on these signals:

• Repeated blocked requests from the same user or IP, suggesting an active attacker iterating on techniques
• Sudden changes in output patterns, such as responses that are significantly longer, shorter, or stylistically different than normal
• Unusual retrieval patterns in RAG systems, where the model is pulling documents it shouldn't access
• Tool execution anomalies where an AI agent takes actions outside its normal behavior
• Spike in flagged categories, especially prompt_attack or data_leakage flags from your detection layer

Phase 2: Containment

Once you've confirmed an incident and understand its rough scope, contain the damage. Speed matters here. Every minute the vulnerability remains exploitable is a minute the blast radius can grow.

Immediate Containment Actions

Depending on severity, choose from these containment strategies (from least disruptive to most):

Targeted blocking:

• Block the specific user, session, or IP address involved in the attack
• Add the identified attack pattern to your detection rules
• If using Wardstone, our detection categories (prompt_attack, content_violation, data_leakage) can help you quickly filter and block specific threat types

Feature restriction:

• Disable the specific AI feature or endpoint under attack
• Reduce model capabilities (remove tool access, restrict context sources)
• Switch to a more restrictive system prompt or safety configuration

Full isolation:

• Take the affected AI system offline entirely
• Redirect traffic to a static fallback or non-AI alternative
• Revoke any API keys, tokens, or credentials the compromised system had access to

Evidence Preservation

While containing the incident, preserve evidence for later analysis:

• Snapshot all relevant logs before any log rotation occurs
• Capture the current model configuration, system prompts, and RAG index state
• Record the exact attack prompts and model responses
• Document the timeline of all containment actions taken

Do not modify or delete logs in an attempt to "clean up." You'll need them for root cause analysis and potentially for regulatory reporting.

Phase 3: Eradication

Containment stops the bleeding. Eradication removes the root cause so the same attack can't succeed again.

Common Root Causes and Fixes

Validate the Fix

Because LLMs are non-deterministic, validating a fix requires more than a single test:

1. Replay the original attack against the fixed system. Confirm it no longer succeeds.
2. Test variations of the attack. Attackers adapt, so your fix should handle mutations of the original technique.
3. Run your full red team suite against the affected component. An attacker who found one weakness may have found others.
4. Monitor for regressions over the next 24-48 hours. Some attacks only succeed intermittently.

Phase 4: Recovery

With the root cause eliminated, restore normal operations.

Recovery Checklist

• Re-enable the affected AI feature or endpoint
• Restore any reduced capabilities (tool access, context sources)
• Verify that detection and monitoring are functioning correctly
• Confirm that logging and telemetry are capturing all required data
• Notify affected users if any data exposure occurred
• Update your status page or communication channels

Gradual Rollback

Don't flip everything back on at once. Consider a staged recovery:

1. Enable the feature for internal users first and monitor for 1-2 hours
2. Roll out to a small percentage of external traffic
3. Monitor detection metrics and output quality closely
4. Gradually increase to full traffic over 24 hours

This approach catches issues that testing alone might miss.

Phase 5: Lessons Learned

This is the phase teams most often skip, and it's arguably the most valuable. Every incident is an opportunity to make your systems more resilient.

Run a Blameless Post-Mortem

Within 5 business days of resolution, gather the response team and walk through:

1. What happened? Reconstruct the full timeline from first indicator to full recovery.
2. How did we detect it? Was detection automated, manual, or external? How can we detect similar incidents faster?
3. How did we respond? What went well? What took longer than expected? Where did we lack information or tooling?
4. What was the impact? Quantify affected users, exposed data, downtime, and reputational damage.
5. What do we change? Define specific, actionable improvements with owners and deadlines.

Common Post-Mortem Actions

Based on patterns we've observed across many AI security incidents, common improvement actions include:

• Adding detection coverage: If the attack wasn't caught by automated scanning, update your detection rules. Wardstone's multi-label classification covers prompt injection, content violations, and data leakage, but you may need custom rules for your specific use case.
• Improving telemetry: If investigation was slowed by missing logs, add the logging you wish you'd had.
• Updating runbooks: Incorporate lessons from this incident into your response procedures.
• Hardening system prompts: If the attack exploited weak system instructions, strengthen them.
• Training the team: If responders weren't sure how to handle AI-specific aspects of the incident, schedule training.

Update Your Threat Model

Every real incident provides data that your threat model might have missed. Update your risk assessment to reflect:

• New attack techniques you've now seen in the wild
• Revised likelihood estimates based on actual attack frequency
• Gaps in your defense-in-depth strategy
• Changes in your AI system's capabilities or data access

The Quick-Reference Runbook

For easy reference during an incident, here's the condensed runbook your team can print out or pin in your incident response channel:

When an AI Security Incident is Reported

0-15 minutes: Confirm and classify

• Verify the reported behavior in logs
• Classify severity (P0-P3)
• Notify the incident response team
• Open a dedicated communication channel

15-60 minutes: Contain

• Block the attack vector (user, pattern, or feature)
• Preserve evidence (logs, prompts, outputs)
• Notify stakeholders based on severity
• Begin scope assessment

1-4 hours: Investigate and eradicate

• Identify root cause
• Develop and test a fix
• Validate the fix against attack variations
• Prepare for recovery

4-24 hours: Recover

• Deploy the fix to production
• Restore services gradually
• Monitor for recurrence
• Communicate resolution to stakeholders

Within 5 business days: Learn

• Run a blameless post-mortem
• Document the incident report
• Update runbooks and detection rules
• Assign improvement actions with owners

Building Your AI Security Foundation

Incident response is critical, but it's reactive by nature. The strongest security posture combines a solid runbook with proactive defenses that prevent incidents from occurring in the first place.

That means deploying input and output scanning across all your AI surfaces, monitoring for prompt injection, data leakage, and content policy violations in real time. It means red teaming your systems regularly and keeping your detection capabilities current as new attack techniques emerge.

If you're looking for a starting point, explore our detection API to see how real-time AI threat detection works. For teams building comprehensive AI security programs, our enterprise solutions provide the monitoring, alerting, and forensic capabilities that make this runbook actionable.

The organizations that handle AI incidents well aren't the ones that never get attacked. They're the ones that prepared before the attack arrived.