AI Security Monitoring: The Metrics You Should Be Tracking

You shipped your LLM-powered application. You added a security layer to catch prompt injections and content violations. Everything looks solid in staging. But now it's running in production with real users, real traffic, and real attackers. How do you know your defenses are actually working?

Most teams deploy AI security and then move on. They don't instrument it. They don't set up alerts. They don't track trends over time. And when something goes wrong, whether that's a missed attack or a spike in false positives blocking legitimate users, they find out from a customer complaint or, worse, a headline.

Monitoring your AI security layer isn't optional. The NIST AI Risk Management Framework emphasizes continuous monitoring as a core function, and NIST AI 600-1 (the Generative AI Risk Profile) specifically calls out the need to track model behavior drift and adversarial input patterns over time. It's the difference between a security system that gets stronger over time and one that slowly degrades without anyone noticing. In this guide, we'll walk through the specific metrics you should be tracking, the thresholds that trigger alerts, and how to build a dashboard that gives your team real visibility into what's happening.

Why AI Security Monitoring Is Different

Traditional application monitoring focuses on uptime, response times, and error rates. Those metrics still matter for your AI security layer, but they don't tell the full story.

AI security monitoring has unique characteristics that standard APM tools don't cover out of the box:

• Confidence scores aren't binary. Unlike a firewall that blocks or allows traffic, an ML-based detection system returns probability scores. You need to track the distribution of those scores, not just the pass/fail outcomes.
• Threat patterns evolve. New jailbreak techniques and evasion methods surface regularly. Your monitoring needs to detect shifts in attack patterns, not just count blocked requests.
• False positives have real business cost. When your security layer incorrectly blocks a legitimate user's request, that's friction, support tickets, and potential churn. Tracking false positive rates is as important as tracking detection rates.
• Latency compounds. Security checks sit in the critical path of every request. A detection system adding 30ms is invisible to users. One adding 500ms is noticeable. You need to know when latency drifts.

The observability ecosystem has evolved to support this. Gartner's 2025 "AI Trust, Risk, and Security Management" (AI TRiSM) research predicted that by 2026, organizations that operationalize AI monitoring and governance would see 50% fewer AI-related incidents than those that don't. Platforms like Datadog, Langfuse, and specialized LLM monitoring tools now offer tracing and metrics tailored to AI workloads. But regardless of which tools you use, the metrics themselves are what matter.

The Core Metrics

Let's break down the metrics into three tiers: operational (keeping the lights on), detection quality (is it working?), and strategic (where are we trending?).

Tier 1: Operational Metrics

These are the basics. If any of these go sideways, you have an immediate problem.

Detection Latency (p50, p95, p99)

How long does each security check take? Track percentiles, not just averages. An average latency of 25ms looks fine until you realize your p99 is 800ms and one in a hundred users is waiting almost a full second for their request to process.

At Wardstone, our ONNX-based inference typically runs at around 30ms. If you see latency climbing, it usually points to one of three things: model loading issues, resource contention on the inference server, or an increase in input length (longer inputs take longer to tokenize and classify).

Throughput (requests per second)

How many security checks are you processing? This is your capacity planning metric. Track it alongside your application's overall request rate. If your application handles 1,000 requests per second but your security layer maxes out at 500, you have a scaling problem waiting to happen.

Error Rate

What percentage of security checks fail entirely? A failed check means one of two things: the request went through unscanned (security gap) or it was blocked by default (user impact). Neither is acceptable. Your error rate should be effectively zero.

Availability

Is your detection endpoint up and responding? This is table stakes, but it's worth calling out because downtime in your security layer has different consequences than downtime in a feature endpoint. If your security checks go down, your application either stops working entirely (if security is in the critical path) or starts operating without protection (if you fail open).

Tier 2: Detection Quality Metrics

These tell you whether your security layer is actually doing its job.

True Positive Rate (Sensitivity)

Of all the actual threats that hit your system, what percentage did you catch? This is the most important metric for your security team. The OWASP Top 10 for LLM Applications recommends continuous red-teaming and evaluation to measure detection effectiveness against evolving prompt injection techniques. A true positive rate below 95% means you're letting one in twenty attacks through.

Measuring this requires labeled data, which is where periodic human review comes in. Sample flagged and unflagged requests weekly, have your team label them, and calculate sensitivity against ground truth. This is manual work, but it's the only way to know if your system is working.

False Positive Rate

Of all the legitimate requests your system scanned, what percentage were incorrectly flagged? This is the most important metric for your product team. A high false positive rate means real users are getting blocked, and they'll tell you about it through support tickets or by leaving.

We've seen production systems achieve false positive rates as low as 2-3% with properly tuned thresholds. Research on ML-based detection systems has demonstrated false positive rates under 0.1% in specialized applications through careful prompt engineering and dataset optimization. The right target depends on your use case: a financial services chatbot can tolerate stricter thresholds (and higher false positives) because the cost of a miss is severe. A creative writing tool needs to be more permissive.

Category-Level Detection Breakdown

Don't just track overall detection rates. Break them down by category. Your system might catch 99% of content violations but only 85% of prompt injection attempts. Or it might flag data leakage reliably while missing a new class of jailbreak that emerged last month.

At Wardstone, we classify across three categories: content violations, prompt attacks, and data leakage. Each gets its own set of metrics. You can explore how these categories work in our documentation or test inputs against them in the playground.

Confidence Score Distribution

Plot the distribution of confidence scores for both flagged and unflagged requests. A healthy distribution shows clear separation: flagged requests cluster at high confidence (0.8-1.0) and unflagged requests cluster at low confidence (0.0-0.2). If you see significant overlap in the middle range (0.3-0.7), your thresholds are operating in ambiguous territory, and you'll likely see both false positives and false negatives.

Track this distribution over time. A shift in the distribution usually signals something meaningful: a change in traffic patterns, a new attack technique, or model drift.

Tier 3: Strategic Metrics

These drive long-term decisions about your security posture.

Threat Category Trends

What types of threats are you seeing over time? A sudden spike in prompt injection attempts might indicate your application is being targeted. A gradual increase in content violations might reflect changes in your user base. A new pattern of data leakage attempts could signal that attackers have found a vector you haven't considered.

Track weekly and monthly trends by category. This is the data your security team needs for threat intelligence and for justifying investment in additional protections.

Block Rate

What percentage of total requests get blocked? This is a high-level health metric. A sudden jump in block rate could mean you're under attack, or it could mean a model update shifted your thresholds. A gradual decline might indicate attackers have found ways around your defenses. Either way, it needs investigation.

Time to Detection

For threats that are eventually identified (through post-hoc analysis, user reports, or downstream impact), how long did it take from the initial request to identification? This metric reveals gaps in your real-time detection and the effectiveness of your asynchronous monitoring.

User Impact Score

Track how security actions affect user experience. Combine false positive rate with user behavior data: did a blocked user retry? Submit a support ticket? Churn? This connects your security metrics to business outcomes and helps you calibrate the tradeoff between safety and usability.

Alerting Thresholds

Metrics are only useful if someone acts on them. Here's a reference table for configuring alerts. Adjust these thresholds based on your traffic volume, risk tolerance, and application context.

A few notes on these thresholds. First, "baseline" means your rolling 7-day average. Daily and weekly seasonality in traffic is normal, so compare against the same time window, not a static number. Second, start with wider thresholds and tighten them as you learn your system's normal behavior. Alert fatigue is real: too many false alerts train your team to ignore all alerts.

Building Your Dashboard

A good AI security dashboard gives your team three views: real-time operational status, daily detection quality, and weekly/monthly strategic trends.

Real-Time Panel

This is what your on-call engineer watches. Keep it simple:

• Current throughput (requests per second, with sparkline)
• Latency percentiles (p50, p95, p99 as gauges with thresholds)
• Error rate (last 5 minutes, last hour)
• Active alerts (count and severity)
• Block rate (last 15 minutes vs. baseline)

Use traffic-light coloring: green when within normal range, yellow at warning thresholds, red at critical. Don't add more than 6-8 panels to this view. Information density kills situational awareness.

Daily Quality Panel

This is for your security team's morning review:

• Detection breakdown by category (stacked bar chart, last 24 hours)
• Confidence score distribution (histogram, flagged vs. unflagged)
• False positive samples (list of recently flagged requests that were overridden or that users contested)
• Top blocked inputs (deduplicated, to identify attack campaigns vs. one-off probes)
• Latency trend (line chart with p50/p95/p99, last 24 hours)

The false positive samples panel is arguably the most valuable. It shows your team exactly where the system is getting it wrong, which drives threshold tuning and surfaces gaps in the model.

Weekly Strategic Panel

This is for leadership and quarterly planning:

• Threat category trends (line chart, 90-day window)
• Block rate trend (with annotations for model updates, threshold changes, and known incidents)
• User impact metrics (false positive rate correlated with support tickets and churn)
• Coverage gaps (categories where detection rates are below target)
• Comparison against industry benchmarks (how your detection rates and latency compare to published standards)

Tooling Choices

You don't need a specialized platform to start. Most teams can build a solid AI security dashboard with the tools they already have:

• Datadog, Grafana, or New Relic for real-time metrics and alerting
• Your existing logging pipeline (ELK, Loki, CloudWatch) for detection event logs
• A simple script that samples flagged/unflagged requests for human review

If you're using Wardstone's detection API, every response includes category-level confidence scores that you can pipe directly into your metrics pipeline. A typical integration logs detection results alongside your application metrics, giving you a unified view without a separate observability stack.

import Wardstone from "wardstone";
 
const wardstone = new Wardstone();
 
async function scanWithMetrics(text: string) {
  const start = performance.now();
  const result = await wardstone.guard(text);
  const latency = performance.now() - start;
 
  // Emit metrics to your observability platform
  metrics.histogram("wardstone.detection.latency_ms", latency);
  metrics.increment("wardstone.detection.total");
 
  if (result.flagged) {
    metrics.increment("wardstone.detection.flagged", {
      tags: [`category:${result.primary_category}`],
    });
 
    // Log detection details for review pipeline
    logger.info("threat_detected", {
      category: result.primary_category,
      scores: result.scores,
      latency_ms: latency,
    });
  }
 
  return result;
}

Real-Time vs. Trend Analysis

Both matter, but they serve different purposes. Getting the balance right is critical.

Real-time monitoring catches acute problems: a sudden spike in latency, an error rate jump, or a burst of attacks hitting your system. These need immediate response. Your on-call engineer should be alerted within minutes, and your runbook should cover the most common scenarios.

Trend analysis reveals slow-moving changes that real-time monitoring misses. A false positive rate that creeps up by 0.5% per week doesn't trigger any single alert, but after two months it's doubled. An evolving attack pattern that shifts gradually from one category to another won't show up in your 15-minute dashboards, but it's visible in a 90-day trend view.

The mistake we see most often is teams investing heavily in real-time alerting while ignoring trend analysis. They catch the fires but miss the slow leaks. Schedule a weekly review of your strategic dashboard. Make it a standing 15-minute meeting for your security team. It's the cheapest way to catch problems that compound over time.

Supplement real-time detection with asynchronous analysis. Run periodic audits on your detection logs: sample requests from the previous week, re-evaluate them with updated models or manual review, and compare results to the real-time classifications. This retroactive analysis catches slow-burn threats and time-delayed exploits that live detection sometimes misses.

Common Pitfalls

We've helped teams set up AI security monitoring across a range of applications. These are the mistakes that come up most often.

Monitoring only the happy path. Teams track detection counts and latency but ignore what happens when the security layer fails. Instrument your fallback behavior. If your detection service is unavailable and you fail open, you need to know about it immediately, not when an attacker notices.

Aggregating too aggressively. An overall detection rate of 97% looks great until you realize it's 99.5% for content violations and only 82% for prompt injection. Always break metrics down by category. The aggregate hides the gaps that attackers will find.

Setting static thresholds. Traffic patterns change. User behavior shifts. Attack techniques evolve. Your alert thresholds should be based on rolling baselines, not fixed numbers you set six months ago and never revisited. Review and adjust thresholds monthly.

Ignoring the feedback loop. When a user contests a blocked request, that's labeled data. When your support team overrides a false positive, that's labeled data. Feed these signals back into your monitoring and, eventually, into your model. Teams that close this loop improve their detection quality continuously. Teams that don't stagnate.

Not correlating with application metrics. Your AI security metrics don't exist in isolation. The MITRE ATLAS framework maps adversarial ML techniques to the broader ATT&CK matrix, reinforcing that AI attacks often overlap with traditional application-layer threats. A spike in detection events that correlates with a traffic spike from a new marketing campaign is different from a spike that correlates with a DDoS attack. Join your security metrics with your application metrics for context.

Getting Started

If you're not currently monitoring your AI security layer, here's a practical path forward.

Week 1: Instrument the basics. Add latency, throughput, and error rate metrics to your detection pipeline. If you're using Wardstone, this means logging the response time and status of each /api/detect call. Set up alerts for error rate (> 1%) and latency (p95 > 250ms).

Week 2: Add detection quality metrics. Start logging confidence scores and categories for every detection event. Build a simple dashboard showing detection counts by category and confidence score distribution. Review it daily.

Week 3: Establish baselines. After two weeks of data, calculate your baseline block rate, latency percentiles, and category distribution. Set up anomaly alerts based on these baselines. Start a weekly review meeting.

Week 4: Close the loop. Set up a process for sampling flagged requests and validating them. Calculate your false positive rate. Identify your weakest detection category and investigate why. Tune thresholds based on what you find.

From there, iterate. Add trend analysis dashboards. Correlate security metrics with user experience metrics. Build out your runbooks for common alert scenarios. The teams that treat AI security as a continuous practice, not a one-time deployment, are the ones that stay ahead of evolving threats.

Check out our documentation for integration details, or explore the playground to understand how detection categories and confidence scores work before you start instrumenting. If you're still building your broader AI security program, our guide on building an AI security program covers the organizational and governance dimensions that complement the monitoring practices outlined here.