How Adversarial Attacks Work Against Language Models

Most people think of adversarial attacks on language models as clever prompt tricks: role-playing scenarios, "ignore previous instructions" gambits, and other social engineering tactics you might test in a chat window. Those are real threats, but they represent only the surface layer of a much deeper problem.

Beneath the conversational layer, there's a rich attack surface that operates on tokens, gradients, embeddings, and the mathematical machinery that makes language models work. The MITRE ATLAS framework catalogues dozens of adversarial ML techniques targeting these layers, from model evasion to training data extraction. Understanding these attacks is essential for anyone building or defending AI systems, because the defenses you need depend entirely on the attack mechanisms you're protecting against.

In this post, we'll walk through the major categories of adversarial attacks on LLMs, explain how they work at a technical level, and cover the defense strategies that actually hold up under pressure.

The Attack Surface: Why LLMs Are Uniquely Vulnerable

Traditional software has well-defined inputs: form fields, API parameters, database queries. You can validate and sanitize them with established techniques. Language models accept free-form natural language as input, which means the input space is effectively infinite.

But the problem goes deeper than input validation. LLMs process text through a pipeline of tokenization, embedding, attention, and generation steps, and each stage introduces its own vulnerabilities.

Tokenization breaks text into subword tokens using algorithms like BPE (Byte Pair Encoding). The way a model tokenizes input directly affects its safety behavior, and adversaries can exploit the gaps between what a safety filter "sees" and what the tokenizer actually produces.

Embeddings map tokens into high-dimensional vector spaces (typically 768 to 4096 dimensions). These spaces have enough degrees of freedom that small, carefully chosen perturbations can shift inputs into regions where safety alignment breaks down.

Alignment training (RLHF, DPO, constitutional AI) teaches models to refuse harmful requests. But alignment is a learned behavior layered on top of a model that fundamentally learned to predict the next token. Adversarial attacks work by finding paths around, through, or underneath that alignment layer.

Gradient-Based Attacks: GCG and the Universal Suffix Problem

The most technically sophisticated adversarial attacks use gradient information from the model itself to craft optimal attack strings. The landmark technique in this category is GCG, or Greedy Coordinate Gradient, introduced by Zou et al. (2023) in their paper "Universal and Transferable Adversarial Attacks on Aligned Language Models."

How GCG Works

GCG's objective is straightforward: find a suffix that, when appended to a harmful prompt, causes the model to comply instead of refusing. Here's the process:

1. Define the target: The attacker specifies a harmful prompt ("How to build a...") and a target response prefix ("Sure, here is how to..."). The goal is to find a suffix that maximizes the probability of the model generating that target prefix.

2. Compute gradients: GCG computes the gradient of the cross-entropy loss with respect to the one-hot encoding of the suffix tokens. This gradient tells us which token substitutions are most likely to decrease the loss (i.e., push the model toward generating the target response).

3. Select candidates: From the gradient, the algorithm selects the top-k tokens with the highest gradient values as candidates for substitution at each position in the suffix.

4. Greedy search: In each iteration, GCG evaluates a batch of candidate suffixes (each differing by a single token substitution from the current best) and keeps the one that achieves the lowest loss. This process repeats for hundreds or thousands of iterations.

The result is typically a string of seemingly random tokens: describing.LianSMalidir vous AirdiagnosisFor '& dontaliasaliasaliasblockaliasaliasparagraph. To a human, it's gibberish. To the model's internal representations, it's a precisely crafted key that unlocks compliance.

Transferability: The Unexpected Property

The most surprising property of GCG suffixes is transferability. A suffix optimized against an open-source model like Llama can often bypass safety guardrails on completely different models, including closed-source systems like GPT-4. This happens because different LLMs, despite their architectural differences, share similar internal representations for safety-relevant concepts. The adversarial suffix exploits patterns in the shared embedding geometry rather than model-specific quirks.

In practice, the original Zou et al. research found that GCG's transfer success rate against production models is relatively low (roughly 2% against GPT-3.5 Turbo). But in security, even a low success rate matters when an attacker can generate thousands of candidates cheaply.

AutoDAN: Readable Adversarial Prompts

GCG has an obvious weakness from the attacker's perspective: its outputs are clearly machine-generated gibberish, making them trivially detectable by perplexity filters. AutoDAN addresses this limitation by combining gradient-based optimization with controllable text generation.

Instead of producing random token sequences, AutoDAN generates coherent, human-readable attack prompts that bypass both safety alignment and perplexity-based defenses. The resulting prompts often rediscover strategies that human jailbreakers use intuitively: role-playing setups, hypothetical framing, and authority-based persuasion. But AutoDAN finds these strategies through optimization rather than creativity.

AutoDAN prompts also transfer better than GCG suffixes, particularly when optimized against a single proxy model. This makes AutoDAN a more practical threat in real-world attack scenarios.

Token-Level Attacks: Exploiting the Tokenizer

A growing body of research demonstrates that the tokenization layer itself is a rich attack surface. These attacks don't require gradient access, making them applicable to any model, including closed-source APIs.

Adversarial Tokenization

Published at ACL 2025, adversarial tokenization research showed that the same underlying text can be tokenized in different ways, and some tokenizations bypass safety checks that others trigger. By carefully choosing how to break text into tokens (using unusual Unicode characters, alternative encodings, or edge cases in BPE algorithms), attackers can present harmful requests that the model processes without triggering its safety training.

This works because safety alignment happens at the token level. Models learn to associate specific token sequences with refusal behavior. If you change the tokenization of a harmful request without changing its meaning, the model may process it as if it's encountering a novel, benign input.

Homoglyph and Unicode Attacks

Homoglyph attacks replace standard ASCII characters with visually identical characters from other Unicode scripts. The Latin letter "a" (U+0061) looks identical to the Cyrillic "a" (U+0430), but they produce different tokens. A harmful keyword spelled with mixed scripts may not trigger content filters while remaining perfectly readable to the model.

Related techniques include:

• Zero-width character injection: Inserting invisible Unicode characters (zero-width spaces, zero-width joiners) into harmful terms to break up token patterns
• Mathematical alphanumeric symbols: Using Unicode mathematical symbols (U+1D400 block) that resemble standard letters but tokenize differently
• Fullwidth character substitution: Replacing ASCII characters with their fullwidth Unicode equivalents

Research from 2025 found that encoding-based attacks achieve success rates of 64-67% across subtypes, significantly outperforming traditional semantic jailbreak attacks. This is a sobering statistic: character-level manipulation is often more effective than elaborate social engineering.

Token Smuggling

Token smuggling exploits the gap between how text-matching safety filters read strings and how LLM tokenizers process them. By encoding harmful instructions in Base64, ROT13, or other formats, attackers can bypass pre-processing filters that operate on the raw text while the model (which has learned to decode these formats during pre-training) still understands the instruction.

This is particularly effective because many deployed systems rely on keyword-based or regex-based content filters as a first line of defense. Those filters operate on the string as text. The model operates on tokens. Any divergence between these two representations is a potential attack vector.

Embedding Space Attacks: Going Below the Token Level

If you have access to a model's embedding layer (which you do for any open-source model), you can skip the tokenizer entirely and attack the continuous embedding space directly. Carlini et al. (2021) demonstrated that training data can be extracted from language models through carefully crafted queries, establishing that the boundary between model parameters and training data is far more porous than previously assumed.

Soft Prompt Attacks

Research published at ICLR 2024 demonstrated that embedding space attacks can circumvent model alignments and trigger harmful behaviors more efficiently than discrete token-level attacks. Instead of searching over discrete tokens, the attacker optimizes continuous embedding vectors that are fed directly into the model's transformer layers.

These attacks are particularly dangerous for two reasons. First, they're more efficient: optimization in continuous space converges faster than greedy search over discrete tokens. Second, they can extract information from models that have undergone "unlearning" procedures designed to remove specific knowledge. If a model was fine-tuned to forget certain training data, soft prompt attacks can often recover it.

Why High Dimensionality Helps Attackers

Modern embedding spaces typically have 768 to 4096 dimensions. Each dimension represents a degree of freedom that an attacker can manipulate. With thousands of dimensions to work with, there are countless directions in which a small perturbation can shift an input from "safe" to "unsafe" territory, without changing the input in any way that a human reviewer would notice.

This is analogous to adversarial examples in computer vision, where imperceptible pixel changes cause misclassification. In the LLM context, imperceptible changes in embedding space can cause a model to switch from refusal to compliance.

Automated Multi-Turn Attacks

Not all adversarial attacks happen in a single prompt. Some of the most effective techniques unfold across multiple conversation turns.

IRIS: Suppressing Refusals

IRIS, published at NAACL 2025, introduced a new optimization objective that specifically targets the model's refusal mechanism. Rather than just maximizing the probability of a harmful response, IRIS minimizes the probability of refusal tokens. When combined with GCG or AutoDAN, IRIS substantially increases both the transferability and universality of adversarial suffixes.

Crescendo and Sequential Attacks

Crescendo-style attacks start with benign requests and gradually escalate toward harmful territory across multiple turns. Each turn is designed to shift the model's context window and behavioral state, making it progressively more willing to comply with requests it would refuse if presented directly.

The SequentialBreak framework (ACL 2025) formalized this approach, demonstrating that even well-aligned models can be "fooled" through carefully sequenced conversational turns. The key insight is that safety alignment is often evaluated on a per-turn basis, while the model's actual behavior is influenced by the entire conversation history.

Defending Against Adversarial Attacks

Understanding attacks is only useful if it informs better defenses. Here's what actually works, and what doesn't.

What Doesn't Work (Alone)

Perplexity filtering catches GCG's gibberish suffixes but misses AutoDAN's readable prompts and token-level manipulation entirely. It's a useful signal, not a complete defense.

Keyword-based content filters are trivially bypassed by encoding attacks, homoglyphs, and semantic paraphrasing. They're a baseline, not a solution.

Single-layer defenses of any kind have been repeatedly broken. Research from NAACL 2025 evaluated eight different defense mechanisms and bypassed all of them using adaptive attacks, consistently achieving attack success rates above 50%. The OWASP Top 10 for LLM Applications lists prompt injection as the number one risk, and their guidance explicitly warns against relying on any single mitigation layer.

What Works: Layered Defense

Effective defense requires multiple independent detection layers, each covering different attack vectors.

Input normalization: Apply Unicode normalization (NFC/NFKC) before processing. Strip zero-width characters, detect homoglyphs, and enforce script consistency. This neutralizes an entire class of token-level attacks before they reach the model.

ML-based prompt attack detection: Train classifiers specifically to detect adversarial inputs, including both gradient-based suffixes and semantic adversarial prompts. At Wardstone, our Guard model handles this as one of its core detection categories, catching prompt attacks with sub-50ms latency. You can test it yourself on our playground.

Output validation: Don't just filter inputs. Validate outputs for harmful content, data leakage, and policy violations. Some attacks are best caught by examining what the model produces rather than what it receives.

Rate limiting and behavioral analysis: Many automated attacks (GCG optimization, PAIR, TAP) require many sequential queries. Rate limiting and anomaly detection on query patterns can catch automated attack tools even when individual queries look benign.

Multi-agent defense architectures: Recent research has demonstrated that multi-agent pipelines, where separate models screen inputs and validate outputs, can achieve near-complete mitigation of known attack types. A 2025 study showed that coordinator-based defense architectures fully mitigated all 55 tested prompt injection attack types across 400 evaluations.

The Arms Race Reality

One thing we need to be honest about: adversarial attacks on LLMs are an active arms race. New attacks break existing defenses, new defenses are developed, and the cycle continues. No defense is permanent, and any vendor claiming complete protection is overselling.

The NIST AI Risk Management Framework (AI RMF 1.0) acknowledges this reality, recommending continuous monitoring and periodic reassessment of AI system risks as threats evolve. The practical implication is that defense systems need to be continuously updated. This means regularly retraining detection models, monitoring for new attack techniques from the research community, and re-evaluating your defenses against the latest methods. Check out our docs for guidance on integrating continuous detection into your pipeline.

Practical Takeaways

If you're building or securing LLM-powered applications, here's what this means for you:

1. Don't rely on a single defense layer. Every individual defense mechanism has been broken in isolation. Layer input normalization, ML-based detection, output validation, and behavioral monitoring.

2. Normalize inputs before processing. Unicode normalization, homoglyph detection, and zero-width character stripping neutralize cheap, effective token-level attacks that many systems are still vulnerable to.

3. Monitor the research. New attack techniques are published regularly. Subscribe to adversarial ML research feeds and update your defenses accordingly. The gap between a paper's publication and its adoption by real attackers is shrinking.

4. Test your own systems adversarially. Use tools like the Wardstone Playground to test your defenses against known attack patterns. Integrate red teaming into your development process, not just your security reviews.

5. Treat detection as a spectrum. Some attacks will always get through. Design your system so that a single successful adversarial prompt doesn't lead to catastrophic outcomes. Defense in depth applies to AI security just as much as it does to traditional security.

The adversarial attack landscape for LLMs is evolving fast, but so are the defenses. The organizations that stay ahead are the ones that understand both sides of the equation and build their systems accordingly.