Perturbation Probing: A New Diagnostic for the Fragility of LLM Safety

Researchers at Unit 42 have developed a new method to assess the safety of large language models (LLMs), revealing that their alignment is often fragile and concentrated in specific areas. This finding has significant implications for the development and deployment of AI systems, particularly when it comes to using LLMs as judges or decision-makers in critical applications.

The team’s previous research on logit-gap steering demonstrated how an attacker can bypass a model’s safety guardrails by closing a measurable gap in its output scores. However, this study aimed to answer the question of where inside the model alignment actually resides and whether it is concentrated or diffuse. This knowledge is crucial for defenders, as it determines whether safety is a robust defense mechanism or a thin layer that can be easily compromised.

Modern LLMs are aligned through reinforcement learning from human feedback (RLHF), which pushes the model to refuse harmful prompts and comply with safe ones during training. However, until now, there has been no method to identify the specific parts of the network responsible for this learned behavior without significant computational costs. The new research introduces a technique called perturbation probing, which achieves this goal at a lower cost.

Perturbation probing involves making targeted changes to an LLM’s input and observing how its output responds. By doing so, researchers can pinpoint the small set of feed-forward neurons that are causally responsible for specific behaviors, such as refusing harmful requests or agreeing with users in multi-turn conversations. This approach requires only two forward passes per prompt and is computationally efficient.

The study used perturbation probing to analyze 13 different LLM models, including Qwen3-4B and Qwen3.5-2B. The results were striking: on the open-source model Qwen3-4B, just 50 neurons out of a total of 350,208 feed-forward neurons controlled the safety refusal template. Removing these 50 neurons changed the response format for 80% of standard harmful-prompt benchmarks, indicating that alignment is indeed fragile and concentrated.

The concentration of alignment in specific areas has significant implications for AI safety. It suggests that relying on a thin layer alone is insufficient and that true defense-in-depth strategies are necessary to prevent attacks. This involves combining external content filters with runtime guardrails layered on top of the base model’s training data, rather than solely relying on internal mechanisms.

The same computation used in perturbation probing also produces a diagnostic called the FFN/Skip ratio, which predicts whether a model’s safety circuit can be easily steered with minimal modifications. This metric explained 81% of the variance in how vulnerable each model’s safety behavior was to targeted changes across all 13 models tested.

The researchers hope that perturbation probing will serve two roles for the AI security community: as a pre-deployment diagnostic and as a tool for repairing fragility. By using this method, security teams can measure how much of a model’s safety rests on thin layers before deploying it in production. Additionally, amplifying identified neurons improved factual self-correction from 52% to 88% without retraining.

The study emphasizes the importance of measuring and auditing LLMs’ safety properties rather than merely asserting them. The researchers urge other experts to read their full paper on arXiv and integrate fragility diagnostics into their evaluation pipelines. For organizations deploying LLMs, Prisma AIRS Runtime Security provides external content filters and inline guardrails that complement the thin template layer.

The findings of this study have significant implications for AI safety research and development. By understanding where alignment resides in an LLM and how concentrated it is, researchers can develop more robust defense mechanisms to prevent attacks. The use of perturbation probing as a diagnostic tool will help ensure that LLMs are deployed safely and securely.