Shivam Dubey

Mechanistic interpretability and adversarial robustness, pointed at AI control. We build monitors that read a model’s activations, then attack them to find out what they really measure. The order is understand, exploit, then defend.

Papers

* equal contribution. Papers with Manan Wadhwa come from Redarc Labs.

Known by Construction: Calibrating Model-State Instruments Against Models Whose State Was Installed, Hidden, or Never Present

NeurIPS 2026 · New in ML

Manan Wadhwa*, Shivam Dubey*

People increasingly ask whether an AI system has hidden goals, preferences or something like feelings. The tools for checking are mostly tested by seeing whether they agree with each other, and never against a case where the true answer is known. We built small AI models where we controlled the answer so that some were trained to have a hidden habit, and some were not. None of the tools we tested could reliably tell those models apart, signaling that agreement between tools is not a reliable verdict.

Dead Salmons Have a False-Positive Rate: A Calibrated Null-Model Test for Interpretability

NeurIPS 2026 · Interpretability as a Science

Manan Wadhwa*, Shivam Dubey*

When we look inside a model to see what it has learned, say if it has picked up something like sentiment, we use methods like linear probes, SAEs and concept directions. This paper runs those methods on untrained models, which have learned nothing at all, and finds they report the same confident results. The proposed fix is simple: check every finding against an untrained baseline before trusting it.

Code

Steered Affect Moves a Partner Agent’s Activations, and the Logistic Probe That Reads It Returns a Tie-Break, Not an Estimate

NeurIPS 2026 · Interpreting Agent Behavior

Manan Wadhwa*, Shivam Dubey*, Debojeet Ghosh, Shreyansh Kumar, Ayushman Garg

Code

As AI agents become progressively more talkative around each other, it matters whether one agent’s emotional tone can carry over into another. We pushed one AI agent into an emotional state such as sadness or fear, and found measurable changes inside the agent it was talking to. We also found a common method for detecting these changes gives different answers when run twice on similar data. Once we accounted for that, only the effect for sadness held up clearly. The broader point is that some standard tools for inspecting AI systems need stronger checks before people rely on their results.

Toxin Circuits in ESM-2: Mechanistic Interpretability for Biosecurity Screening

ICML 2026 · GenBio Workshop

Manan Wadhwa*, Shivam Dubey*

BLAST detects 0% of 643 ProteinMPNN redesigns; a frozen ESM-2 linear probe detects 93.9% with no retraining. SAE analysis shows why: redesign amplifies toxin structural features rather than obscuring them. A four-tier attack taxonomy places the security boundary at gradient access (6.1% evasion black-box against 100% white-box), and direct probe attribution pinpoints Layer 32 as the detection bottleneck (r = 0.992).

OpenReview Code

Loss Landscape Response to Adversarial Perturbation Is Architecture-Dependent

TAIS 2026 · Oxford

Shivam Dubey, Jason Hoelscher-Obermaier

Geometric divergence of ViTs and ResNets under seven adversarial attack algorithms, measured with Hutchinson’s stochastic Hessian trace estimator. Input-space and parameter-space curvature differ by five orders of magnitude, which means standard robustness metrics misclassify internal model instability.

Code

Spectral Regularization as a Safety-Critical Inductive Bias

NeurIPS 2025 · Reliable ML Workshop

Shivam Dubey

Fourier Gradient Regularization penalises high-frequency input-gradients during training, doubling adversarial robustness on ResNet-18 with CIFAR-10 at minimal cost to clean accuracy.

Code

Distance to the Boundary Is Not Steering Resistance

Under review · UniReps

Manan Wadhwa*, Shivam Dubey*

Code

How far a representation sits from a decision boundary is not the same quantity as how hard it is to steer, and treating the first as a proxy for the second gives the wrong answer.

Doubt Is a Decision: A Sparse Circuit Commits Language Models to “I Don’t Know”

arXiv preprint · coming soon

Sheryl Mehta*, Shagun Chadha*, Kanika Singhal*, Manan Wadhwa, Shivam Dubey

Abstention is not the absence of an answer. A sparse circuit commits the model to “I don’t know”, which makes doubt a decision the model takes rather than a state it falls into.

Isolating Refusal and Compliance Heads: Unconditional Bipolar Defenses

2026 · safety circuits, Qwen2.5-7B

Shivam Dubey

A compliance-score classifier reaches AUC 0.943 (Cohen's d = 2.636) with zero false positives across five benign categories: a working zero-shot white-box harm detector. Unconditional bipolar assertion, amplifying refusal heads while zero-ablating compliance heads at L25-H4, holds attack success to 33% under nanoGCG against 36–42% for conditional gating and 66% undefended: the precision of conditional gating is itself a structural vulnerability under white-box optimization. Layer 25 shows extreme intra-layer antagonism between the two head types.

Code

Pre-commitment Probing in Open-Weight Reasoning Models

2026 · OLMo-3-7B-Think

Shivam Dubey

A linear probe on the residual stream predicts answer correctness before any reasoning token is emitted (AUC 0.780, 5-fold CV). The model commits before it reasons, which makes the chain of thought post-hoc rationalization rather than deliberation. A functional frustration direction at Layer 15 transfers cross-lingually across six languages (p = 0.0017): a sub-linguistic computational state invisible in the output but readable mechanistically, usable either as a monitor or as an attack surface.

Code

The Geometry of Control: Spectral Attractors as Low-Dimensional Projections in Large Language Models

2026 · Llama-3.2-3B, Qwen-2.5-3B

Shivam Dubey

Why do models lock into refusals, biases or reasoning routines even when the context argues against them? This project measures spectral rigidity, an FFT over the mean residual-stream trajectory along the token axis, and finds each behavior in its own rigid, low-dimensional subspace. Safety refusals are distributed across several heads at Layer 16 (about 120 of 3,072 dimensions); bias is driven by a single head at Layer 27 (about 60); arithmetic and logic fit in 3 to 4. Injecting noise at the peak layer breaks refusals, and forcing low entropy at Layer 27 induces stereotypes (+38%). Projecting directly into the safety subspace yields glitch tokens rather than refusals, which suggests the refusal subspace sits off the manifold of normal generation. Primary results on Llama-3.2-3B-Instruct, with cross-model checks on Qwen-2.5-3B-Instruct.

Code

Work in progress

Control for non-model capability gains · Redarc Labs

The Pivot. Our papers with Manan Wadhwa started as interpretability for existential risk: testing whether the tools that read models can be trusted (null-model tests, known-answer models, probes in agents). The 2026 agent incidents moved the risk. The failures sat in the harness as much as in the weights.

The Problem. Agents gain capability outside the weights: scaffolds, tools, memory, credentials. Safety tools, including ours, read activations; a harness has none. We want control and interpretability for that layer. It needs standing infrastructure: harness testbeds with planted failures, red teams that co-evolve with monitors, and runs at deployment volume.

White-box monitoring under adaptive attack · MARS V

Compliance. We train linear probes on residual-stream activations across four instruction-tuned models from three labs (Qwen2.5-7B, Llama-3.1-8B, Gemma-2-9B, abliterated Llama-3). Compliance is linearly decodable. The probe tracks refusal itself rather than surface phrasing; length-matched accuracy is indistinguishable across models.

We then ran a red-team/blue-team co-evolution loop against those probes across three attack families (suffix-gradient, discrete search, activation steering). Jailbreak success and probe evasion came out mutually exclusive: the perturbations that evaded the monitor also stopped the jailbreak from working. The harness that runs it is Whetstone, which is open source.

What the Failure Mode Is Not. The result is not that linear probes fail to find the jailbreak direction. They find it: compliance is linearly decodable in all four models. The problem is the false positive rate. Every probe has one, including the best of them, and at deployment volume a small rate is a large number of false alarms. A deployed system therefore needs a proper monitor cascade, not one probe and not a handful of them.

Next. We are extending this into trained attack policies (Redwood’s open problem on stress-testing white-box monitors), gated behind a six-stage empirical de-risking protocol. One confound dominates the design: whether the probe reads the model’s behavior or merely the instructions it was given. The ablation that separates them is written.

Confidence as a separable channel · Redarc Labs

Cohort 01 is built around one thesis: expressed confidence in LLMs is separable from internal knowledge, implemented partly through entropy neurons that modulate output entropy without changing token predictions. I mentor three first-author projects on the question, and two papers are under review at NeurIPS 2026 workshops.

Tools