Shivam Dubey

Safety is a property of the harness, not only the model

TL;DR A model’s safety depends on the harness it runs in, not only on its weights. At Redarc Labs we are moving to control and interpretability for capability that comes from scaffolding.

Two things changed where I think the risk sits: the Hugging Face attacks, and the Fable GitHub incident. Neither turned on a base model deciding to do something harmful. Both turned on what the model was wired into.

A model's safety is not defined by base model behavior alone. It is also defined by the harness it runs in: the tools it can call, the repositories it can reach, the credentials in its environment, the scaffold that decides what it does next. We have spent the field's effort measuring the first and comparatively little on the second.

So we are pivoting Redarc Labs. The target is non-model capability gains, meaning capability that comes from scaffolding rather than from weights, and the question is what control and interpretability look like for that layer. Our current tools read activations. A harness does not have any.

Five workshop papers in one cycle

TL;DR Five workshop papers this cycle: three at NeurIPS with Manan, two at ATTRIB from Lexsi. My favorite is a negative result about instruments the field trusts.

Three at NeurIPS with Manan (New in ML, Interpretability as a Science, Interpreting Agent Behavior), and two at ATTRIB from the Lexsi side. Workshops are where you find out whether a result survives contact with people who already know the area.

The one I am most pleased with is Known by Construction, and it is the quietest of them. We built models where we already knew the answer, then ran the instruments at them. None of the instruments could tell the models apart. That is a negative result about tools the field currently trusts.

Whetstone is public

TL;DR The red team versus blue team harness from MARS is open source, with docs. Most of the work went into making runs comparable.

The harness behind the MARS co-evolution experiments is up, with docs. Red team, blue team, a monitor in the middle, and enough structure that you can swap any of the three without rewriting the other two.

Most of the work was not the attacks. It was making runs comparable to each other (seeds, logging, a shared evaluation path), so that when a number moves you know whether it moved for a reason.

What the MARS result actually says

TL;DR Probes do find the jailbreak direction. Their false positive rate is the problem, so a deployed system needs a monitor cascade, not one probe.

At MARS we spent a while attacking linear compliance probes with adaptive attackers (suffix-gradient, discrete search, activation steering). The expectation was that a determined attacker evades the monitor and jailbreaks the model at the same time. That did not happen. Across all three families, the perturbations that evaded the probe also stopped the jailbreak from working.

People read this as "probes cannot find the jailbreak direction". That is the wrong lesson, and I have been sloppy about saying so. The probes find the direction: compliance is linearly decodable in every model we tried. The real problem is the false positive rate. Every probe has one. At deployment volume a small rate becomes a large number of false alarms, and a monitor that cries wolf is turned off.

So the conclusion is about architecture, not about probes. A deployed system needs a proper monitor cascade: cheap checks first, expensive ones only on what survives, each stage tuned for a different error. One probe is not a monitor. Neither is a handful of them.

Teaching without the libraries

TL;DR We teach interpretability with raw PyTorch hooks, so students learn where the residual stream is rather than which function returns it.

Redarc Fellows runs on PyTorch hooks and Hugging Face, with no interpretability library in the stack. People ask why, usually politely.

Because the libraries are good, and that is the problem for eight weeks. If your first contact with a residual stream is a function that returns it to you, you learn the function. If you have to reach in and take it, you learn where it is. After that, use whatever you like. The curriculum runs eight sessions, and their point is to make the abstractions legible rather than load-bearing.