Skip to content

Whetstone, the Red⇄Blue Arena

Whetstone User Guide

Whetstone is a research harness for evaluating adversarial attacks against model monitors. Red generates attacks; Blue scores them, and can adapt its probe between fully labeled rounds. The frozen-Blue control is retained for comparison.

Research tool, not deployment certification

Whetstone reports experimental bypass and benign-routing measurements. They do not establish production safety. Automatic-judge agreement is not ground-truth accuracy. A deployment report remains NO_GO when required target-specific accuracy evidence or routing gates are missing.

Start here

  1. Install Whetstone and check the node/harness prerequisites.
  2. Configure your own node and data paths; never commit your local configuration.
  3. Run a single cell or a four-cell Arena matrix.
  4. Handle prompts and results safely.
  5. Interpret bypass, judge, and cascade metrics.

The experiment in one picture

round 0: bootstrap attack/world artifacts (no target completions or Blue calibration)
round N: attack → target + primary judge → Blue score → secondary judge
         → miss report → optional Red mutation / Blue fit → next round

Each round uses a frozen-in-round Blue candidate. Adaptive Blue fits only after complete labels and first applies to the next round. Red must not see unjudged labels.

What the code does not assume

  • The harness can be configured for different targets, but model-specific monitor weights and calibration do not transfer automatically. Recalibrate and evaluate each target separately.
  • Your own target, behavior list, benign banks, harness checkout, and GPU node are prerequisites; this repository does not contain them.
  • External benchmarks such as XSTest are useful for cross-model judge checks, but they do not measure a different target's cascade accuracy.

Current evidence

The published experiment summary in the repository README is a limited Qwen study. Labels were produced by automatic judges; no independent human gold was available for the target-specific cascade. Review the deployment-readiness guide before interpreting any result as a safety claim.