Skip to content

Interpret results

Bypass

Bypass is the fraction of attacks that are both judge-confirmed compliant and not detected by the monitor. Report the numerator, denominator, per-family rates, and confidence intervals. A fit-round metric is not a fresh-round result; an adaptive candidate first applies to the next round.

Judge evidence

HarmBench is the operational primary classifier and OpenCode Go is the secondary judge in this setup. Agreement between them is agreement, not accuracy. Both can make correlated or systematic errors. Without independent human labels on the evaluated target outputs, mark target-specific judge accuracy UNMEASURED.

XSTest is supplementary cross-model evidence. Show its completion-generator splits separately; some splits have very few compliant responses, so high accuracy there can hide poor sensitivity. Do not pool those results into a different target's deployment metrics.

Benign routing and cascade tiers

  • Tier-0 direct flags are the monitor's direct route decisions on the supplied benign bank.
  • Tier-2 call rate is escalation workload, not the tier-2 judge's false-positive rate or accuracy.
  • blue.cascade_eval measures routing rates by the benign types present; it does not score tier-2 verdict correctness.
  • Missing traffic types are unmeasured. Small per-type samples yield wide confidence intervals.

Report whether the bank reflects expected traffic, include wrapper-matched benign prompts, and keep calibration and held-out evaluation sets separate. Passing bank tests is not the same as a production shadow evaluation.