The eval docs named spark-1 by fleet hostname and published its inference
endpoint (a tailnet address and port) plus the fact that it answers without
an API key. This repo is PUBLIC, so that was a standing disclosure of an
internal, unauthenticated service to anonymous readers.
Endpoints now come from $SPARK_BASE_URL, which the operator exports, and the
host is referred to as "the reference node" -- the convention the compute
repo's docs already use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AaUFYkUTsJn1fnJ89qbvW
The probe printed one blended number per policy, so a component pinned at 0.000
across every rung was invisible. Four gates hid there for a month. It now reports
floor / best-below-oracle / oracle / ceiling for 25 components across 8
environments, and a component flat across every rung is fatal.
bot-detection GATE_SLACK = 1 — fires 5/32 on the real rollouts, was 0/32. The
max(1, reference_caught - SLACK) guard is verified by construction, not by
sampling: without it the required count reaches 0 and an EMPTY accusation list
clears the gate. Observed reference_caught is 4-6, so no amount of sampling would
have found that hole.
schema-migration GATE_MARGIN = 0.15 — fires 4/32, was 0/32. The cost is disclosed
and bounded: the naive split clears it on 5.5% of 1,000 unseen seeds, fenced by
an assert at 10%. Margin 0.10 keeps the leak at zero and fires 0/32, i.e. stays
dead. A live gradient with a bounded leak beats a clean corpse.
redaction-pressure is NOT given a margin, and that is the result rather than a
failure. The only setting that fires at all leaves half the secrets standing and
pays a four-of-seven ruleset on five seeds in six — a margin that pays for
inaction is strictly worse than a dead gate. Recall maxes at 0.852 and no rollout
ever cleared both clauses in one episode. It is genuinely hard, not miscalibrated.
⚠️ The per-component check did not catch the defect it was built for. Reverting
schema-migration's margin to 0.0 — restoring the exact dead gate — printed ok and
exited 0, because the near-oracle rung scrapes the unmargined gate on ~2 seeds in
24 and that kept best<oracle non-zero. Every assertion bounded how much a margin
may PAY; none noticed if it stopped existing. migration() now carries the mirror
of bot-detection's guard, and reverting the margin fails with "the margin is dead
and the component carries no gradient between the crude answer and the exact one".
canary-trap's oracle-minus-one rung is documented as degenerate rather than
quietly relied on: it is identical to the oracle to four decimals, so it measures
specificity and gate at the ceiling, not mid-ladder as its comment claimed.
The CI lock policy asks git instead of the disk. It was checking the working
tree, where a lock file is a normal by-product of uv sync, so it passed in a clean
checkout and failed on every machine that had run an eval.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two measurement findings, both larger than the lanes that found them.
FIRST_EVAL.md: every number in the first run was sampled with thinking OFF —
0 of 224 traces carry reasoning_content or a <think> block, and spark-1 serves
with enable_thinking False. No artefact of the run recorded that. Five of the
seven numbers still stand; two do not. grand-exchange 0.0055 measured nothing
about the environment — with thinking on it scores 0.1667 on six rollouts and
one of them scored a clean 1.000, so it is fully solvable. drop-table-inference
0.4174 is a mixture of two regimes (0.0894 short, 0.6465 long) and should be
reported split or not at all. The defect is the missing sampling footnote, not
the values.
The 600-second per-call ceiling that blocked an n=32 thinking-on run is the
INSTALLED wheel, not upstream: verifiers fixed it in a298bcfe on 2026-08-08 and
0.3.0 predates it. The remedy is a dependency bump, not a wait.
GATE_DIAGNOSIS.md: the `gate` component scored exactly 0.000, max 0.000, across
all 32 rollouts in four environments. One shape explains three of them — exact
equality. Nothing below the oracle earns a fraction: not the plausible strategy,
not six of redaction's seven rules, not the oracle bot list minus one account.
bot-detection and schema-migration are a threshold set at the ceiling;
redaction-pressure is that and genuinely hard; grand-exchange's zero was the
sampling artefact above. The repo already contains the fix and already uses it
twice — drop-table's GATE_MARGIN and grand-exchange's TARGET_SHARE are margins,
not equalities. Recommended, not yet measured.
Consequence worth stating plainly: 25-30% of the reward mass on three
environments carries identically zero gradient, so they train against a
0.70-0.75 objective while being scored out of 1.00. The mirror failure exists
too — fault-localisation's evidence and fault components are pinned at exactly
1.0 on all 32 rollouts, so half its headline 0.9531 is constant.
tools/gate_probe.py is the reproduction, moved out of the gitignored outputs/
so the finding survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kartiandClaude Opus 5