Eight environments gated, one warning left (tera-crow-nav). The probe calls
measure_ladder.ladder() rather than re-deriving the policies, and seeds from the
skip-aware baskets() helper — a contiguous range() grades a different set of
baskets than the model is served.
Beyond the four-rung ladder it asserts what house rule 4 actually claims: the
turn budget binds in every block of twenty-four, sweeps all 64 cells of the
reference family, scans the rendered turns for held-out ticks, and checks that
hostile replies parse to a hold and score zero.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
grand-exchange, bot-detection and drop-table-inference. All three produce
unbounded or irreducible-error scores, unlike the first four, so all three
normalise against a REFERENCE STRATEGY rather than an absolute -- which is what
makes 1.000 reachable rather than aspirational.
drop-table-inference is the clearest case. Scoring KL against the true drop
table would put the ceiling out of reach, because sampling error is irreducible:
ground truth scores 0.900 against the Bayesian posterior's 1.000, and that
ordering is correct. The best estimate available from 1,200 kills is not the
true table, and an environment that demands it is measuring luck.
grand-exchange needed the market to carry structure a model can actually infer,
or profit is noise and no oracle exists. Measured over 120 baskets: the
reference earns 38,821 gp and is profitable in 120/120, random orders lose 561,
and trading only the random-walk items -- which look like the widest-swinging
lines on the board -- loses 16,860.
bot-detection generates naive bots, cloaked bots that jitter on purpose, and
efficient humans who look bot-like on every naive statistic. Timing features are
drawn BEFORE the generator decides who is a bot, so every latency rule sits at
chance. Reference discriminator F1 1.000, random 0.430.
Each was built by one agent then attacked by two independent reviewers on
exploitability and soundness. They earned their keep: drop-table-inference's
first reward let a memorised constant score 0.901 without opening the kill log,
and its reference was 19% worse than a flat number. probe.py now carries three
named fences asserting those attacks stay dead.
First evaluation also lands. Nemotron 3.5 Lightning, thinking off, 32 rollouts
per environment: 0.570 down to 0.059, every gate at or near zero, nothing
solved and nothing unsolvable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
canary-trap, fault-localisation and schema-migration, in the three domains the
Environments Hub has nothing in: contamination, operations, data engineering.
None is a port. Each is graded on a slice the model never read.
canary-trap runs a probe set against a model that memorised the corpus AND one
that learned the same facts from a reworded copy with every identifier
regenerated. The obvious probe -- plant a GUID, ask for it back -- is sound and
caps at 0.525 because it cannot see the paraphrase. It scored higher than that
at first: facts with no distinct rewording had the paraphrase restating the
corpus verbatim, so GUID probes caught it by accident. Every fact now carries a
"core" token present in both wordings and in no public one, asserted at import,
which is also what makes the ceiling reachable.
fault-localisation puts the fault upstream and the noise downstream, the way
incidents actually present. The broken service logs one line and goes quiet
while its dependents log a dozen timeouts, so ranking by error volume answers
the victim -- 0/64 incidents have the loudest service as the root, and probe.py
asserts it rather than trusting the generator to stay that way.
schema-migration executes the model's SQL against forty held-out rows carrying
thousands separators, negatives, a unit with a slash in it and a value with no
unit. Row preservation started as a reward beside fidelity and paid 0.15 for
adding two empty columns and touching nothing; it is a multiplier now, so
losing rows still zeroes a perfect migration and keeping them earns nothing on
its own.
probe.py is house rule 3 as an exit code: 0.000 for inaction and 1.000 for an
oracle, or it returns 1. It loads the leaf modules through a namespace shim
rather than the packages, so it runs without verifiers installed -- a self-check
that needs the training stack is a self-check nobody runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An environment is an eval you take the gradient of, so it carries obligations a
benchmark does not: a held-out slice, a counterweighted reward, and a demonstrated
0.0 floor and 1.0 ceiling. README states the three as house rules.
redaction-pressure is Forge's redact stage as a task. The first reward design scored
survivors — secrets caught, decoys kept, prose kept — and an empty ruleset tied the
best real attempt at 0.500, because destroying nothing keeps everything. Counting
removals instead puts inaction at 0.000, shredding at 0.380, and an oracle at 1.000.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kartiandClaude Opus 5