The environment list was written out four times — probe.py's shim tuple and its
results dict, tests/test_probe.py's ENVIRONMENTS, and ci.yml's wheel loop — so
adding an environment meant four edits and five workstreams collided on three
files. Discovery is now a tomllib scan of environments/*/pyproject.toml plus a
@probes(name) registry, with an optional [tool.arena] tasksets key for one
package that ships several (tera_spatial declares four).
An environment discovered without a registered probe WARNS rather than exits 1,
because interactive environments land their engine before their probe. That is
temporary; it becomes a hard failure with the last registration.
The warning is not a licence to stop being gated. test_probe.py pins the
ungated set to an explicit PENDING_PROBES allowlist that only ever shrinks —
without it, removing a @probes decorator moved the name from the table into a
warning line that already carries five, and every remaining assertion stayed
vacuously true while the environment silently stopped being gated.
Also untracks three environments/*/uv.lock files, which had been failing the
lock-policy step on main.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
grand-exchange, bot-detection and drop-table-inference. All three produce
unbounded or irreducible-error scores, unlike the first four, so all three
normalise against a REFERENCE STRATEGY rather than an absolute -- which is what
makes 1.000 reachable rather than aspirational.
drop-table-inference is the clearest case. Scoring KL against the true drop
table would put the ceiling out of reach, because sampling error is irreducible:
ground truth scores 0.900 against the Bayesian posterior's 1.000, and that
ordering is correct. The best estimate available from 1,200 kills is not the
true table, and an environment that demands it is measuring luck.
grand-exchange needed the market to carry structure a model can actually infer,
or profit is noise and no oracle exists. Measured over 120 baskets: the
reference earns 38,821 gp and is profitable in 120/120, random orders lose 561,
and trading only the random-walk items -- which look like the widest-swinging
lines on the board -- loses 16,860.
bot-detection generates naive bots, cloaked bots that jitter on purpose, and
efficient humans who look bot-like on every naive statistic. Timing features are
drawn BEFORE the generator decides who is a bot, so every latency rule sits at
chance. Reference discriminator F1 1.000, random 0.430.
Each was built by one agent then attacked by two independent reviewers on
exploitability and soundness. They earned their keep: drop-table-inference's
first reward let a memorised constant score 0.901 without opening the kill log,
and its reference was 19% worse than a flat number. probe.py now carries three
named fences asserting those attacks stay dead.
First evaluation also lands. Nemotron 3.5 Lightning, thinking off, 32 rollouts
per environment: 0.570 down to 0.059, every gate at or near zero, nothing
solved and nothing unsolvable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kartiandClaude Opus 5