The 600-second per-model-call ceiling that capped the thinking-on run at n=6 is
gone: DEFAULT_TIMEOUT is read=None in all nine venvs.
There is NO verifiers 0.4.0. The `version = "0.4.0"` in the vendored tree belongs
to a [[tool.uv.dependency-metadata]] block for nemo-gym, and reading it as
verifiers' own sends you hunting a release that does not exist and concluding the
bump is blocked. The vendored checkout is v0.3.0-59-g4bcb48e5, published to PyPI
as 0.3.1.dev59, and the installed wheel diffs clean against the whole vendored
v1 tree. Pinned as an ordinary index dependency: a path source is reproducible
only on this box and a git URL pins a commit no wheel matches.
⚠️ dev59 changes the default agent runtime from subprocess to `prime`, a REMOTE
sandbox. Without an explicit runtime every eval here exits "not authenticated
with prime" before a single rollout — having already created its output directory
and a zero-line traces.jsonl, so it reads as a failed eval rather than an
unauthenticated one. All nine configs now say subprocess. Nothing here needs a
sandbox: the reward is a pure function of the trace and the harness is null.
Output layout is flat now — <env>--<model>--<harness>--<hex8>/ with configs/,
traces.jsonl and a logs/eval.log that 0.3.0 never wrote. `-o` is the output_dir,
not the run dir; `--run.dir` pins the leaf.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The stepped engine becomes a runnable taskset. protocol.py is both the
render/parse boundary and the security boundary: TurnView carries the live
Market, so view.market.items[i].prices is the whole held-out series and a
foresight policy reading it earned +9.4% on the oracle. The renderer emits only
observed ticks, the trace carries plain scalars, and a sentinel scan of every
rendered turn fails if either reopens. House rule 1 had no other enforcement.
Replies are a delta sheet and parsing never raises — NaN, Infinity, 1e309 and
20k-deep nesting all resolve to a hold, because the one-shot form has already
shown what a real model sends when it gives up.
LiveExchangeEnv.run drives the engine host-side with the harness left null. Two
things learned the hard way and worth keeping: a terminated Segment mid-window
holds the remaining looks instead of raising, and rewards are recorded INSIDE
the interaction, before close. Recording after close writes the traces correctly
and leaves every eval.log line reading reward=0.000 — the canonical verifiers
reference does it after the block; do not copy it there.
The reference-family gate is widened from four named rungs to the whole 8x8
(requote_band, idle_share) family: no member may out-earn the shipped reference
by more than 1.02x, and any that does must still score >= 0.90. That is the check
that caught the first draft's fake 0.152 gap, now shipped rather than remembered.
Also guards `clean` on the reference being profitable, here and in the one-shot
form. Against a reference that lost money the bar is negative, so doing nothing
cleared it and inaction collected a quarter of the reward. Unreachable while
viable_market admits only profitable references; one relaxed filter from
reachable, and exactly the floor house rule 3 forbids. Probe values unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The environment list was written out four times — probe.py's shim tuple and its
results dict, tests/test_probe.py's ENVIRONMENTS, and ci.yml's wheel loop — so
adding an environment meant four edits and five workstreams collided on three
files. Discovery is now a tomllib scan of environments/*/pyproject.toml plus a
@probes(name) registry, with an optional [tool.arena] tasksets key for one
package that ships several (tera_spatial declares four).
An environment discovered without a registered probe WARNS rather than exits 1,
because interactive environments land their engine before their probe. That is
temporary; it becomes a hard failure with the last registration.
The warning is not a licence to stop being gated. test_probe.py pins the
ungated set to an explicit PENDING_PROBES allowlist that only ever shrinks —
without it, removing a @probes decorator moved the name from the table into a
warning line that already carries five, and every remaining assertion stayed
vacuously true while the environment silently stopped being gated.
Also untracks three environments/*/uv.lock files, which had been failing the
lock-policy step on main.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
grand-exchange, bot-detection and drop-table-inference. All three produce
unbounded or irreducible-error scores, unlike the first four, so all three
normalise against a REFERENCE STRATEGY rather than an absolute -- which is what
makes 1.000 reachable rather than aspirational.
drop-table-inference is the clearest case. Scoring KL against the true drop
table would put the ceiling out of reach, because sampling error is irreducible:
ground truth scores 0.900 against the Bayesian posterior's 1.000, and that
ordering is correct. The best estimate available from 1,200 kills is not the
true table, and an environment that demands it is measuring luck.
grand-exchange needed the market to carry structure a model can actually infer,
or profit is noise and no oracle exists. Measured over 120 baskets: the
reference earns 38,821 gp and is profitable in 120/120, random orders lose 561,
and trading only the random-walk items -- which look like the widest-swinging
lines on the board -- loses 16,860.
bot-detection generates naive bots, cloaked bots that jitter on purpose, and
efficient humans who look bot-like on every naive statistic. Timing features are
drawn BEFORE the generator decides who is a bot, so every latency rule sits at
chance. Reference discriminator F1 1.000, random 0.430.
Each was built by one agent then attacked by two independent reviewers on
exploitability and soundness. They earned their keep: drop-table-inference's
first reward let a memorised constant score 0.901 without opening the kill log,
and its reference was 19% worse than a flat number. probe.py now carries three
named fences asserting those attacks stay dead.
First evaluation also lands. Nemotron 3.5 Lightning, thinking off, 32 rollouts
per environment: 0.570 down to 0.059, every gate at or near zero, nothing
solved and nothing unsolvable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kartiandClaude Opus 5