The 600-second per-model-call ceiling that capped the thinking-on run at n=6 is
gone: DEFAULT_TIMEOUT is read=None in all nine venvs.
There is NO verifiers 0.4.0. The `version = "0.4.0"` in the vendored tree belongs
to a [[tool.uv.dependency-metadata]] block for nemo-gym, and reading it as
verifiers' own sends you hunting a release that does not exist and concluding the
bump is blocked. The vendored checkout is v0.3.0-59-g4bcb48e5, published to PyPI
as 0.3.1.dev59, and the installed wheel diffs clean against the whole vendored
v1 tree. Pinned as an ordinary index dependency: a path source is reproducible
only on this box and a git URL pins a commit no wheel matches.
⚠️ dev59 changes the default agent runtime from subprocess to `prime`, a REMOTE
sandbox. Without an explicit runtime every eval here exits "not authenticated
with prime" before a single rollout — having already created its output directory
and a zero-line traces.jsonl, so it reads as a failed eval rather than an
unauthenticated one. All nine configs now say subprocess. Nothing here needs a
sandbox: the reward is a pure function of the trace and the harness is null.
Output layout is flat now — <env>--<model>--<harness>--<hex8>/ with configs/,
traces.jsonl and a logs/eval.log that 0.3.0 never wrote. `-o` is the output_dir,
not the run dir; `--run.dir` pins the leaf.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bridge becomes a runnable taskset. The crux is that these are continuous
control environments and the agent emits text: the model issues a control
decision that is held for N steps, and the hold length is measured rather than
guessed. At one decision every 32 steps the scripted baseline returns -2.807 and
reaches the waypoint 0% of the time, against 5.484 and 100% at every frame.
Open loop — one decision for the whole flight — returns -3.764 against an
inaction floor of -4.161. A bird that sets a course and leaves does no better
than one that does nothing, which is why crow-nav carries the strongest rule-4
claim of the five spatial environments. The budget bites from both ends: 8 turns
reach the goal 44% of the time, 12 reach 88%, 14 reach 100%.
32 episodes against Qwen3.8-27B: mean 0.4603, no malformed replies in 412, every
episode replaying to an identical checksum, and 16 of 32 spending all 16 turns.
The gate fires 43.75% of the time and separates the four scenarios cleanly —
whatever is wrong with the four dead data-environment gates, it is not that
binary gates cannot discriminate.
The oracle comes from the worker's own op. Nothing recomputes in Python a number
that came out of TypeScript, least of all the reward denominator.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The stepped engine becomes a runnable taskset. protocol.py is both the
render/parse boundary and the security boundary: TurnView carries the live
Market, so view.market.items[i].prices is the whole held-out series and a
foresight policy reading it earned +9.4% on the oracle. The renderer emits only
observed ticks, the trace carries plain scalars, and a sentinel scan of every
rendered turn fails if either reopens. House rule 1 had no other enforcement.
Replies are a delta sheet and parsing never raises — NaN, Infinity, 1e309 and
20k-deep nesting all resolve to a hold, because the one-shot form has already
shown what a real model sends when it gives up.
LiveExchangeEnv.run drives the engine host-side with the harness left null. Two
things learned the hard way and worth keeping: a terminated Segment mid-window
holds the remaining looks instead of raising, and rewards are recorded INSIDE
the interaction, before close. Recording after close writes the traces correctly
and leaves every eval.log line reading reward=0.000 — the canonical verifiers
reference does it after the block; do not copy it there.
The reference-family gate is widened from four named rungs to the whole 8x8
(requote_band, idle_share) family: no member may out-earn the shipped reference
by more than 1.02x, and any that does must still score >= 0.90. That is the check
that caught the first draft's fake 0.152 gap, now shipped rather than remembered.
Also guards `clean` on the reference being profitable, here and in the one-shot
form. Against a reference that lost money the bar is negative, so doing nothing
cleared it and inaction collected a quarter of the reward. Unreachable while
viable_market admits only profitable references; one relaxed filter from
reachable, and exactly the floor house rule 3 forbids. Probe values unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tera's four (now five) spatial environments already implement a Gym-style
contract with checksums, train/dev splits and a replay gate — in TypeScript,
unreachable from `uv run eval`. This bridges them rather than porting them: a
resident NDJSON worker over stdio drives the vendored TS closure, and no reward
arithmetic exists in Python anywhere in the package, including the denominator.
Correctness is replay, not assertion: every scenario x seed must round-trip
through Python and replay to an identical FNV-1a-64 checksum. 32/32 do, and 52
tamper attempts are rejected, 24 of them re-sealed with Tera's own checksum so
it is replay() catching divergence rather than the seal.
Lives in arena rather than tera because tera has no Python toolchain. That
overturns the earlier plan's claim that this workstream shares no files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Arena's first interactive environment, as its own wheel beside grand-exchange so
the one-shot scores stay comparable rather than conflated. One basket of orders
becomes a loop: quote, see fills and the next tick, re-quote, with inventory,
open orders and cash carrying between turns.
The engine and its measurement only. No taskset, no config, no probe row — the
spec's constants were derived from a throwaway engine and did not reconcile
(impatient computed to 0.612 against a reported 0.623), so every constant here
is chosen from a ladder re-measured in the code that ships.
House rule 4 asks whether the turn budget binds. It does: exhaustive 0.631 vs
oracle 1.000, gap 0.369, stdev 0.054 over five blocks, worst block 0.316, and
0.354-0.405 at four further seed bases. That is 18x the proposed margin and it
clears the 0.25 quantum of the binary gate outright, so the comparison is not a
cliff. The saturation worry does not fire: per-basket clipping is not clipping
the mean, so exhaustive averages 0.849 on profit_ratio rather than 1.000.
The first draft's gap was FAKE — a clean-looking 0.152 produced by an under-tuned
denominator whose every-look sibling earned 22% more gp while scoring below it.
It was caught only by sweeping the reference's own family. Nothing that scores
below the oracle may earn more than it; that check belongs in every interactive
environment that follows this one.
freeze is derived from market.seed inside build_market so viability and scoring
see one value, roc is on peak rather than average capital employed, and the
reference is cached per seed so scoring never re-simulates the played episode.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The environment list was written out four times — probe.py's shim tuple and its
results dict, tests/test_probe.py's ENVIRONMENTS, and ci.yml's wheel loop — so
adding an environment meant four edits and five workstreams collided on three
files. Discovery is now a tomllib scan of environments/*/pyproject.toml plus a
@probes(name) registry, with an optional [tool.arena] tasksets key for one
package that ships several (tera_spatial declares four).
An environment discovered without a registered probe WARNS rather than exits 1,
because interactive environments land their engine before their probe. That is
temporary; it becomes a hard failure with the last registration.
The warning is not a licence to stop being gated. test_probe.py pins the
ungated set to an explicit PENDING_PROBES allowlist that only ever shrinks —
without it, removing a @probes decorator moved the name from the table into a
warning line that already carries five, and every remaining assertion stayed
vacuously true while the environment silently stopped being gated.
Also untracks three environments/*/uv.lock files, which had been failing the
lock-policy step on main.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
grand-exchange, bot-detection and drop-table-inference. All three produce
unbounded or irreducible-error scores, unlike the first four, so all three
normalise against a REFERENCE STRATEGY rather than an absolute -- which is what
makes 1.000 reachable rather than aspirational.
drop-table-inference is the clearest case. Scoring KL against the true drop
table would put the ceiling out of reach, because sampling error is irreducible:
ground truth scores 0.900 against the Bayesian posterior's 1.000, and that
ordering is correct. The best estimate available from 1,200 kills is not the
true table, and an environment that demands it is measuring luck.
grand-exchange needed the market to carry structure a model can actually infer,
or profit is noise and no oracle exists. Measured over 120 baskets: the
reference earns 38,821 gp and is profitable in 120/120, random orders lose 561,
and trading only the random-walk items -- which look like the widest-swinging
lines on the board -- loses 16,860.
bot-detection generates naive bots, cloaked bots that jitter on purpose, and
efficient humans who look bot-like on every naive statistic. Timing features are
drawn BEFORE the generator decides who is a bot, so every latency rule sits at
chance. Reference discriminator F1 1.000, random 0.430.
Each was built by one agent then attacked by two independent reviewers on
exploitability and soundness. They earned their keep: drop-table-inference's
first reward let a memorised constant score 0.901 without opening the kill log,
and its reference was 19% worse than a flat number. probe.py now carries three
named fences asserting those attacks stay dead.
First evaluation also lands. Nemotron 3.5 Lightning, thinking off, 32 rollouts
per environment: 0.570 down to 0.059, every gate at or near zero, nothing
solved and nothing unsolvable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
canary-trap, fault-localisation and schema-migration, in the three domains the
Environments Hub has nothing in: contamination, operations, data engineering.
None is a port. Each is graded on a slice the model never read.
canary-trap runs a probe set against a model that memorised the corpus AND one
that learned the same facts from a reworded copy with every identifier
regenerated. The obvious probe -- plant a GUID, ask for it back -- is sound and
caps at 0.525 because it cannot see the paraphrase. It scored higher than that
at first: facts with no distinct rewording had the paraphrase restating the
corpus verbatim, so GUID probes caught it by accident. Every fact now carries a
"core" token present in both wordings and in no public one, asserted at import,
which is also what makes the ceiling reachable.
fault-localisation puts the fault upstream and the noise downstream, the way
incidents actually present. The broken service logs one line and goes quiet
while its dependents log a dozen timeouts, so ranking by error volume answers
the victim -- 0/64 incidents have the loudest service as the root, and probe.py
asserts it rather than trusting the generator to stay that way.
schema-migration executes the model's SQL against forty held-out rows carrying
thousands separators, negatives, a unit with a slash in it and a value with no
unit. Row preservation started as a reward beside fidelity and paid 0.15 for
adding two empty columns and touching nothing; it is a multiplier now, so
losing rows still zeroes a perfect migration and keeping them earns nothing on
its own.
probe.py is house rule 3 as an exit code: 0.000 for inaction and 1.000 for an
oracle, or it returns 1. It loads the leaf modules through a namespace shim
rather than the packages, so it runs without verifiers installed -- a self-check
that needs the training stack is a self-check nobody runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An environment is an eval you take the gradient of, so it carries obligations a
benchmark does not: a held-out slice, a counterweighted reward, and a demonstrated
0.0 floor and 1.0 ceiling. README states the three as house rules.
redaction-pressure is Forge's redact stage as a task. The first reward design scored
survivors — secrets caught, decoys kept, prose kept — and an empty ruleset tied the
best real attempt at 0.500, because destroying nothing keeps everything. Counting
removals instead puts inaction at 0.000, shredding at 0.380, and an oracle at 1.000.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kartiandClaude Opus 5