The 600-second per-model-call ceiling that capped the thinking-on run at n=6 is
gone: DEFAULT_TIMEOUT is read=None in all nine venvs.
There is NO verifiers 0.4.0. The `version = "0.4.0"` in the vendored tree belongs
to a [[tool.uv.dependency-metadata]] block for nemo-gym, and reading it as
verifiers' own sends you hunting a release that does not exist and concluding the
bump is blocked. The vendored checkout is v0.3.0-59-g4bcb48e5, published to PyPI
as 0.3.1.dev59, and the installed wheel diffs clean against the whole vendored
v1 tree. Pinned as an ordinary index dependency: a path source is reproducible
only on this box and a git URL pins a commit no wheel matches.
⚠️ dev59 changes the default agent runtime from subprocess to `prime`, a REMOTE
sandbox. Without an explicit runtime every eval here exits "not authenticated
with prime" before a single rollout — having already created its output directory
and a zero-line traces.jsonl, so it reads as a failed eval rather than an
unauthenticated one. All nine configs now say subprocess. Nothing here needs a
sandbox: the reward is a pure function of the trace and the harness is null.
Output layout is flat now — <env>--<model>--<harness>--<hex8>/ with configs/,
traces.jsonl and a logs/eval.log that 0.3.0 never wrote. `-o` is the output_dir,
not the run dir; `--run.dir` pins the leaf.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
canary-trap, fault-localisation and schema-migration, in the three domains the
Environments Hub has nothing in: contamination, operations, data engineering.
None is a port. Each is graded on a slice the model never read.
canary-trap runs a probe set against a model that memorised the corpus AND one
that learned the same facts from a reworded copy with every identifier
regenerated. The obvious probe -- plant a GUID, ask for it back -- is sound and
caps at 0.525 because it cannot see the paraphrase. It scored higher than that
at first: facts with no distinct rewording had the paraphrase restating the
corpus verbatim, so GUID probes caught it by accident. Every fact now carries a
"core" token present in both wordings and in no public one, asserted at import,
which is also what makes the ceiling reachable.
fault-localisation puts the fault upstream and the noise downstream, the way
incidents actually present. The broken service logs one line and goes quiet
while its dependents log a dozen timeouts, so ranking by error volume answers
the victim -- 0/64 incidents have the loudest service as the root, and probe.py
asserts it rather than trusting the generator to stay that way.
schema-migration executes the model's SQL against forty held-out rows carrying
thousands separators, negatives, a unit with a slash in it and a value with no
unit. Row preservation started as a reward beside fidelity and paid 0.15 for
adding two empty columns and touching nothing; it is a multiplier now, so
losing rows still zeroes a perfect migration and keeping them earns nothing on
its own.
probe.py is house rule 3 as an exit code: 0.000 for inaction and 1.000 for an
oracle, or it returns 1. It loads the leaf modules through a namespace shim
rather than the packages, so it runs without verifiers installed -- a self-check
that needs the training stack is a self-check nobody runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An environment is an eval you take the gradient of, so it carries obligations a
benchmark does not: a held-out slice, a counterweighted reward, and a demonstrated
0.0 floor and 1.0 ceiling. README states the three as house rules.
redaction-pressure is Forge's redact stage as a task. The first reward design scored
survivors — secrets caught, decoys kept, prose kept — and an empty ruleset tied the
best real attempt at 0.500, because destroying nothing keeps everything. Counting
removals instead puts inaction at 0.000, shredding at 0.380, and an oracle at 1.000.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kartiandClaude Opus 5