lumbridge-public/arena
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

pin verifiers 0.3.1.dev59, and make every config say subprocess

Browse Source 
The 600-second per-model-call ceiling that capped the thinking-on run at n=6 is
gone: DEFAULT_TIMEOUT is read=None in all nine venvs.

There is NO verifiers 0.4.0. The `version = "0.4.0"` in the vendored tree belongs
to a [[tool.uv.dependency-metadata]] block for nemo-gym, and reading it as
verifiers' own sends you hunting a release that does not exist and concluding the
bump is blocked. The vendored checkout is v0.3.0-59-g4bcb48e5, published to PyPI
as 0.3.1.dev59, and the installed wheel diffs clean against the whole vendored
v1 tree. Pinned as an ordinary index dependency: a path source is reproducible
only on this box and a git URL pins a commit no wheel matches.

⚠️ dev59 changes the default agent runtime from subprocess to `prime`, a REMOTE
sandbox. Without an explicit runtime every eval here exits "not authenticated
with prime" before a single rollout — having already created its output directory
and a zero-line traces.jsonl, so it reads as a failed eval rather than an
unauthenticated one. All nine configs now say subprocess. Nothing here needs a
sandbox: the reward is a pure function of the trace and the harness is null.

Output layout is flat now — <env>--<model>--<harness>--<hex8>/ with configs/,
traces.jsonl and a logs/eval.log that 0.3.0 never wrote. `-o` is the output_dir,
not the run dir; `--run.dir` pins the leaf.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
karti
2026-08-21 23:14:05 -07:00
co-authored by Claude Opus 5
parent 8048a61a1b
commit 122a858b61
19 changed files with 388 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
configs/tera_crow_nav.toml
+9
View File
@@ -23,3 +23,12 @@ seed_base = 115
[env.agent.harness]
id = "null" # one completion per turn, no tools; the env programs the loop
[env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
# an unauthenticated one. Nothing here needs a sandbox: the reward is a pure function of the
# trace and the harness is `null`.
type = "subprocess"