4 Commits

Include renames

Author	SHA1	Message	Date
karti	61939676a5	Pin Arena environments to stable Verifiers	2026-08-25 10:33:35 -07:00
kartiandClaude Opus 5	122a858b61	pin verifiers 0.3.1.dev59, and make every config say subprocess ... The 600-second per-model-call ceiling that capped the thinking-on run at n=6 is gone: DEFAULT_TIMEOUT is read=None in all nine venvs. There is NO verifiers 0.4.0. The `version = "0.4.0"` in the vendored tree belongs to a [[tool.uv.dependency-metadata]] block for nemo-gym, and reading it as verifiers' own sends you hunting a release that does not exist and concluding the bump is blocked. The vendored checkout is v0.3.0-59-g4bcb48e5, published to PyPI as 0.3.1.dev59, and the installed wheel diffs clean against the whole vendored v1 tree. Pinned as an ordinary index dependency: a path source is reproducible only on this box and a git URL pins a commit no wheel matches. ⚠️ dev59 changes the default agent runtime from subprocess to `prime`, a REMOTE sandbox. Without an explicit runtime every eval here exits "not authenticated with prime" before a single rollout — having already created its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than an unauthenticated one. All nine configs now say subprocess. Nothing here needs a sandbox: the reward is a pure function of the trace and the harness is null. Output layout is flat now — <env>--<model>--<harness>--<hex8>/ with configs/, traces.jsonl and a logs/eval.log that 0.3.0 never wrote. `-o` is the output_dir, not the run dir; `--run.dir` pins the leaf. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 23:14:05 -07:00
kartiandClaude Opus 5	4147de0282	tera-crow-nav: the first spatial taskset, and the hold sweep that sized its budget ... The bridge becomes a runnable taskset. The crux is that these are continuous control environments and the agent emits text: the model issues a control decision that is held for N steps, and the hold length is measured rather than guessed. At one decision every 32 steps the scripted baseline returns -2.807 and reaches the waypoint 0% of the time, against 5.484 and 100% at every frame. Open loop — one decision for the whole flight — returns -3.764 against an inaction floor of -4.161. A bird that sets a course and leaves does no better than one that does nothing, which is why crow-nav carries the strongest rule-4 claim of the five spatial environments. The budget bites from both ends: 8 turns reach the goal 44% of the time, 12 reach 88%, 14 reach 100%. 32 episodes against Qwen3.8-27B: mean 0.4603, no malformed replies in 412, every episode replaying to an identical checksum, and 16 of 32 spending all 16 turns. The gate fires 43.75% of the time and separates the four scenarios cleanly — whatever is wrong with the four dead data-environment gates, it is not that binary gates cannot discriminate. The oracle comes from the worker's own op. Nothing recomputes in Python a number that came out of TypeScript, least of all the reward denominator. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 17:44:46 -07:00
kartiandClaude Opus 5	162d67a83c	tera-spatial: an NDJSON replay bridge to the TypeScript spatial environments ... Tera's four (now five) spatial environments already implement a Gym-style contract with checksums, train/dev splits and a replay gate — in TypeScript, unreachable from `uv run eval`. This bridges them rather than porting them: a resident NDJSON worker over stdio drives the vendored TS closure, and no reward arithmetic exists in Python anywhere in the package, including the denominator. Correctness is replay, not assertion: every scenario x seed must round-trip through Python and replay to an identical FNV-1a-64 checksum. 32/32 do, and 52 tamper attempts are rejected, 24 of them re-sealed with Tera's own checksum so it is replay() catching divergence rather than the seal. Lives in arena rather than tera because tera has no Python toolchain. That overturns the earlier plan's claim that this workstream shares no files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 15:55:12 -07:00