4 Commits

Include renames

Author	SHA1	Message	Date
karti	61939676a5	Pin Arena environments to stable Verifiers	2026-08-25 10:33:35 -07:00
kartiandClaude Opus 5	122a858b61	pin verifiers 0.3.1.dev59, and make every config say subprocess ... The 600-second per-model-call ceiling that capped the thinking-on run at n=6 is gone: DEFAULT_TIMEOUT is read=None in all nine venvs. There is NO verifiers 0.4.0. The `version = "0.4.0"` in the vendored tree belongs to a [[tool.uv.dependency-metadata]] block for nemo-gym, and reading it as verifiers' own sends you hunting a release that does not exist and concluding the bump is blocked. The vendored checkout is v0.3.0-59-g4bcb48e5, published to PyPI as 0.3.1.dev59, and the installed wheel diffs clean against the whole vendored v1 tree. Pinned as an ordinary index dependency: a path source is reproducible only on this box and a git URL pins a commit no wheel matches. ⚠️ dev59 changes the default agent runtime from subprocess to `prime`, a REMOTE sandbox. Without an explicit runtime every eval here exits "not authenticated with prime" before a single rollout — having already created its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than an unauthenticated one. All nine configs now say subprocess. Nothing here needs a sandbox: the reward is a pure function of the trace and the harness is null. Output layout is flat now — <env>--<model>--<harness>--<hex8>/ with configs/, traces.jsonl and a logs/eval.log that 0.3.0 never wrote. `-o` is the output_dir, not the run dir; `--run.dir` pins the leaf. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 23:14:05 -07:00
kartiandClaude Opus 5	f953c03bd1	grand-exchange-live: protocol layer, taskset, config, widened reference gate ... The stepped engine becomes a runnable taskset. protocol.py is both the render/parse boundary and the security boundary: TurnView carries the live Market, so view.market.items[i].prices is the whole held-out series and a foresight policy reading it earned +9.4% on the oracle. The renderer emits only observed ticks, the trace carries plain scalars, and a sentinel scan of every rendered turn fails if either reopens. House rule 1 had no other enforcement. Replies are a delta sheet and parsing never raises — NaN, Infinity, 1e309 and 20k-deep nesting all resolve to a hold, because the one-shot form has already shown what a real model sends when it gives up. LiveExchangeEnv.run drives the engine host-side with the harness left null. Two things learned the hard way and worth keeping: a terminated Segment mid-window holds the remaining looks instead of raising, and rewards are recorded INSIDE the interaction, before close. Recording after close writes the traces correctly and leaves every eval.log line reading reward=0.000 — the canonical verifiers reference does it after the block; do not copy it there. The reference-family gate is widened from four named rungs to the whole 8x8 (requote_band, idle_share) family: no member may out-earn the shipped reference by more than 1.02x, and any that does must still score >= 0.90. That is the check that caught the first draft's fake 0.152 gap, now shipped rather than remembered. Also guards `clean` on the reference being profitable, here and in the one-shot form. Against a reference that lost money the bar is negative, so doing nothing cleared it and inaction collected a quarter of the reward. Unreachable while viable_market admits only profitable references; one relaxed filter from reachable, and exactly the floor house rule 3 forbids. Probe values unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 17:44:20 -07:00
kartiandClaude Opus 5	00d21025ff	grand-exchange-live: the stepped market engine, and the turn budget binds ... Arena's first interactive environment, as its own wheel beside grand-exchange so the one-shot scores stay comparable rather than conflated. One basket of orders becomes a loop: quote, see fills and the next tick, re-quote, with inventory, open orders and cash carrying between turns. The engine and its measurement only. No taskset, no config, no probe row — the spec's constants were derived from a throwaway engine and did not reconcile (impatient computed to 0.612 against a reported 0.623), so every constant here is chosen from a ladder re-measured in the code that ships. House rule 4 asks whether the turn budget binds. It does: exhaustive 0.631 vs oracle 1.000, gap 0.369, stdev 0.054 over five blocks, worst block 0.316, and 0.354-0.405 at four further seed bases. That is 18x the proposed margin and it clears the 0.25 quantum of the binary gate outright, so the comparison is not a cliff. The saturation worry does not fire: per-basket clipping is not clipping the mean, so exhaustive averages 0.849 on profit_ratio rather than 1.000. The first draft's gap was FAKE — a clean-looking 0.152 produced by an under-tuned denominator whose every-look sibling earned 22% more gp while scoring below it. It was caught only by sweeping the reference's own family. Nothing that scores below the oracle may earn more than it; that check belongs in every interactive environment that follows this one. freeze is derived from market.seed inside build_market so viability and scoring see one value, roc is on peak rather than average capital employed, and the reference is cached per seed so scoring never re-simulates the played episode. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 15:55:12 -07:00