4 Commits

Include renames

Author	SHA1	Message	Date
karti	61939676a5	Pin Arena environments to stable Verifiers	2026-08-25 10:33:35 -07:00
kartiandClaude Opus 5	122a858b61	pin verifiers 0.3.1.dev59, and make every config say subprocess ... The 600-second per-model-call ceiling that capped the thinking-on run at n=6 is gone: DEFAULT_TIMEOUT is read=None in all nine venvs. There is NO verifiers 0.4.0. The `version = "0.4.0"` in the vendored tree belongs to a [[tool.uv.dependency-metadata]] block for nemo-gym, and reading it as verifiers' own sends you hunting a release that does not exist and concluding the bump is blocked. The vendored checkout is v0.3.0-59-g4bcb48e5, published to PyPI as 0.3.1.dev59, and the installed wheel diffs clean against the whole vendored v1 tree. Pinned as an ordinary index dependency: a path source is reproducible only on this box and a git URL pins a commit no wheel matches. ⚠️ dev59 changes the default agent runtime from subprocess to `prime`, a REMOTE sandbox. Without an explicit runtime every eval here exits "not authenticated with prime" before a single rollout — having already created its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than an unauthenticated one. All nine configs now say subprocess. Nothing here needs a sandbox: the reward is a pure function of the trace and the harness is null. Output layout is flat now — <env>--<model>--<harness>--<hex8>/ with configs/, traces.jsonl and a logs/eval.log that 0.3.0 never wrote. `-o` is the output_dir, not the run dir; `--run.dir` pins the leaf. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 23:14:05 -07:00
karti	d691acf131	Harden redaction-pressure reward and provenance	2026-08-19 01:26:21 -07:00
kartiandClaude Opus 5	c28d864766	Arena: the environment repository, and redaction-pressure ... An environment is an eval you take the gradient of, so it carries obligations a benchmark does not: a held-out slice, a counterweighted reward, and a demonstrated 0.0 floor and 1.0 ceiling. README states the three as house rules. redaction-pressure is Forge's redact stage as a task. The first reward design scored survivors — secrets caught, decoys kept, prose kept — and an empty ruleset tied the best real attempt at 0.500, because destroying nothing keeps everything. Counting removals instead puts inaction at 0.000, shredding at 0.380, and an oracle at 1.000. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 00:09:31 -07:00