arena

16 Commits 2 Branches 0 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
kartiandClaude Opus 5	122a858b61	pin verifiers 0.3.1.dev59, and make every config say subprocess The 600-second per-model-call ceiling that capped the thinking-on run at n=6 is gone: DEFAULT_TIMEOUT is read=None in all nine venvs. There is NO verifiers 0.4.0. The `version = "0.4.0"` in the vendored tree belongs to a [[tool.uv.dependency-metadata]] block for nemo-gym, and reading it as verifiers' own sends you hunting a release that does not exist and concluding the bump is blocked. The vendored checkout is v0.3.0-59-g4bcb48e5, published to PyPI as 0.3.1.dev59, and the installed wheel diffs clean against the whole vendored v1 tree. Pinned as an ordinary index dependency: a path source is reproducible only on this box and a git URL pins a commit no wheel matches. ⚠️ dev59 changes the default agent runtime from subprocess to `prime`, a REMOTE sandbox. Without an explicit runtime every eval here exits "not authenticated with prime" before a single rollout — having already created its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than an unauthenticated one. All nine configs now say subprocess. Nothing here needs a sandbox: the reward is a pure function of the trace and the harness is null. Output layout is flat now — <env>--<model>--<harness>--<hex8>/ with configs/, traces.jsonl and a logs/eval.log that 0.3.0 never wrote. `-o` is the output_dir, not the run dir; `--run.dir` pins the leaf. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 23:14:05 -07:00
karti	d691acf131	Harden redaction-pressure reward and provenance arena-environments / validate (3.12) (push) Successful in 31s Details arena-environments / validate (3.11) (push) Successful in 1m8s Details	2026-08-19 01:26:21 -07:00
kartiandClaude Opus 5	c28d864766	Arena: the environment repository, and redaction-pressure An environment is an eval you take the gradient of, so it carries obligations a benchmark does not: a held-out slice, a counterweighted reward, and a demonstrated 0.0 floor and 1.0 ceiling. README states the three as house rules. redaction-pressure is Forge's redact stage as a task. The first reward design scored survivors — secrets caught, decoys kept, prose kept — and an empty ruleset tied the best real attempt at 0.500, because destroying nothing keeps everything. Counting removals instead puts inaction at 0.000, shredding at 0.380, and an oracle at 1.000. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 00:09:31 -07:00

Author

SHA1

Message

Date

kartiandClaude Opus 5

122a858b61

pin verifiers 0.3.1.dev59, and make every config say subprocess

The 600-second per-model-call ceiling that capped the thinking-on run at n=6 is
gone: DEFAULT_TIMEOUT is read=None in all nine venvs.

There is NO verifiers 0.4.0. The `version = "0.4.0"` in the vendored tree belongs
to a [[tool.uv.dependency-metadata]] block for nemo-gym, and reading it as
verifiers' own sends you hunting a release that does not exist and concluding the
bump is blocked. The vendored checkout is v0.3.0-59-g4bcb48e5, published to PyPI
as 0.3.1.dev59, and the installed wheel diffs clean against the whole vendored
v1 tree. Pinned as an ordinary index dependency: a path source is reproducible
only on this box and a git URL pins a commit no wheel matches.

⚠️ dev59 changes the default agent runtime from subprocess to `prime`, a REMOTE
sandbox. Without an explicit runtime every eval here exits "not authenticated
with prime" before a single rollout — having already created its output directory
and a zero-line traces.jsonl, so it reads as a failed eval rather than an
unauthenticated one. All nine configs now say subprocess. Nothing here needs a
sandbox: the reward is a pure function of the trace and the harness is null.

Output layout is flat now — <env>--<model>--<harness>--<hex8>/ with configs/,
traces.jsonl and a logs/eval.log that 0.3.0 never wrote. `-o` is the output_dir,
not the run dir; `--run.dir` pins the leaf.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-21 23:14:05 -07:00

karti

d691acf131

Harden redaction-pressure reward and provenance

arena-environments / validate (3.12) (push) Successful in 31s

Details

arena-environments / validate (3.11) (push) Successful in 1m8s

Details

2026-08-19 01:26:21 -07:00

kartiandClaude Opus 5

c28d864766

Arena: the environment repository, and redaction-pressure

An environment is an eval you take the gradient of, so it carries obligations a
benchmark does not: a held-out slice, a counterweighted reward, and a demonstrated
0.0 floor and 1.0 ceiling. README states the three as house rules.

redaction-pressure is Forge's redact stage as a task. The first reward design scored
survivors — secrets caught, decoys kept, prose kept — and an empty ruleset tied the
best real attempt at 0.500, because destroying nothing keeps everything. Counting
removals instead puts inaction at 0.000, shredding at 0.380, and an oracle at 1.000.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-19 00:09:31 -07:00