Pin Arena environments to stable Verifiers
This commit is contained in:
+17
-18
@@ -7,8 +7,8 @@ training table — was waiting on a run that had never completed.
|
||||
|
||||
This is that run, written down so the next one is a copy-paste.
|
||||
|
||||
⚠️ **Updated 2026-08-21 evening.** The pin moved from `verifiers==0.3.0` to
|
||||
`verifiers==0.3.1.dev59`, which removes the 600-second model-call ceiling §5 blames for
|
||||
⚠️ **Updated 2026-08-25.** The pin is now the stable `verifiers==0.3.1`
|
||||
release published 2026-08-24. It removes the 600-second model-call ceiling §5 blames for
|
||||
every truncation here, and moves the output layout. §1 and §2 are the post-bump
|
||||
instructions; §4 and §5 are kept as written because they are the record of what the
|
||||
thinking-**off** run measured, and §6 is the thinking-**on** re-measurement the bump made
|
||||
@@ -50,8 +50,8 @@ see §5. The seven data environments have thinking-on copies in `outputs/thinkin
|
||||
| `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. |
|
||||
| `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. |
|
||||
| `-c 8` | spark-1 is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. |
|
||||
| `--env.agent.runtime.type subprocess` | ⚠️ **New, and not optional.** 0.3.0 defaulted the agent runtime to `subprocess`; `0.3.1.dev59` defaults it to `prime` — a *remote* sandbox. A remote role makes `Env._runs_local()` false, which makes the interception pool mint a Prime tunnel, which calls `ensure_prime_auth()` and raises `SystemExit: not authenticated with prime`. The run dies before a single rollout, having already created its output directory and an empty `traces.jsonl`. The null harness needs no runtime at all; naming a local one is the whole fix. |
|
||||
| `-o <dir>` + `--run.dir <name>` | `-o` is `output_dir` and the run writes to `output_dir / run.dir`. On 0.3.0 a bare `-o` gave you a flat directory; on dev59 it gives you `outputs/<your-dir>/<env>--<model>--<harness>--<hex8>/`. Pass `--run.dir` to name the leaf yourself. |
|
||||
| `--env.agent.runtime.type subprocess` | ⚠️ **New, and not optional.** 0.3.0 defaulted the agent runtime to `subprocess`; `0.3.1` defaults it to `prime` — a *remote* sandbox. A remote role makes `Env._runs_local()` false, which makes the interception pool mint a Prime tunnel, which calls `ensure_prime_auth()` and raises `SystemExit: not authenticated with prime`. The run dies before a single rollout, having already created its output directory and an empty `traces.jsonl`. The null harness needs no runtime at all; naming a local one is the whole fix. |
|
||||
| `-o <dir>` + `--run.dir <name>` | `-o` is `output_dir` and the run writes to `output_dir / run.dir`. On 0.3.0 a bare `-o` gave you a flat directory; on 0.3.1 it gives you `outputs/<your-dir>/<env>--<model>--<harness>--<hex8>/`. Pass `--run.dir` to name the leaf yourself. |
|
||||
|
||||
### Flags our own docs get wrong
|
||||
|
||||
@@ -80,18 +80,17 @@ earlier notes read off its `pyproject.toml:136` belongs to a `[[tool.uv.dependen
|
||||
block for **nemo-gym**, not to verifiers. Anyone who goes looking for `verifiers==0.4.0`
|
||||
will not find it.
|
||||
|
||||
What *is* published is the dev stream: PyPI carries 45 `0.3.1.devN` releases, one per
|
||||
merged commit. `0.3.1.dev59` was uploaded 2026-08-20 and is exactly the vendored checkout —
|
||||
`verifiers/v1/cli/output.py` extracted from the wheel is byte-identical to the file on
|
||||
disk, and `verifiers/v1/clients/base.py:12` reads:
|
||||
The stable `0.3.1` wheel was published to PyPI on 2026-08-24 after the dev stream.
|
||||
It contains the unbounded model-call timeout and output-layout changes this runbook
|
||||
requires; `verifiers/v1/clients/base.py` reads:
|
||||
|
||||
```python
|
||||
DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=None, write=None, pool=None)
|
||||
```
|
||||
|
||||
**So the pin is an ordinary index dependency, not a path or git source.** Every
|
||||
`environments/*/pyproject.toml` now reads `verifiers==0.3.1.dev59`. An exact `==` on a
|
||||
pre-release is enough for uv — no `--prerelease=allow`, no `[tool.uv.sources]`, no
|
||||
`environments/*/pyproject.toml` now reads `verifiers==0.3.1`. It is an ordinary
|
||||
stable index dependency — no `--prerelease=allow`, no `[tool.uv.sources]`, no
|
||||
`tool.uv` block, and nothing that resolves differently in CI than it does here. A path
|
||||
dependency on `~/vendor` would have been reproducible only on this box; a git URL would
|
||||
have pinned a commit no wheel matches. Neither was necessary.
|
||||
@@ -99,7 +98,7 @@ have pinned a commit no wheel matches. Neither was necessary.
|
||||
```
|
||||
Resolved 114 packages in 620ms
|
||||
- verifiers==0.3.0
|
||||
+ verifiers==0.3.1.dev59
|
||||
+ verifiers==0.3.1
|
||||
```
|
||||
|
||||
`uv sync` each environment after pulling. Verify the thing you bumped *for*, not the
|
||||
@@ -120,7 +119,7 @@ is empty), so the bump does not put a lock file into a commit. CI's lock-policy
|
||||
git, so it passes on a fresh checkout and would fail on a working tree that has been
|
||||
synced. Do not run it locally and conclude the repo is broken.
|
||||
|
||||
### The layout dev59 writes
|
||||
### The layout 0.3.1 writes
|
||||
|
||||
`verifiers/v1/cli/output.py` — `output_path()` is `config.output_dir / config.run.dir`,
|
||||
and `run.dir` defaults to `run.name`, which is
|
||||
@@ -136,7 +135,7 @@ outputs/<output-dir>/<env>--<model>--<harness>--<hex8>/
|
||||
|
||||
Three differences from 0.3.0 that an ingester or a script has to handle:
|
||||
|
||||
1. **The uuid leaf is gone.** 0.3.0 wrote `<env>--<model>--<harness>/<uuid>/`; dev59
|
||||
1. **The uuid leaf is gone.** 0.3.0 wrote `<env>--<model>--<harness>/<uuid>/`; 0.3.1
|
||||
writes one flat directory with an 8-hex suffix on the name. `-o` no longer flattens
|
||||
anything — it sets `output_dir`, and the run still makes its own leaf underneath.
|
||||
`--run.dir <name>` is what pins the leaf, and it is what `outputs/thinking-n32/run_all.sh`
|
||||
@@ -164,7 +163,7 @@ training-native` landing in this range. One Episode per line, not one Trace:
|
||||
```
|
||||
|
||||
`outputs/count_rewards.py`, written against 0.3.0, reproduces every published number from
|
||||
`run-20260821-1401` unchanged against a dev59 run. The two traps in that file are also
|
||||
`run-20260821-1401` unchanged against a 0.3.1 run. The two traps in that file are also
|
||||
unchanged, and both still bite an ingester:
|
||||
|
||||
1. **`Reward.value` is a `@property` and never appears in the JSON.** Each entry
|
||||
@@ -481,11 +480,11 @@ Three consequences for anyone re-running this suite against a reasoning model:
|
||||
makes the "grand-exchange is solvable" claim — currently resting on six rollouts — cheap
|
||||
to settle properly.
|
||||
|
||||
✅ **RETIRED 2026-08-21 evening.** The bump happened — `verifiers==0.3.1.dev59`, §2 — and
|
||||
✅ **RETIRED 2026-08-21 evening.** The bump happened — `verifiers==0.3.1`, §2 — and
|
||||
every venv on this box now reads `read=None`. One correction to the paragraph above, which
|
||||
matters because it is the version anyone will type: the fix is **not** `verifiers==0.4.0`.
|
||||
There is no such release. `a298bcfe` is `v0.3.0-8-ga298bcfe`; the vendored HEAD is
|
||||
`v0.3.0-59-g4bcb48e5`; the wheel that carries them is `0.3.1.dev59`. Everything below this
|
||||
`v0.3.0-59-g4bcb48e5`; the wheel that carries them is `0.3.1`. Everything below this
|
||||
line about 600 seconds describes the old pin and is kept as the record of why the bump was
|
||||
worth doing, not as current behaviour. The `max_tokens` / wall-clock contradiction it
|
||||
describes — "the two limits point in opposite directions and there is no setting that
|
||||
@@ -532,12 +531,12 @@ recorded.
|
||||
|
||||
## 6. The thinking-ON re-measurement, n=32
|
||||
|
||||
Run 2026-08-21 evening, after the `verifiers==0.3.1.dev59` bump of §2. This section
|
||||
Run 2026-08-21 evening, after the `verifiers==0.3.1` bump of §2. This section
|
||||
exists because two of §4's seven numbers were artefacts of a sampling configuration
|
||||
nobody chose and nothing recorded, and because §5's remedy — "an n=32 thinking-on run is
|
||||
achievable today" — was owed a run.
|
||||
|
||||
> **Sampling configuration for every number in this section.** `verifiers==0.3.1.dev59`
|
||||
> **Sampling configuration for every number in this section.** `verifiers==0.3.1`
|
||||
> (`read=None`); `chat_template_kwargs = {enable_thinking = true}`; `max_tokens = 49152`;
|
||||
> `temperature`, `top_p`, `reasoning_effort` unset; `-c 8`; harness `null`; agent runtime
|
||||
> `subprocess`; model `brain-qwen38-dspark` on spark-1. The configs are
|
||||
|
||||
Reference in New Issue
Block a user