Pin Arena environments to stable Verifiers
arena-environments / validate (3.11) (push) Successful in 1m11s
arena-environments / validate (3.12) (push) Successful in 1m12s

This commit is contained in:
2026-08-25 10:33:35 -07:00
parent 3305be4ff7
commit 61939676a5
24 changed files with 87 additions and 41 deletions
+2 -2
View File
@@ -87,7 +87,7 @@ Corrected, blocking:
6. **`portal_host_samples` and `portal_container_samples` have no node identity** — no `node_id`, no host column, no PK. Per-node charts need `ALTER TABLE … ADD COLUMN IF NOT EXISTS node_id text` plus an index first. 6. **`portal_host_samples` and `portal_container_samples` have no node identity** — no `node_id`, no host column, no PK. Per-node charts need `ALTER TABLE … ADD COLUMN IF NOT EXISTS node_id text` plus an index first.
7. **`scored = ok AND rewards non-empty` is wrong.** `write_episode` dumps with `exclude_none=True`, so a `None` reward is dropped from the file entirely — "not measured" and "key absent" are indistinguishable on the wire. Require at least one non-null `Reward`, and carry the run's declared reward-name set from the config. 7. **`scored = ok AND rewards non-empty` is wrong.** `write_episode` dumps with `exclude_none=True`, so a `None` reward is dropped from the file entirely — "not measured" and "key absent" are indistinguishable on the wire. Require at least one non-null `Reward`, and carry the run's declared reward-name set from the config.
8. **`Reward.value` is a `@property`** and never appears in dumped JSON. Compute `score × weight` in the ingester. 8. **`Reward.value` is a `@property`** and never appears in dumped JSON. Compute `score × weight` in the ingester.
9. ⚠️ **CORRECTED 2026-08-21: there is NO verifiers 0.4.0.** The `version = "0.4.0"` at `~/vendor/prime-intellect/verifiers/pyproject.toml:136` belongs to a `[[tool.uv.dependency-metadata]]` block for **nemo-gym**, not to verifiers — read as verifiers' own version it sends you hunting a release that does not exist and concluding the bump is blocked. The vendored checkout is `v0.3.0-59-g4bcb48e5`, published to PyPI as **`0.3.1.dev59`**, byte-identical to the vendored tree, and that is what all nine manifests now pin. The layout warning itself is CORRECT: **0.3.0 → dev59 changes the directory layout, not just the config filename** — HEAD produces one flat `<env>--<model>--<harness>--<hex8>` dir with no nested uuid, and `RunConfig.id` is a `PrivateAttr` absent from `configs/eval.json`. Derive the run id from the directory basename. 9. ⚠️ **CORRECTED 2026-08-25: there is NO verifiers 0.4.0.** The `version = "0.4.0"` at `~/vendor/prime-intellect/verifiers/pyproject.toml:136` belongs to a `[[tool.uv.dependency-metadata]]` block for **nemo-gym**, not to verifiers. Stable **`verifiers==0.3.1`** was published 2026-08-24 and is what all nine manifests pin. The layout warning itself is correct: 0.3.1 produces one flat `<env>--<model>--<harness>--<hex8>` directory with no nested UUID, and `RunConfig.id` is a `PrivateAttr` absent from `configs/eval.json`. Derive the run id from the directory basename.
10. **Name the private console `/nodes`**, not `/compute``/compute` is already a public marketing section with its own nav. 10. **Name the private console `/nodes`**, not `/compute``/compute` is already a public marketing section with its own nav.
11. Put `isAppHost` in `packages/shared` — two independent string checks that must agree, with a silent asymmetric failure mode (dashboard rendered under a 404 status). 11. Put `isAppHost` in `packages/shared` — two independent string checks that must agree, with a silent asymmetric failure mode (dashboard rendered under a 404 status).
12. `sendShell(url, reply)` takes no request; the signature and both call sites change. `spa-fallback.test.ts` needs a case. `@radix-ui/react-select` is **not** installed. 12. `sendShell(url, reply)` takes no request; the signature and both call sites change. `spa-fallback.test.ts` needs a case. `@radix-ui/react-select` is **not** installed.
@@ -278,4 +278,4 @@ Two edges that were wrong in the source specs and are now correct:
**`environments/grand_exchange_live/grand_exchange_live/live.py` — the stepped engine and its unit tests — and, concurrently, workstream B1's discovery de-hardcoding.** **`environments/grand_exchange_live/grand_exchange_live/live.py` — the stepped engine and its unit tests — and, concurrently, workstream B1's discovery de-hardcoding.**
Everything else in the programme is downstream of two facts we do not yet have: whether the live market's reward ladder actually reconciles (risk 1, which invalidates A's horizon, freeze, TARGET_SHARE and turn budget if it does not, and takes B2, C1-C4 and half the compute argument with it), and whether three files can stop being a five-way merge conflict before five workstreams reach for them. B1 is an afternoon and removes the repo's only structural collision permanently; `live.py` is the smallest artifact that can turn six specs' worth of `/tmp` measurements into something the shipped code agrees with. Everything else in the programme is downstream of two facts we do not yet have: whether the live market's reward ladder actually reconciles (risk 1, which invalidates A's horizon, freeze, TARGET_SHARE and turn budget if it does not, and takes B2, C1-C4 and half the compute argument with it), and whether three files can stop being a five-way merge conflict before five workstreams reach for them. B1 is an afternoon and removes the repo's only structural collision permanently; `live.py` is the smallest artifact that can turn six specs' worth of `/tmp` measurements into something the shipped code agrees with.
+3 -2
View File
@@ -13,7 +13,8 @@ separable tiers, not one, and the cheapest tier needs no new infrastructure at a
## What upstream actually requires ## What upstream actually requires
Read from `~/vendor/prime-intellect/verifiers` (v1, installed version 0.3.0), not Read from stable `verifiers==0.3.1` and the matching checkout under
`~/vendor/prime-intellect/verifiers`, not
from blog posts. from blog posts.
**Multi-turn does not imply a container.** `vf.Env.run(task, agents)` programs the **Multi-turn does not imply a container.** `vf.Env.run(task, agents)` programs the
@@ -160,7 +161,7 @@ it is configured here.
## Preconditions, verified on this box ## Preconditions, verified on this box
- `uv run eval @ configs/canary_trap.toml --dry-run` → exit 0, config written ✓ - `uv run eval @ configs/canary_trap.toml --dry-run` → exit 0, config written ✓
- verifiers 0.3.0 with all 13 harnesses and 4 runtimes present ✓ - stable verifiers 0.3.1 with upstream harnesses and runtimes present ✓
- Docker 29.1.3 running (tier 2/3 runtime) ✓ - Docker 29.1.3 running (tier 2/3 runtime) ✓
- spark-1 `brain` on `http://100.127.247.67:8001/v1`, model id `brain` - spark-1 `brain` on `http://100.127.247.67:8001/v1`, model id `brain`
- `prime` CLI installed (0.6.25; 0.6.26 available) ✓ - `prime` CLI installed (0.6.25; 0.6.26 available) ✓
+1 -1
View File
@@ -216,7 +216,7 @@ files. Everything downstream of B fans out four ways.
## Preconditions, verified on amd-server ## Preconditions, verified on amd-server
- `uv run eval @ configs/<env>.toml --dry-run` → exit 0 ✓ - `uv run eval @ configs/<env>.toml --dry-run` → exit 0 ✓
- verifiers 0.3.0, all 13 harnesses and 4 runtimes present - stable verifiers 0.3.1, using its upstream `null` harness and runtimes ✓
- Docker 29.1.3 (tier 2/3 only; tier 1 needs none) ✓ - Docker 29.1.3 (tier 2/3 only; tier 1 needs none) ✓
- spark-1 `brain` at `http://100.127.247.67:8001/v1`, model id `brain` - spark-1 `brain` at `http://100.127.247.67:8001/v1`, model id `brain`
- `prime` CLI 0.6.25 ✓ - `prime` CLI 0.6.25 ✓
+1 -1
View File
@@ -12,7 +12,7 @@ graded = 12 # unlabelled accounts the answer is scored on; 9 is the floor
id = "null" # pure-trace reward: one completion, no tools, no runtime id = "null" # pure-trace reward: one completion, no tools, no runtime
[env.agent.runtime] [env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from # ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits # `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created # `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than # its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -11,7 +11,7 @@ id = "canary-trap"
id = "null" # pure-trace reward: one completion, no tools, no runtime id = "null" # pure-trace reward: one completion, no tools, no runtime
[env.agent.runtime] [env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from # ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits # `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created # `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than # its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -13,7 +13,7 @@ held_out_kills = 40000 # kills it is scored on
id = "null" # pure-trace reward: one completion, no tools, no runtime id = "null" # pure-trace reward: one completion, no tools, no runtime
[env.agent.runtime] [env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from # ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits # `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created # `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than # its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -11,7 +11,7 @@ id = "fault-localisation"
id = "null" # pure-trace reward: one completion, no tools, no runtime id = "null" # pure-trace reward: one completion, no tools, no runtime
[env.agent.runtime] [env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from # ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits # `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created # `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than # its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -14,7 +14,7 @@ held_out = 30 # ticks its orders are executed against
id = "null" # pure-trace reward: one completion, no tools, no runtime id = "null" # pure-trace reward: one completion, no tools, no runtime
[env.agent.runtime] [env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from # ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits # `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created # `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than # its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -23,7 +23,7 @@ visible = 56 # warmup ticks, read before the first look
id = "null" # pure-trace reward: a chat loop, no tools, no runtime id = "null" # pure-trace reward: a chat loop, no tools, no runtime
[env.agent.runtime] [env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from # ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits # `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created # `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than # its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -14,7 +14,7 @@ held_out = 12 # records it is scored on
id = "null" # a pure-trace reward: one completion, no tools, no runtime id = "null" # a pure-trace reward: one completion, no tools, no runtime
[env.agent.runtime] [env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from # ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits # `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created # `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than # its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -11,7 +11,7 @@ id = "schema-migration"
id = "null" # pure-trace reward: one completion, no tools, no runtime id = "null" # pure-trace reward: one completion, no tools, no runtime
[env.agent.runtime] [env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from # ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits # `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created # `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than # its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -25,7 +25,7 @@ seed_base = 115
id = "null" # one completion per turn, no tools; the env programs the loop id = "null" # one completion per turn, no tools; the env programs the loop
[env.agent.runtime] [env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from # ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits # `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created # `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than # its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+17 -18
View File
@@ -7,8 +7,8 @@ training table — was waiting on a run that had never completed.
This is that run, written down so the next one is a copy-paste. This is that run, written down so the next one is a copy-paste.
⚠️ **Updated 2026-08-21 evening.** The pin moved from `verifiers==0.3.0` to ⚠️ **Updated 2026-08-25.** The pin is now the stable `verifiers==0.3.1`
`verifiers==0.3.1.dev59`, which removes the 600-second model-call ceiling §5 blames for release published 2026-08-24. It removes the 600-second model-call ceiling §5 blames for
every truncation here, and moves the output layout. §1 and §2 are the post-bump every truncation here, and moves the output layout. §1 and §2 are the post-bump
instructions; §4 and §5 are kept as written because they are the record of what the instructions; §4 and §5 are kept as written because they are the record of what the
thinking-**off** run measured, and §6 is the thinking-**on** re-measurement the bump made thinking-**off** run measured, and §6 is the thinking-**on** re-measurement the bump made
@@ -50,8 +50,8 @@ see §5. The seven data environments have thinking-on copies in `outputs/thinkin
| `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. | | `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. |
| `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. | | `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. |
| `-c 8` | spark-1 is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. | | `-c 8` | spark-1 is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. |
| `--env.agent.runtime.type subprocess` | ⚠️ **New, and not optional.** 0.3.0 defaulted the agent runtime to `subprocess`; `0.3.1.dev59` defaults it to `prime` — a *remote* sandbox. A remote role makes `Env._runs_local()` false, which makes the interception pool mint a Prime tunnel, which calls `ensure_prime_auth()` and raises `SystemExit: not authenticated with prime`. The run dies before a single rollout, having already created its output directory and an empty `traces.jsonl`. The null harness needs no runtime at all; naming a local one is the whole fix. | | `--env.agent.runtime.type subprocess` | ⚠️ **New, and not optional.** 0.3.0 defaulted the agent runtime to `subprocess`; `0.3.1` defaults it to `prime` — a *remote* sandbox. A remote role makes `Env._runs_local()` false, which makes the interception pool mint a Prime tunnel, which calls `ensure_prime_auth()` and raises `SystemExit: not authenticated with prime`. The run dies before a single rollout, having already created its output directory and an empty `traces.jsonl`. The null harness needs no runtime at all; naming a local one is the whole fix. |
| `-o <dir>` + `--run.dir <name>` | `-o` is `output_dir` and the run writes to `output_dir / run.dir`. On 0.3.0 a bare `-o` gave you a flat directory; on dev59 it gives you `outputs/<your-dir>/<env>--<model>--<harness>--<hex8>/`. Pass `--run.dir` to name the leaf yourself. | | `-o <dir>` + `--run.dir <name>` | `-o` is `output_dir` and the run writes to `output_dir / run.dir`. On 0.3.0 a bare `-o` gave you a flat directory; on 0.3.1 it gives you `outputs/<your-dir>/<env>--<model>--<harness>--<hex8>/`. Pass `--run.dir` to name the leaf yourself. |
### Flags our own docs get wrong ### Flags our own docs get wrong
@@ -80,18 +80,17 @@ earlier notes read off its `pyproject.toml:136` belongs to a `[[tool.uv.dependen
block for **nemo-gym**, not to verifiers. Anyone who goes looking for `verifiers==0.4.0` block for **nemo-gym**, not to verifiers. Anyone who goes looking for `verifiers==0.4.0`
will not find it. will not find it.
What *is* published is the dev stream: PyPI carries 45 `0.3.1.devN` releases, one per The stable `0.3.1` wheel was published to PyPI on 2026-08-24 after the dev stream.
merged commit. `0.3.1.dev59` was uploaded 2026-08-20 and is exactly the vendored checkout — It contains the unbounded model-call timeout and output-layout changes this runbook
`verifiers/v1/cli/output.py` extracted from the wheel is byte-identical to the file on requires; `verifiers/v1/clients/base.py` reads:
disk, and `verifiers/v1/clients/base.py:12` reads:
```python ```python
DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=None, write=None, pool=None) DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=None, write=None, pool=None)
``` ```
**So the pin is an ordinary index dependency, not a path or git source.** Every **So the pin is an ordinary index dependency, not a path or git source.** Every
`environments/*/pyproject.toml` now reads `verifiers==0.3.1.dev59`. An exact `==` on a `environments/*/pyproject.toml` now reads `verifiers==0.3.1`. It is an ordinary
pre-release is enough for uv — no `--prerelease=allow`, no `[tool.uv.sources]`, no stable index dependency — no `--prerelease=allow`, no `[tool.uv.sources]`, no
`tool.uv` block, and nothing that resolves differently in CI than it does here. A path `tool.uv` block, and nothing that resolves differently in CI than it does here. A path
dependency on `~/vendor` would have been reproducible only on this box; a git URL would dependency on `~/vendor` would have been reproducible only on this box; a git URL would
have pinned a commit no wheel matches. Neither was necessary. have pinned a commit no wheel matches. Neither was necessary.
@@ -99,7 +98,7 @@ have pinned a commit no wheel matches. Neither was necessary.
``` ```
Resolved 114 packages in 620ms Resolved 114 packages in 620ms
- verifiers==0.3.0 - verifiers==0.3.0
+ verifiers==0.3.1.dev59 + verifiers==0.3.1
``` ```
`uv sync` each environment after pulling. Verify the thing you bumped *for*, not the `uv sync` each environment after pulling. Verify the thing you bumped *for*, not the
@@ -120,7 +119,7 @@ is empty), so the bump does not put a lock file into a commit. CI's lock-policy
git, so it passes on a fresh checkout and would fail on a working tree that has been git, so it passes on a fresh checkout and would fail on a working tree that has been
synced. Do not run it locally and conclude the repo is broken. synced. Do not run it locally and conclude the repo is broken.
### The layout dev59 writes ### The layout 0.3.1 writes
`verifiers/v1/cli/output.py``output_path()` is `config.output_dir / config.run.dir`, `verifiers/v1/cli/output.py``output_path()` is `config.output_dir / config.run.dir`,
and `run.dir` defaults to `run.name`, which is and `run.dir` defaults to `run.name`, which is
@@ -136,7 +135,7 @@ outputs/<output-dir>/<env>--<model>--<harness>--<hex8>/
Three differences from 0.3.0 that an ingester or a script has to handle: Three differences from 0.3.0 that an ingester or a script has to handle:
1. **The uuid leaf is gone.** 0.3.0 wrote `<env>--<model>--<harness>/<uuid>/`; dev59 1. **The uuid leaf is gone.** 0.3.0 wrote `<env>--<model>--<harness>/<uuid>/`; 0.3.1
writes one flat directory with an 8-hex suffix on the name. `-o` no longer flattens writes one flat directory with an 8-hex suffix on the name. `-o` no longer flattens
anything — it sets `output_dir`, and the run still makes its own leaf underneath. anything — it sets `output_dir`, and the run still makes its own leaf underneath.
`--run.dir <name>` is what pins the leaf, and it is what `outputs/thinking-n32/run_all.sh` `--run.dir <name>` is what pins the leaf, and it is what `outputs/thinking-n32/run_all.sh`
@@ -164,7 +163,7 @@ training-native` landing in this range. One Episode per line, not one Trace:
``` ```
`outputs/count_rewards.py`, written against 0.3.0, reproduces every published number from `outputs/count_rewards.py`, written against 0.3.0, reproduces every published number from
`run-20260821-1401` unchanged against a dev59 run. The two traps in that file are also `run-20260821-1401` unchanged against a 0.3.1 run. The two traps in that file are also
unchanged, and both still bite an ingester: unchanged, and both still bite an ingester:
1. **`Reward.value` is a `@property` and never appears in the JSON.** Each entry 1. **`Reward.value` is a `@property` and never appears in the JSON.** Each entry
@@ -481,11 +480,11 @@ Three consequences for anyone re-running this suite against a reasoning model:
makes the "grand-exchange is solvable" claim — currently resting on six rollouts — cheap makes the "grand-exchange is solvable" claim — currently resting on six rollouts — cheap
to settle properly. to settle properly.
**RETIRED 2026-08-21 evening.** The bump happened — `verifiers==0.3.1.dev59`, §2 — and **RETIRED 2026-08-21 evening.** The bump happened — `verifiers==0.3.1`, §2 — and
every venv on this box now reads `read=None`. One correction to the paragraph above, which every venv on this box now reads `read=None`. One correction to the paragraph above, which
matters because it is the version anyone will type: the fix is **not** `verifiers==0.4.0`. matters because it is the version anyone will type: the fix is **not** `verifiers==0.4.0`.
There is no such release. `a298bcfe` is `v0.3.0-8-ga298bcfe`; the vendored HEAD is There is no such release. `a298bcfe` is `v0.3.0-8-ga298bcfe`; the vendored HEAD is
`v0.3.0-59-g4bcb48e5`; the wheel that carries them is `0.3.1.dev59`. Everything below this `v0.3.0-59-g4bcb48e5`; the wheel that carries them is `0.3.1`. Everything below this
line about 600 seconds describes the old pin and is kept as the record of why the bump was line about 600 seconds describes the old pin and is kept as the record of why the bump was
worth doing, not as current behaviour. The `max_tokens` / wall-clock contradiction it worth doing, not as current behaviour. The `max_tokens` / wall-clock contradiction it
describes — "the two limits point in opposite directions and there is no setting that describes — "the two limits point in opposite directions and there is no setting that
@@ -532,12 +531,12 @@ recorded.
## 6. The thinking-ON re-measurement, n=32 ## 6. The thinking-ON re-measurement, n=32
Run 2026-08-21 evening, after the `verifiers==0.3.1.dev59` bump of §2. This section Run 2026-08-21 evening, after the `verifiers==0.3.1` bump of §2. This section
exists because two of §4's seven numbers were artefacts of a sampling configuration exists because two of §4's seven numbers were artefacts of a sampling configuration
nobody chose and nothing recorded, and because §5's remedy — "an n=32 thinking-on run is nobody chose and nothing recorded, and because §5's remedy — "an n=32 thinking-on run is
achievable today" — was owed a run. achievable today" — was owed a run.
> **Sampling configuration for every number in this section.** `verifiers==0.3.1.dev59` > **Sampling configuration for every number in this section.** `verifiers==0.3.1`
> (`read=None`); `chat_template_kwargs = {enable_thinking = true}`; `max_tokens = 49152`; > (`read=None`); `chat_template_kwargs = {enable_thinking = true}`; `max_tokens = 49152`;
> `temperature`, `top_p`, `reasoning_effort` unset; `-c 8`; harness `null`; agent runtime > `temperature`, `top_p`, `reasoning_effort` unset; `-c 8`; harness `null`; agent runtime
> `subprocess`; model `brain-qwen38-dspark` on spark-1. The configs are > `subprocess`; model `brain-qwen38-dspark` on spark-1. The configs are
+22
View File
@@ -0,0 +1,22 @@
# Prime-first boundary
Arena publishes Lumbridge environments; it does not implement another generic
harness, trainer, runtime marketplace, or environment hub.
- Every environment pins stable `verifiers==0.3.1`.
- Pure-trace environments select Verifiers' upstream `null` harness.
- Local model-free and spark-1 runs name the `subprocess` runtime explicitly so
tests never provision a paid Prime sandbox.
- Prime Hub is the public distribution target for approved environment wheels.
- Prime-RL consumes the same taskset packages for training; Forge owns the
explicit training handoff and artifact rendering.
Lumbridge's unique work is the environment: task distributions, interactive
state, rewards, probes, held-out gates, Tera bridges, and result interpretation.
Generic rollout and training machinery stays upstream. A Hub publication must
exclude secrets, private task splits, customer data, and proprietary Tera source.
The preferred future compute path is Prime's spot market. GH200/GB300 names in
runbooks are desired hardware profiles, not a promise that inventory exists and
not authorization to rent it. An operator checks current Prime availability and
price before launching any paid run.
+1 -1
View File
@@ -3,7 +3,7 @@ name = "bot-detection"
version = "0.1.0" version = "0.1.0"
description = "bot-detection — find the scripts in an activity log without banning the players who grind like one." description = "bot-detection — find the scripts in an activity log without banning the players who grind like one."
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"] dependencies = ["verifiers==0.3.1"]
[build-system] [build-system]
requires = ["hatchling"] requires = ["hatchling"]
+1 -1
View File
@@ -3,7 +3,7 @@ name = "canary-trap"
version = "0.1.0" version = "0.1.0"
description = "canary-trap — write contamination probes that survive a paraphrase, without accusing a clean model." description = "canary-trap — write contamination probes that survive a paraphrase, without accusing a clean model."
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"] dependencies = ["verifiers==0.3.1"]
[build-system] [build-system]
requires = ["hatchling"] requires = ["hatchling"]
@@ -3,7 +3,7 @@ name = "drop-table-inference"
version = "0.1.0" version = "0.1.0"
description = "drop-table-inference — estimate a drop table from a kill log, graded on the kills you never saw." description = "drop-table-inference — estimate a drop table from a kill log, graded on the kills you never saw."
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"] dependencies = ["verifiers==0.3.1"]
[build-system] [build-system]
requires = ["hatchling"] requires = ["hatchling"]
@@ -3,7 +3,7 @@ name = "fault-localisation"
version = "0.1.0" version = "0.1.0"
description = "fault-localisation — name the service that broke, not the ones complaining about it." description = "fault-localisation — name the service that broke, not the ones complaining about it."
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"] dependencies = ["verifiers==0.3.1"]
[build-system] [build-system]
requires = ["hatchling"] requires = ["hatchling"]
+1 -1
View File
@@ -3,7 +3,7 @@ name = "grand-exchange"
version = "0.1.0" version = "0.1.0"
description = "grand-exchange — place orders executed against a market window you never saw." description = "grand-exchange — place orders executed against a market window you never saw."
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"] dependencies = ["verifiers==0.3.1"]
[build-system] [build-system]
requires = ["hatchling"] requires = ["hatchling"]
@@ -3,7 +3,7 @@ name = "grand-exchange-live"
version = "0.1.0" version = "0.1.0"
description = "grand-exchange-live — quote, watch the tape, re-quote, over a turn budget." description = "grand-exchange-live — quote, watch the tape, re-quote, over a turn budget."
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"] dependencies = ["verifiers==0.3.1"]
[build-system] [build-system]
requires = ["hatchling"] requires = ["hatchling"]
@@ -6,7 +6,7 @@ requires-python = ">=3.11"
dependencies = [ dependencies = [
"pydantic==2.13.4", "pydantic==2.13.4",
"regex==2026.7.19", "regex==2026.7.19",
"verifiers==0.3.1.dev59", "verifiers==0.3.1",
] ]
[build-system] [build-system]
+1 -1
View File
@@ -3,7 +3,7 @@ name = "schema-migration"
version = "0.1.0" version = "0.1.0"
description = "schema-migration — split a text column into a number and a unit, graded on rows you never saw." description = "schema-migration — split a text column into a number and a unit, graded on rows you never saw."
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"] dependencies = ["verifiers==0.3.1"]
[build-system] [build-system]
requires = ["hatchling"] requires = ["hatchling"]
+1 -1
View File
@@ -3,7 +3,7 @@ name = "tera-spatial"
version = "0.1.0" version = "0.1.0"
description = "tera-spatial — Arena's bridge to Tera's four renderer-independent spatial environments, executed in TypeScript and graded in Python, and the tasksets that sit on it." description = "tera-spatial — Arena's bridge to Tera's four renderer-independent spatial environments, executed in TypeScript and graded in Python, and the tasksets that sit on it."
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"] dependencies = ["verifiers==0.3.1"]
# One directory, several tasksets: the environments share a vendored simulator and # One directory, several tasksets: the environments share a vendored simulator and
# a single worker, so splitting them into four wheels would ship the same 283 KB # a single worker, so splitting them into four wheels would ship the same 283 KB
+24
View File
@@ -74,6 +74,30 @@ def discovered_tasksets() -> set[str]:
class RootProbeIntegrationTests(unittest.TestCase): class RootProbeIntegrationTests(unittest.TestCase):
def test_prime_verifiers_boundary_is_pinned_and_upstream_owned(self) -> None:
manifests = sorted((ROOT / "environments").glob("*/pyproject.toml"))
configs = sorted((ROOT / "configs").glob("*.toml"))
self.assertTrue(manifests)
self.assertTrue(configs)
for manifest in manifests:
with self.subTest(manifest=manifest.parent.name):
data = tomllib.loads(manifest.read_text(encoding="utf-8"))
self.assertIn("verifiers==0.3.1", data["project"]["dependencies"])
for config in configs:
with self.subTest(config=config.stem):
data = tomllib.loads(config.read_text(encoding="utf-8"))
agent = data["env"]["agent"]
self.assertEqual(agent["harness"]["id"], "null")
self.assertEqual(agent["runtime"]["type"], "subprocess")
self.assertEqual(
list((ROOT / "environments").glob("**/harness.py")),
[],
msg="Arena environments must use Verifiers harnesses, not ship a parallel one",
)
def test_public_root_probe_gates_every_environment(self) -> None: def test_public_root_probe_gates_every_environment(self) -> None:
completed = subprocess.run( completed = subprocess.run(
[sys.executable, "probe.py"], [sys.executable, "probe.py"],