Pin Arena environments to stable Verifiers
arena-environments / validate (3.11) (push) Successful in 1m11s
arena-environments / validate (3.12) (push) Successful in 1m12s

This commit is contained in:
2026-08-25 10:33:35 -07:00
parent 3305be4ff7
commit 61939676a5
24 changed files with 87 additions and 41 deletions
+2 -2
View File
@@ -87,7 +87,7 @@ Corrected, blocking:
6. **`portal_host_samples` and `portal_container_samples` have no node identity** — no `node_id`, no host column, no PK. Per-node charts need `ALTER TABLE … ADD COLUMN IF NOT EXISTS node_id text` plus an index first.
7. **`scored = ok AND rewards non-empty` is wrong.** `write_episode` dumps with `exclude_none=True`, so a `None` reward is dropped from the file entirely — "not measured" and "key absent" are indistinguishable on the wire. Require at least one non-null `Reward`, and carry the run's declared reward-name set from the config.
8. **`Reward.value` is a `@property`** and never appears in dumped JSON. Compute `score × weight` in the ingester.
9. ⚠️ **CORRECTED 2026-08-21: there is NO verifiers 0.4.0.** The `version = "0.4.0"` at `~/vendor/prime-intellect/verifiers/pyproject.toml:136` belongs to a `[[tool.uv.dependency-metadata]]` block for **nemo-gym**, not to verifiers — read as verifiers' own version it sends you hunting a release that does not exist and concluding the bump is blocked. The vendored checkout is `v0.3.0-59-g4bcb48e5`, published to PyPI as **`0.3.1.dev59`**, byte-identical to the vendored tree, and that is what all nine manifests now pin. The layout warning itself is CORRECT: **0.3.0 → dev59 changes the directory layout, not just the config filename** — HEAD produces one flat `<env>--<model>--<harness>--<hex8>` dir with no nested uuid, and `RunConfig.id` is a `PrivateAttr` absent from `configs/eval.json`. Derive the run id from the directory basename.
9. ⚠️ **CORRECTED 2026-08-25: there is NO verifiers 0.4.0.** The `version = "0.4.0"` at `~/vendor/prime-intellect/verifiers/pyproject.toml:136` belongs to a `[[tool.uv.dependency-metadata]]` block for **nemo-gym**, not to verifiers. Stable **`verifiers==0.3.1`** was published 2026-08-24 and is what all nine manifests pin. The layout warning itself is correct: 0.3.1 produces one flat `<env>--<model>--<harness>--<hex8>` directory with no nested UUID, and `RunConfig.id` is a `PrivateAttr` absent from `configs/eval.json`. Derive the run id from the directory basename.
10. **Name the private console `/nodes`**, not `/compute``/compute` is already a public marketing section with its own nav.
11. Put `isAppHost` in `packages/shared` — two independent string checks that must agree, with a silent asymmetric failure mode (dashboard rendered under a 404 status).
12. `sendShell(url, reply)` takes no request; the signature and both call sites change. `spa-fallback.test.ts` needs a case. `@radix-ui/react-select` is **not** installed.
@@ -278,4 +278,4 @@ Two edges that were wrong in the source specs and are now correct:
**`environments/grand_exchange_live/grand_exchange_live/live.py` — the stepped engine and its unit tests — and, concurrently, workstream B1's discovery de-hardcoding.**
Everything else in the programme is downstream of two facts we do not yet have: whether the live market's reward ladder actually reconciles (risk 1, which invalidates A's horizon, freeze, TARGET_SHARE and turn budget if it does not, and takes B2, C1-C4 and half the compute argument with it), and whether three files can stop being a five-way merge conflict before five workstreams reach for them. B1 is an afternoon and removes the repo's only structural collision permanently; `live.py` is the smallest artifact that can turn six specs' worth of `/tmp` measurements into something the shipped code agrees with.
Everything else in the programme is downstream of two facts we do not yet have: whether the live market's reward ladder actually reconciles (risk 1, which invalidates A's horizon, freeze, TARGET_SHARE and turn budget if it does not, and takes B2, C1-C4 and half the compute argument with it), and whether three files can stop being a five-way merge conflict before five workstreams reach for them. B1 is an afternoon and removes the repo's only structural collision permanently; `live.py` is the smallest artifact that can turn six specs' worth of `/tmp` measurements into something the shipped code agrees with.
+3 -2
View File
@@ -13,7 +13,8 @@ separable tiers, not one, and the cheapest tier needs no new infrastructure at a
## What upstream actually requires
Read from `~/vendor/prime-intellect/verifiers` (v1, installed version 0.3.0), not
Read from stable `verifiers==0.3.1` and the matching checkout under
`~/vendor/prime-intellect/verifiers`, not
from blog posts.
**Multi-turn does not imply a container.** `vf.Env.run(task, agents)` programs the
@@ -160,7 +161,7 @@ it is configured here.
## Preconditions, verified on this box
- `uv run eval @ configs/canary_trap.toml --dry-run` → exit 0, config written ✓
- verifiers 0.3.0 with all 13 harnesses and 4 runtimes present ✓
- stable verifiers 0.3.1 with upstream harnesses and runtimes present ✓
- Docker 29.1.3 running (tier 2/3 runtime) ✓
- spark-1 `brain` on `http://100.127.247.67:8001/v1`, model id `brain`
- `prime` CLI installed (0.6.25; 0.6.26 available) ✓
+1 -1
View File
@@ -216,7 +216,7 @@ files. Everything downstream of B fans out four ways.
## Preconditions, verified on amd-server
- `uv run eval @ configs/<env>.toml --dry-run` → exit 0 ✓
- verifiers 0.3.0, all 13 harnesses and 4 runtimes present
- stable verifiers 0.3.1, using its upstream `null` harness and runtimes ✓
- Docker 29.1.3 (tier 2/3 only; tier 1 needs none) ✓
- spark-1 `brain` at `http://100.127.247.67:8001/v1`, model id `brain`
- `prime` CLI 0.6.25 ✓
+1 -1
View File
@@ -12,7 +12,7 @@ graded = 12 # unlabelled accounts the answer is scored on; 9 is the floor
id = "null" # pure-trace reward: one completion, no tools, no runtime
[env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -11,7 +11,7 @@ id = "canary-trap"
id = "null" # pure-trace reward: one completion, no tools, no runtime
[env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -13,7 +13,7 @@ held_out_kills = 40000 # kills it is scored on
id = "null" # pure-trace reward: one completion, no tools, no runtime
[env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -11,7 +11,7 @@ id = "fault-localisation"
id = "null" # pure-trace reward: one completion, no tools, no runtime
[env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -14,7 +14,7 @@ held_out = 30 # ticks its orders are executed against
id = "null" # pure-trace reward: one completion, no tools, no runtime
[env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -23,7 +23,7 @@ visible = 56 # warmup ticks, read before the first look
id = "null" # pure-trace reward: a chat loop, no tools, no runtime
[env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -14,7 +14,7 @@ held_out = 12 # records it is scored on
id = "null" # a pure-trace reward: one completion, no tools, no runtime
[env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -11,7 +11,7 @@ id = "schema-migration"
id = "null" # pure-trace reward: one completion, no tools, no runtime
[env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+1 -1
View File
@@ -25,7 +25,7 @@ seed_base = 115
id = "null" # one completion per turn, no tools; the env programs the loop
[env.agent.runtime]
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
# `SystemExit: not authenticated with prime` before a single rollout, having already created
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
+17 -18
View File
@@ -7,8 +7,8 @@ training table — was waiting on a run that had never completed.
This is that run, written down so the next one is a copy-paste.
⚠️ **Updated 2026-08-21 evening.** The pin moved from `verifiers==0.3.0` to
`verifiers==0.3.1.dev59`, which removes the 600-second model-call ceiling §5 blames for
⚠️ **Updated 2026-08-25.** The pin is now the stable `verifiers==0.3.1`
release published 2026-08-24. It removes the 600-second model-call ceiling §5 blames for
every truncation here, and moves the output layout. §1 and §2 are the post-bump
instructions; §4 and §5 are kept as written because they are the record of what the
thinking-**off** run measured, and §6 is the thinking-**on** re-measurement the bump made
@@ -50,8 +50,8 @@ see §5. The seven data environments have thinking-on copies in `outputs/thinkin
| `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. |
| `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. |
| `-c 8` | spark-1 is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. |
| `--env.agent.runtime.type subprocess` | ⚠️ **New, and not optional.** 0.3.0 defaulted the agent runtime to `subprocess`; `0.3.1.dev59` defaults it to `prime` — a *remote* sandbox. A remote role makes `Env._runs_local()` false, which makes the interception pool mint a Prime tunnel, which calls `ensure_prime_auth()` and raises `SystemExit: not authenticated with prime`. The run dies before a single rollout, having already created its output directory and an empty `traces.jsonl`. The null harness needs no runtime at all; naming a local one is the whole fix. |
| `-o <dir>` + `--run.dir <name>` | `-o` is `output_dir` and the run writes to `output_dir / run.dir`. On 0.3.0 a bare `-o` gave you a flat directory; on dev59 it gives you `outputs/<your-dir>/<env>--<model>--<harness>--<hex8>/`. Pass `--run.dir` to name the leaf yourself. |
| `--env.agent.runtime.type subprocess` | ⚠️ **New, and not optional.** 0.3.0 defaulted the agent runtime to `subprocess`; `0.3.1` defaults it to `prime` — a *remote* sandbox. A remote role makes `Env._runs_local()` false, which makes the interception pool mint a Prime tunnel, which calls `ensure_prime_auth()` and raises `SystemExit: not authenticated with prime`. The run dies before a single rollout, having already created its output directory and an empty `traces.jsonl`. The null harness needs no runtime at all; naming a local one is the whole fix. |
| `-o <dir>` + `--run.dir <name>` | `-o` is `output_dir` and the run writes to `output_dir / run.dir`. On 0.3.0 a bare `-o` gave you a flat directory; on 0.3.1 it gives you `outputs/<your-dir>/<env>--<model>--<harness>--<hex8>/`. Pass `--run.dir` to name the leaf yourself. |
### Flags our own docs get wrong
@@ -80,18 +80,17 @@ earlier notes read off its `pyproject.toml:136` belongs to a `[[tool.uv.dependen
block for **nemo-gym**, not to verifiers. Anyone who goes looking for `verifiers==0.4.0`
will not find it.
What *is* published is the dev stream: PyPI carries 45 `0.3.1.devN` releases, one per
merged commit. `0.3.1.dev59` was uploaded 2026-08-20 and is exactly the vendored checkout —
`verifiers/v1/cli/output.py` extracted from the wheel is byte-identical to the file on
disk, and `verifiers/v1/clients/base.py:12` reads:
The stable `0.3.1` wheel was published to PyPI on 2026-08-24 after the dev stream.
It contains the unbounded model-call timeout and output-layout changes this runbook
requires; `verifiers/v1/clients/base.py` reads:
```python
DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=None, write=None, pool=None)
```
**So the pin is an ordinary index dependency, not a path or git source.** Every
`environments/*/pyproject.toml` now reads `verifiers==0.3.1.dev59`. An exact `==` on a
pre-release is enough for uv — no `--prerelease=allow`, no `[tool.uv.sources]`, no
`environments/*/pyproject.toml` now reads `verifiers==0.3.1`. It is an ordinary
stable index dependency — no `--prerelease=allow`, no `[tool.uv.sources]`, no
`tool.uv` block, and nothing that resolves differently in CI than it does here. A path
dependency on `~/vendor` would have been reproducible only on this box; a git URL would
have pinned a commit no wheel matches. Neither was necessary.
@@ -99,7 +98,7 @@ have pinned a commit no wheel matches. Neither was necessary.
```
Resolved 114 packages in 620ms
- verifiers==0.3.0
+ verifiers==0.3.1.dev59
+ verifiers==0.3.1
```
`uv sync` each environment after pulling. Verify the thing you bumped *for*, not the
@@ -120,7 +119,7 @@ is empty), so the bump does not put a lock file into a commit. CI's lock-policy
git, so it passes on a fresh checkout and would fail on a working tree that has been
synced. Do not run it locally and conclude the repo is broken.
### The layout dev59 writes
### The layout 0.3.1 writes
`verifiers/v1/cli/output.py``output_path()` is `config.output_dir / config.run.dir`,
and `run.dir` defaults to `run.name`, which is
@@ -136,7 +135,7 @@ outputs/<output-dir>/<env>--<model>--<harness>--<hex8>/
Three differences from 0.3.0 that an ingester or a script has to handle:
1. **The uuid leaf is gone.** 0.3.0 wrote `<env>--<model>--<harness>/<uuid>/`; dev59
1. **The uuid leaf is gone.** 0.3.0 wrote `<env>--<model>--<harness>/<uuid>/`; 0.3.1
writes one flat directory with an 8-hex suffix on the name. `-o` no longer flattens
anything — it sets `output_dir`, and the run still makes its own leaf underneath.
`--run.dir <name>` is what pins the leaf, and it is what `outputs/thinking-n32/run_all.sh`
@@ -164,7 +163,7 @@ training-native` landing in this range. One Episode per line, not one Trace:
```
`outputs/count_rewards.py`, written against 0.3.0, reproduces every published number from
`run-20260821-1401` unchanged against a dev59 run. The two traps in that file are also
`run-20260821-1401` unchanged against a 0.3.1 run. The two traps in that file are also
unchanged, and both still bite an ingester:
1. **`Reward.value` is a `@property` and never appears in the JSON.** Each entry
@@ -481,11 +480,11 @@ Three consequences for anyone re-running this suite against a reasoning model:
makes the "grand-exchange is solvable" claim — currently resting on six rollouts — cheap
to settle properly.
**RETIRED 2026-08-21 evening.** The bump happened — `verifiers==0.3.1.dev59`, §2 — and
**RETIRED 2026-08-21 evening.** The bump happened — `verifiers==0.3.1`, §2 — and
every venv on this box now reads `read=None`. One correction to the paragraph above, which
matters because it is the version anyone will type: the fix is **not** `verifiers==0.4.0`.
There is no such release. `a298bcfe` is `v0.3.0-8-ga298bcfe`; the vendored HEAD is
`v0.3.0-59-g4bcb48e5`; the wheel that carries them is `0.3.1.dev59`. Everything below this
`v0.3.0-59-g4bcb48e5`; the wheel that carries them is `0.3.1`. Everything below this
line about 600 seconds describes the old pin and is kept as the record of why the bump was
worth doing, not as current behaviour. The `max_tokens` / wall-clock contradiction it
describes — "the two limits point in opposite directions and there is no setting that
@@ -532,12 +531,12 @@ recorded.
## 6. The thinking-ON re-measurement, n=32
Run 2026-08-21 evening, after the `verifiers==0.3.1.dev59` bump of §2. This section
Run 2026-08-21 evening, after the `verifiers==0.3.1` bump of §2. This section
exists because two of §4's seven numbers were artefacts of a sampling configuration
nobody chose and nothing recorded, and because §5's remedy — "an n=32 thinking-on run is
achievable today" — was owed a run.
> **Sampling configuration for every number in this section.** `verifiers==0.3.1.dev59`
> **Sampling configuration for every number in this section.** `verifiers==0.3.1`
> (`read=None`); `chat_template_kwargs = {enable_thinking = true}`; `max_tokens = 49152`;
> `temperature`, `top_p`, `reasoning_effort` unset; `-c 8`; harness `null`; agent runtime
> `subprocess`; model `brain-qwen38-dspark` on spark-1. The configs are
+22
View File
@@ -0,0 +1,22 @@
# Prime-first boundary
Arena publishes Lumbridge environments; it does not implement another generic
harness, trainer, runtime marketplace, or environment hub.
- Every environment pins stable `verifiers==0.3.1`.
- Pure-trace environments select Verifiers' upstream `null` harness.
- Local model-free and spark-1 runs name the `subprocess` runtime explicitly so
tests never provision a paid Prime sandbox.
- Prime Hub is the public distribution target for approved environment wheels.
- Prime-RL consumes the same taskset packages for training; Forge owns the
explicit training handoff and artifact rendering.
Lumbridge's unique work is the environment: task distributions, interactive
state, rewards, probes, held-out gates, Tera bridges, and result interpretation.
Generic rollout and training machinery stays upstream. A Hub publication must
exclude secrets, private task splits, customer data, and proprietary Tera source.
The preferred future compute path is Prime's spot market. GH200/GB300 names in
runbooks are desired hardware profiles, not a promise that inventory exists and
not authorization to rent it. An operator checks current Prime availability and
price before launching any paid run.
+1 -1
View File
@@ -3,7 +3,7 @@ name = "bot-detection"
version = "0.1.0"
description = "bot-detection — find the scripts in an activity log without banning the players who grind like one."
requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"]
dependencies = ["verifiers==0.3.1"]
[build-system]
requires = ["hatchling"]
+1 -1
View File
@@ -3,7 +3,7 @@ name = "canary-trap"
version = "0.1.0"
description = "canary-trap — write contamination probes that survive a paraphrase, without accusing a clean model."
requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"]
dependencies = ["verifiers==0.3.1"]
[build-system]
requires = ["hatchling"]
@@ -3,7 +3,7 @@ name = "drop-table-inference"
version = "0.1.0"
description = "drop-table-inference — estimate a drop table from a kill log, graded on the kills you never saw."
requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"]
dependencies = ["verifiers==0.3.1"]
[build-system]
requires = ["hatchling"]
@@ -3,7 +3,7 @@ name = "fault-localisation"
version = "0.1.0"
description = "fault-localisation — name the service that broke, not the ones complaining about it."
requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"]
dependencies = ["verifiers==0.3.1"]
[build-system]
requires = ["hatchling"]
+1 -1
View File
@@ -3,7 +3,7 @@ name = "grand-exchange"
version = "0.1.0"
description = "grand-exchange — place orders executed against a market window you never saw."
requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"]
dependencies = ["verifiers==0.3.1"]
[build-system]
requires = ["hatchling"]
@@ -3,7 +3,7 @@ name = "grand-exchange-live"
version = "0.1.0"
description = "grand-exchange-live — quote, watch the tape, re-quote, over a turn budget."
requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"]
dependencies = ["verifiers==0.3.1"]
[build-system]
requires = ["hatchling"]
@@ -6,7 +6,7 @@ requires-python = ">=3.11"
dependencies = [
"pydantic==2.13.4",
"regex==2026.7.19",
"verifiers==0.3.1.dev59",
"verifiers==0.3.1",
]
[build-system]
+1 -1
View File
@@ -3,7 +3,7 @@ name = "schema-migration"
version = "0.1.0"
description = "schema-migration — split a text column into a number and a unit, graded on rows you never saw."
requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"]
dependencies = ["verifiers==0.3.1"]
[build-system]
requires = ["hatchling"]
+1 -1
View File
@@ -3,7 +3,7 @@ name = "tera-spatial"
version = "0.1.0"
description = "tera-spatial — Arena's bridge to Tera's four renderer-independent spatial environments, executed in TypeScript and graded in Python, and the tasksets that sit on it."
requires-python = ">=3.11"
dependencies = ["verifiers==0.3.1.dev59"]
dependencies = ["verifiers==0.3.1"]
# One directory, several tasksets: the environments share a vendored simulator and
# a single worker, so splitting them into four wheels would ship the same 283 KB
+24
View File
@@ -74,6 +74,30 @@ def discovered_tasksets() -> set[str]:
class RootProbeIntegrationTests(unittest.TestCase):
def test_prime_verifiers_boundary_is_pinned_and_upstream_owned(self) -> None:
manifests = sorted((ROOT / "environments").glob("*/pyproject.toml"))
configs = sorted((ROOT / "configs").glob("*.toml"))
self.assertTrue(manifests)
self.assertTrue(configs)
for manifest in manifests:
with self.subTest(manifest=manifest.parent.name):
data = tomllib.loads(manifest.read_text(encoding="utf-8"))
self.assertIn("verifiers==0.3.1", data["project"]["dependencies"])
for config in configs:
with self.subTest(config=config.stem):
data = tomllib.loads(config.read_text(encoding="utf-8"))
agent = data["env"]["agent"]
self.assertEqual(agent["harness"]["id"], "null")
self.assertEqual(agent["runtime"]["type"], "subprocess")
self.assertEqual(
list((ROOT / "environments").glob("**/harness.py")),
[],
msg="Arena environments must use Verifiers harnesses, not ship a parallel one",
)
def test_public_root_probe_gates_every_environment(self) -> None:
completed = subprocess.run(
[sys.executable, "probe.py"],