Pin Arena environments to stable Verifiers
This commit is contained in:
+1
-1
@@ -87,7 +87,7 @@ Corrected, blocking:
|
||||
6. **`portal_host_samples` and `portal_container_samples` have no node identity** — no `node_id`, no host column, no PK. Per-node charts need `ALTER TABLE … ADD COLUMN IF NOT EXISTS node_id text` plus an index first.
|
||||
7. **`scored = ok AND rewards non-empty` is wrong.** `write_episode` dumps with `exclude_none=True`, so a `None` reward is dropped from the file entirely — "not measured" and "key absent" are indistinguishable on the wire. Require at least one non-null `Reward`, and carry the run's declared reward-name set from the config.
|
||||
8. **`Reward.value` is a `@property`** and never appears in dumped JSON. Compute `score × weight` in the ingester.
|
||||
9. ⚠️ **CORRECTED 2026-08-21: there is NO verifiers 0.4.0.** The `version = "0.4.0"` at `~/vendor/prime-intellect/verifiers/pyproject.toml:136` belongs to a `[[tool.uv.dependency-metadata]]` block for **nemo-gym**, not to verifiers — read as verifiers' own version it sends you hunting a release that does not exist and concluding the bump is blocked. The vendored checkout is `v0.3.0-59-g4bcb48e5`, published to PyPI as **`0.3.1.dev59`**, byte-identical to the vendored tree, and that is what all nine manifests now pin. The layout warning itself is CORRECT: **0.3.0 → dev59 changes the directory layout, not just the config filename** — HEAD produces one flat `<env>--<model>--<harness>--<hex8>` dir with no nested uuid, and `RunConfig.id` is a `PrivateAttr` absent from `configs/eval.json`. Derive the run id from the directory basename.
|
||||
9. ⚠️ **CORRECTED 2026-08-25: there is NO verifiers 0.4.0.** The `version = "0.4.0"` at `~/vendor/prime-intellect/verifiers/pyproject.toml:136` belongs to a `[[tool.uv.dependency-metadata]]` block for **nemo-gym**, not to verifiers. Stable **`verifiers==0.3.1`** was published 2026-08-24 and is what all nine manifests pin. The layout warning itself is correct: 0.3.1 produces one flat `<env>--<model>--<harness>--<hex8>` directory with no nested UUID, and `RunConfig.id` is a `PrivateAttr` absent from `configs/eval.json`. Derive the run id from the directory basename.
|
||||
10. **Name the private console `/nodes`**, not `/compute` — `/compute` is already a public marketing section with its own nav.
|
||||
11. Put `isAppHost` in `packages/shared` — two independent string checks that must agree, with a silent asymmetric failure mode (dashboard rendered under a 404 status).
|
||||
12. `sendShell(url, reply)` takes no request; the signature and both call sites change. `spa-fallback.test.ts` needs a case. `@radix-ui/react-select` is **not** installed.
|
||||
|
||||
+3
-2
@@ -13,7 +13,8 @@ separable tiers, not one, and the cheapest tier needs no new infrastructure at a
|
||||
|
||||
## What upstream actually requires
|
||||
|
||||
Read from `~/vendor/prime-intellect/verifiers` (v1, installed version 0.3.0), not
|
||||
Read from stable `verifiers==0.3.1` and the matching checkout under
|
||||
`~/vendor/prime-intellect/verifiers`, not
|
||||
from blog posts.
|
||||
|
||||
**Multi-turn does not imply a container.** `vf.Env.run(task, agents)` programs the
|
||||
@@ -160,7 +161,7 @@ it is configured here.
|
||||
## Preconditions, verified on this box
|
||||
|
||||
- `uv run eval @ configs/canary_trap.toml --dry-run` → exit 0, config written ✓
|
||||
- verifiers 0.3.0 with all 13 harnesses and 4 runtimes present ✓
|
||||
- stable verifiers 0.3.1 with upstream harnesses and runtimes present ✓
|
||||
- Docker 29.1.3 running (tier 2/3 runtime) ✓
|
||||
- spark-1 `brain` on `http://100.127.247.67:8001/v1`, model id `brain` ✓
|
||||
- `prime` CLI installed (0.6.25; 0.6.26 available) ✓
|
||||
|
||||
@@ -216,7 +216,7 @@ files. Everything downstream of B fans out four ways.
|
||||
## Preconditions, verified on amd-server
|
||||
|
||||
- `uv run eval @ configs/<env>.toml --dry-run` → exit 0 ✓
|
||||
- verifiers 0.3.0, all 13 harnesses and 4 runtimes present ✓
|
||||
- stable verifiers 0.3.1, using its upstream `null` harness and runtimes ✓
|
||||
- Docker 29.1.3 (tier 2/3 only; tier 1 needs none) ✓
|
||||
- spark-1 `brain` at `http://100.127.247.67:8001/v1`, model id `brain` ✓
|
||||
- `prime` CLI 0.6.25 ✓
|
||||
|
||||
@@ -12,7 +12,7 @@ graded = 12 # unlabelled accounts the answer is scored on; 9 is the floor
|
||||
id = "null" # pure-trace reward: one completion, no tools, no runtime
|
||||
|
||||
[env.agent.runtime]
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
|
||||
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
|
||||
# `SystemExit: not authenticated with prime` before a single rollout, having already created
|
||||
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
|
||||
|
||||
@@ -11,7 +11,7 @@ id = "canary-trap"
|
||||
id = "null" # pure-trace reward: one completion, no tools, no runtime
|
||||
|
||||
[env.agent.runtime]
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
|
||||
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
|
||||
# `SystemExit: not authenticated with prime` before a single rollout, having already created
|
||||
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
|
||||
|
||||
@@ -13,7 +13,7 @@ held_out_kills = 40000 # kills it is scored on
|
||||
id = "null" # pure-trace reward: one completion, no tools, no runtime
|
||||
|
||||
[env.agent.runtime]
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
|
||||
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
|
||||
# `SystemExit: not authenticated with prime` before a single rollout, having already created
|
||||
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
|
||||
|
||||
@@ -11,7 +11,7 @@ id = "fault-localisation"
|
||||
id = "null" # pure-trace reward: one completion, no tools, no runtime
|
||||
|
||||
[env.agent.runtime]
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
|
||||
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
|
||||
# `SystemExit: not authenticated with prime` before a single rollout, having already created
|
||||
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
|
||||
|
||||
@@ -14,7 +14,7 @@ held_out = 30 # ticks its orders are executed against
|
||||
id = "null" # pure-trace reward: one completion, no tools, no runtime
|
||||
|
||||
[env.agent.runtime]
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
|
||||
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
|
||||
# `SystemExit: not authenticated with prime` before a single rollout, having already created
|
||||
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
|
||||
|
||||
@@ -23,7 +23,7 @@ visible = 56 # warmup ticks, read before the first look
|
||||
id = "null" # pure-trace reward: a chat loop, no tools, no runtime
|
||||
|
||||
[env.agent.runtime]
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
|
||||
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
|
||||
# `SystemExit: not authenticated with prime` before a single rollout, having already created
|
||||
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
|
||||
|
||||
@@ -14,7 +14,7 @@ held_out = 12 # records it is scored on
|
||||
id = "null" # a pure-trace reward: one completion, no tools, no runtime
|
||||
|
||||
[env.agent.runtime]
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
|
||||
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
|
||||
# `SystemExit: not authenticated with prime` before a single rollout, having already created
|
||||
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
|
||||
|
||||
@@ -11,7 +11,7 @@ id = "schema-migration"
|
||||
id = "null" # pure-trace reward: one completion, no tools, no runtime
|
||||
|
||||
[env.agent.runtime]
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
|
||||
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
|
||||
# `SystemExit: not authenticated with prime` before a single rollout, having already created
|
||||
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
|
||||
|
||||
@@ -25,7 +25,7 @@ seed_base = 115
|
||||
id = "null" # one completion per turn, no tools; the env programs the loop
|
||||
|
||||
[env.agent.runtime]
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1.dev59, which changed the default agent runtime from
|
||||
# ⚠️ REQUIRED since verifiers 0.3.1, which changed the default agent runtime from
|
||||
# `subprocess` to `prime` — a REMOTE sandbox. Without this the run exits
|
||||
# `SystemExit: not authenticated with prime` before a single rollout, having already created
|
||||
# its output directory and a zero-line traces.jsonl, so it reads as a failed eval rather than
|
||||
|
||||
+17
-18
@@ -7,8 +7,8 @@ training table — was waiting on a run that had never completed.
|
||||
|
||||
This is that run, written down so the next one is a copy-paste.
|
||||
|
||||
⚠️ **Updated 2026-08-21 evening.** The pin moved from `verifiers==0.3.0` to
|
||||
`verifiers==0.3.1.dev59`, which removes the 600-second model-call ceiling §5 blames for
|
||||
⚠️ **Updated 2026-08-25.** The pin is now the stable `verifiers==0.3.1`
|
||||
release published 2026-08-24. It removes the 600-second model-call ceiling §5 blames for
|
||||
every truncation here, and moves the output layout. §1 and §2 are the post-bump
|
||||
instructions; §4 and §5 are kept as written because they are the record of what the
|
||||
thinking-**off** run measured, and §6 is the thinking-**on** re-measurement the bump made
|
||||
@@ -50,8 +50,8 @@ see §5. The seven data environments have thinking-on copies in `outputs/thinkin
|
||||
| `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. |
|
||||
| `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. |
|
||||
| `-c 8` | spark-1 is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. |
|
||||
| `--env.agent.runtime.type subprocess` | ⚠️ **New, and not optional.** 0.3.0 defaulted the agent runtime to `subprocess`; `0.3.1.dev59` defaults it to `prime` — a *remote* sandbox. A remote role makes `Env._runs_local()` false, which makes the interception pool mint a Prime tunnel, which calls `ensure_prime_auth()` and raises `SystemExit: not authenticated with prime`. The run dies before a single rollout, having already created its output directory and an empty `traces.jsonl`. The null harness needs no runtime at all; naming a local one is the whole fix. |
|
||||
| `-o <dir>` + `--run.dir <name>` | `-o` is `output_dir` and the run writes to `output_dir / run.dir`. On 0.3.0 a bare `-o` gave you a flat directory; on dev59 it gives you `outputs/<your-dir>/<env>--<model>--<harness>--<hex8>/`. Pass `--run.dir` to name the leaf yourself. |
|
||||
| `--env.agent.runtime.type subprocess` | ⚠️ **New, and not optional.** 0.3.0 defaulted the agent runtime to `subprocess`; `0.3.1` defaults it to `prime` — a *remote* sandbox. A remote role makes `Env._runs_local()` false, which makes the interception pool mint a Prime tunnel, which calls `ensure_prime_auth()` and raises `SystemExit: not authenticated with prime`. The run dies before a single rollout, having already created its output directory and an empty `traces.jsonl`. The null harness needs no runtime at all; naming a local one is the whole fix. |
|
||||
| `-o <dir>` + `--run.dir <name>` | `-o` is `output_dir` and the run writes to `output_dir / run.dir`. On 0.3.0 a bare `-o` gave you a flat directory; on 0.3.1 it gives you `outputs/<your-dir>/<env>--<model>--<harness>--<hex8>/`. Pass `--run.dir` to name the leaf yourself. |
|
||||
|
||||
### Flags our own docs get wrong
|
||||
|
||||
@@ -80,18 +80,17 @@ earlier notes read off its `pyproject.toml:136` belongs to a `[[tool.uv.dependen
|
||||
block for **nemo-gym**, not to verifiers. Anyone who goes looking for `verifiers==0.4.0`
|
||||
will not find it.
|
||||
|
||||
What *is* published is the dev stream: PyPI carries 45 `0.3.1.devN` releases, one per
|
||||
merged commit. `0.3.1.dev59` was uploaded 2026-08-20 and is exactly the vendored checkout —
|
||||
`verifiers/v1/cli/output.py` extracted from the wheel is byte-identical to the file on
|
||||
disk, and `verifiers/v1/clients/base.py:12` reads:
|
||||
The stable `0.3.1` wheel was published to PyPI on 2026-08-24 after the dev stream.
|
||||
It contains the unbounded model-call timeout and output-layout changes this runbook
|
||||
requires; `verifiers/v1/clients/base.py` reads:
|
||||
|
||||
```python
|
||||
DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=None, write=None, pool=None)
|
||||
```
|
||||
|
||||
**So the pin is an ordinary index dependency, not a path or git source.** Every
|
||||
`environments/*/pyproject.toml` now reads `verifiers==0.3.1.dev59`. An exact `==` on a
|
||||
pre-release is enough for uv — no `--prerelease=allow`, no `[tool.uv.sources]`, no
|
||||
`environments/*/pyproject.toml` now reads `verifiers==0.3.1`. It is an ordinary
|
||||
stable index dependency — no `--prerelease=allow`, no `[tool.uv.sources]`, no
|
||||
`tool.uv` block, and nothing that resolves differently in CI than it does here. A path
|
||||
dependency on `~/vendor` would have been reproducible only on this box; a git URL would
|
||||
have pinned a commit no wheel matches. Neither was necessary.
|
||||
@@ -99,7 +98,7 @@ have pinned a commit no wheel matches. Neither was necessary.
|
||||
```
|
||||
Resolved 114 packages in 620ms
|
||||
- verifiers==0.3.0
|
||||
+ verifiers==0.3.1.dev59
|
||||
+ verifiers==0.3.1
|
||||
```
|
||||
|
||||
`uv sync` each environment after pulling. Verify the thing you bumped *for*, not the
|
||||
@@ -120,7 +119,7 @@ is empty), so the bump does not put a lock file into a commit. CI's lock-policy
|
||||
git, so it passes on a fresh checkout and would fail on a working tree that has been
|
||||
synced. Do not run it locally and conclude the repo is broken.
|
||||
|
||||
### The layout dev59 writes
|
||||
### The layout 0.3.1 writes
|
||||
|
||||
`verifiers/v1/cli/output.py` — `output_path()` is `config.output_dir / config.run.dir`,
|
||||
and `run.dir` defaults to `run.name`, which is
|
||||
@@ -136,7 +135,7 @@ outputs/<output-dir>/<env>--<model>--<harness>--<hex8>/
|
||||
|
||||
Three differences from 0.3.0 that an ingester or a script has to handle:
|
||||
|
||||
1. **The uuid leaf is gone.** 0.3.0 wrote `<env>--<model>--<harness>/<uuid>/`; dev59
|
||||
1. **The uuid leaf is gone.** 0.3.0 wrote `<env>--<model>--<harness>/<uuid>/`; 0.3.1
|
||||
writes one flat directory with an 8-hex suffix on the name. `-o` no longer flattens
|
||||
anything — it sets `output_dir`, and the run still makes its own leaf underneath.
|
||||
`--run.dir <name>` is what pins the leaf, and it is what `outputs/thinking-n32/run_all.sh`
|
||||
@@ -164,7 +163,7 @@ training-native` landing in this range. One Episode per line, not one Trace:
|
||||
```
|
||||
|
||||
`outputs/count_rewards.py`, written against 0.3.0, reproduces every published number from
|
||||
`run-20260821-1401` unchanged against a dev59 run. The two traps in that file are also
|
||||
`run-20260821-1401` unchanged against a 0.3.1 run. The two traps in that file are also
|
||||
unchanged, and both still bite an ingester:
|
||||
|
||||
1. **`Reward.value` is a `@property` and never appears in the JSON.** Each entry
|
||||
@@ -481,11 +480,11 @@ Three consequences for anyone re-running this suite against a reasoning model:
|
||||
makes the "grand-exchange is solvable" claim — currently resting on six rollouts — cheap
|
||||
to settle properly.
|
||||
|
||||
✅ **RETIRED 2026-08-21 evening.** The bump happened — `verifiers==0.3.1.dev59`, §2 — and
|
||||
✅ **RETIRED 2026-08-21 evening.** The bump happened — `verifiers==0.3.1`, §2 — and
|
||||
every venv on this box now reads `read=None`. One correction to the paragraph above, which
|
||||
matters because it is the version anyone will type: the fix is **not** `verifiers==0.4.0`.
|
||||
There is no such release. `a298bcfe` is `v0.3.0-8-ga298bcfe`; the vendored HEAD is
|
||||
`v0.3.0-59-g4bcb48e5`; the wheel that carries them is `0.3.1.dev59`. Everything below this
|
||||
`v0.3.0-59-g4bcb48e5`; the wheel that carries them is `0.3.1`. Everything below this
|
||||
line about 600 seconds describes the old pin and is kept as the record of why the bump was
|
||||
worth doing, not as current behaviour. The `max_tokens` / wall-clock contradiction it
|
||||
describes — "the two limits point in opposite directions and there is no setting that
|
||||
@@ -532,12 +531,12 @@ recorded.
|
||||
|
||||
## 6. The thinking-ON re-measurement, n=32
|
||||
|
||||
Run 2026-08-21 evening, after the `verifiers==0.3.1.dev59` bump of §2. This section
|
||||
Run 2026-08-21 evening, after the `verifiers==0.3.1` bump of §2. This section
|
||||
exists because two of §4's seven numbers were artefacts of a sampling configuration
|
||||
nobody chose and nothing recorded, and because §5's remedy — "an n=32 thinking-on run is
|
||||
achievable today" — was owed a run.
|
||||
|
||||
> **Sampling configuration for every number in this section.** `verifiers==0.3.1.dev59`
|
||||
> **Sampling configuration for every number in this section.** `verifiers==0.3.1`
|
||||
> (`read=None`); `chat_template_kwargs = {enable_thinking = true}`; `max_tokens = 49152`;
|
||||
> `temperature`, `top_p`, `reasoning_effort` unset; `-c 8`; harness `null`; agent runtime
|
||||
> `subprocess`; model `brain-qwen38-dspark` on spark-1. The configs are
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
# Prime-first boundary
|
||||
|
||||
Arena publishes Lumbridge environments; it does not implement another generic
|
||||
harness, trainer, runtime marketplace, or environment hub.
|
||||
|
||||
- Every environment pins stable `verifiers==0.3.1`.
|
||||
- Pure-trace environments select Verifiers' upstream `null` harness.
|
||||
- Local model-free and spark-1 runs name the `subprocess` runtime explicitly so
|
||||
tests never provision a paid Prime sandbox.
|
||||
- Prime Hub is the public distribution target for approved environment wheels.
|
||||
- Prime-RL consumes the same taskset packages for training; Forge owns the
|
||||
explicit training handoff and artifact rendering.
|
||||
|
||||
Lumbridge's unique work is the environment: task distributions, interactive
|
||||
state, rewards, probes, held-out gates, Tera bridges, and result interpretation.
|
||||
Generic rollout and training machinery stays upstream. A Hub publication must
|
||||
exclude secrets, private task splits, customer data, and proprietary Tera source.
|
||||
|
||||
The preferred future compute path is Prime's spot market. GH200/GB300 names in
|
||||
runbooks are desired hardware profiles, not a promise that inventory exists and
|
||||
not authorization to rent it. An operator checks current Prime availability and
|
||||
price before launching any paid run.
|
||||
@@ -3,7 +3,7 @@ name = "bot-detection"
|
||||
version = "0.1.0"
|
||||
description = "bot-detection — find the scripts in an activity log without banning the players who grind like one."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers==0.3.1.dev59"]
|
||||
dependencies = ["verifiers==0.3.1"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
|
||||
@@ -3,7 +3,7 @@ name = "canary-trap"
|
||||
version = "0.1.0"
|
||||
description = "canary-trap — write contamination probes that survive a paraphrase, without accusing a clean model."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers==0.3.1.dev59"]
|
||||
dependencies = ["verifiers==0.3.1"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
|
||||
@@ -3,7 +3,7 @@ name = "drop-table-inference"
|
||||
version = "0.1.0"
|
||||
description = "drop-table-inference — estimate a drop table from a kill log, graded on the kills you never saw."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers==0.3.1.dev59"]
|
||||
dependencies = ["verifiers==0.3.1"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
|
||||
@@ -3,7 +3,7 @@ name = "fault-localisation"
|
||||
version = "0.1.0"
|
||||
description = "fault-localisation — name the service that broke, not the ones complaining about it."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers==0.3.1.dev59"]
|
||||
dependencies = ["verifiers==0.3.1"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
|
||||
@@ -3,7 +3,7 @@ name = "grand-exchange"
|
||||
version = "0.1.0"
|
||||
description = "grand-exchange — place orders executed against a market window you never saw."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers==0.3.1.dev59"]
|
||||
dependencies = ["verifiers==0.3.1"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
|
||||
@@ -3,7 +3,7 @@ name = "grand-exchange-live"
|
||||
version = "0.1.0"
|
||||
description = "grand-exchange-live — quote, watch the tape, re-quote, over a turn budget."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers==0.3.1.dev59"]
|
||||
dependencies = ["verifiers==0.3.1"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
|
||||
@@ -6,7 +6,7 @@ requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"pydantic==2.13.4",
|
||||
"regex==2026.7.19",
|
||||
"verifiers==0.3.1.dev59",
|
||||
"verifiers==0.3.1",
|
||||
]
|
||||
|
||||
[build-system]
|
||||
|
||||
@@ -3,7 +3,7 @@ name = "schema-migration"
|
||||
version = "0.1.0"
|
||||
description = "schema-migration — split a text column into a number and a unit, graded on rows you never saw."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers==0.3.1.dev59"]
|
||||
dependencies = ["verifiers==0.3.1"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
|
||||
@@ -3,7 +3,7 @@ name = "tera-spatial"
|
||||
version = "0.1.0"
|
||||
description = "tera-spatial — Arena's bridge to Tera's four renderer-independent spatial environments, executed in TypeScript and graded in Python, and the tasksets that sit on it."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers==0.3.1.dev59"]
|
||||
dependencies = ["verifiers==0.3.1"]
|
||||
|
||||
# One directory, several tasksets: the environments share a vendored simulator and
|
||||
# a single worker, so splitting them into four wheels would ship the same 283 KB
|
||||
|
||||
@@ -74,6 +74,30 @@ def discovered_tasksets() -> set[str]:
|
||||
|
||||
|
||||
class RootProbeIntegrationTests(unittest.TestCase):
|
||||
def test_prime_verifiers_boundary_is_pinned_and_upstream_owned(self) -> None:
|
||||
manifests = sorted((ROOT / "environments").glob("*/pyproject.toml"))
|
||||
configs = sorted((ROOT / "configs").glob("*.toml"))
|
||||
self.assertTrue(manifests)
|
||||
self.assertTrue(configs)
|
||||
|
||||
for manifest in manifests:
|
||||
with self.subTest(manifest=manifest.parent.name):
|
||||
data = tomllib.loads(manifest.read_text(encoding="utf-8"))
|
||||
self.assertIn("verifiers==0.3.1", data["project"]["dependencies"])
|
||||
|
||||
for config in configs:
|
||||
with self.subTest(config=config.stem):
|
||||
data = tomllib.loads(config.read_text(encoding="utf-8"))
|
||||
agent = data["env"]["agent"]
|
||||
self.assertEqual(agent["harness"]["id"], "null")
|
||||
self.assertEqual(agent["runtime"]["type"], "subprocess")
|
||||
|
||||
self.assertEqual(
|
||||
list((ROOT / "environments").glob("**/harness.py")),
|
||||
[],
|
||||
msg="Arena environments must use Verifiers harnesses, not ship a parallel one",
|
||||
)
|
||||
|
||||
def test_public_root_probe_gates_every_environment(self) -> None:
|
||||
completed = subprocess.run(
|
||||
[sys.executable, "probe.py"],
|
||||
|
||||
Reference in New Issue
Block a user