Document the current public Arena boundary
arena-environments / validate (3.11) (push) Successful in 1m16s
arena-environments / validate (3.12) (push) Successful in 1m43s

This commit is contained in:
2026-08-25 14:59:00 -07:00
parent 4836d3e644
commit 3025c4cf85
5 changed files with 21 additions and 22 deletions
+1 -1
View File
@@ -87,7 +87,7 @@ Corrected, blocking:
6. **`portal_host_samples` and `portal_container_samples` have no node identity** — no `node_id`, no host column, no PK. Per-node charts need `ALTER TABLE … ADD COLUMN IF NOT EXISTS node_id text` plus an index first. 6. **`portal_host_samples` and `portal_container_samples` have no node identity** — no `node_id`, no host column, no PK. Per-node charts need `ALTER TABLE … ADD COLUMN IF NOT EXISTS node_id text` plus an index first.
7. **`scored = ok AND rewards non-empty` is wrong.** `write_episode` dumps with `exclude_none=True`, so a `None` reward is dropped from the file entirely — "not measured" and "key absent" are indistinguishable on the wire. Require at least one non-null `Reward`, and carry the run's declared reward-name set from the config. 7. **`scored = ok AND rewards non-empty` is wrong.** `write_episode` dumps with `exclude_none=True`, so a `None` reward is dropped from the file entirely — "not measured" and "key absent" are indistinguishable on the wire. Require at least one non-null `Reward`, and carry the run's declared reward-name set from the config.
8. **`Reward.value` is a `@property`** and never appears in dumped JSON. Compute `score × weight` in the ingester. 8. **`Reward.value` is a `@property`** and never appears in dumped JSON. Compute `score × weight` in the ingester.
9. ⚠️ **CORRECTED 2026-08-25: there is NO verifiers 0.4.0.** The `version = "0.4.0"` at `~/vendor/prime-intellect/verifiers/pyproject.toml:136` belongs to a `[[tool.uv.dependency-metadata]]` block for **nemo-gym**, not to verifiers. Stable **`verifiers==0.3.1`** was published 2026-08-24 and is what all nine manifests pin. The layout warning itself is correct: 0.3.1 produces one flat `<env>--<model>--<harness>--<hex8>` directory with no nested UUID, and `RunConfig.id` is a `PrivateAttr` absent from `configs/eval.json`. Derive the run id from the directory basename. 9. ⚠️ **CORRECTED 2026-08-25: there is NO verifiers 0.4.0.** The `version = "0.4.0"` at `~/vendor/prime-intellect/verifiers/pyproject.toml:136` belongs to a `[[tool.uv.dependency-metadata]]` block for **nemo-gym**, not to verifiers. Stable **`verifiers==0.3.1`** was published 2026-08-24 and is what all eight current public manifests pin; the Tera spatial package moved to private source. The layout warning itself is correct: 0.3.1 produces one flat `<env>--<model>--<harness>--<hex8>` directory with no nested UUID, and `RunConfig.id` is a `PrivateAttr` absent from `configs/eval.json`. Derive the run id from the directory basename.
10. **Name the private console `/nodes`**, not `/compute``/compute` is already a public marketing section with its own nav. 10. **Name the private console `/nodes`**, not `/compute``/compute` is already a public marketing section with its own nav.
11. Put `isAppHost` in `packages/shared` — two independent string checks that must agree, with a silent asymmetric failure mode (dashboard rendered under a 404 status). 11. Put `isAppHost` in `packages/shared` — two independent string checks that must agree, with a silent asymmetric failure mode (dashboard rendered under a 404 status).
12. `sendShell(url, reply)` takes no request; the signature and both call sites change. `spa-fallback.test.ts` needs a case. `@radix-ui/react-select` is **not** installed. 12. `sendShell(url, reply)` takes no request; the signature and both call sites change. `spa-fallback.test.ts` needs a case. `@radix-ui/react-select` is **not** installed.
+10 -6
View File
@@ -1,8 +1,8 @@
# Arena # Arena
RL environments from Lumbridge. Apache-2.0, built to the [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) RL environments from Lumbridge. Apache-2.0, built to the [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers)
v1 spec so the same wheel installs from the Prime Intellect Environments Hub and runs here v1 spec so the same wheel can run locally or, after a separate owner-approved release,
unchanged. through the Prime Intellect Environments Hub unchanged.
An environment is an eval you take the gradient of. That is not a rename — it changes what An environment is an eval you take the gradient of. That is not a rename — it changes what
the artifact has to carry. A benchmark's score is its output, so a benchmark can be a number the artifact has to carry. A benchmark's score is its output, so a benchmark can be a number
@@ -167,12 +167,16 @@ uv run --with regex python probe.py
Arena contains independently published environment libraries, so per-environment Arena contains independently published environment libraries, so per-environment
`uv.lock` files are intentionally not committed. Redaction v0.2 pins its runtime contract `uv.lock` files are intentionally not committed. Redaction v0.2 pins its runtime contract
in `pyproject.toml`; CI resolves it on Python 3.11 and 3.12, builds every environment, and in `pyproject.toml`; CI resolves it on Python 3.11 and 3.12, builds all eight public
runs the shared four-environment probe. environments, and runs the auto-discovered probe for every one of them.
## Publishing ## Publishing (deferred)
The layout mirrors `verifiers`' own, so an environment goes to the Hub without a fork: The layout mirrors `verifiers`' own, so an environment can eventually go to the Hub without
a fork. No current environment is published there from this repository. Hub publication is
an explicit owner-approved outward action; ordinary development and CI must never run it.
After that separate approval, the upstream command is:
```bash ```bash
prime env push redaction-pressure -v PUBLIC prime env push redaction-pressure -v PUBLIC
+2 -2
View File
@@ -31,7 +31,7 @@ uv run --project environments/canary_trap eval @ configs/canary_trap.toml \
-c 8 -o outputs/run-<stamp> --run.dir canary-trap -c 8 -o outputs/run-<stamp> --run.dir canary-trap
``` ```
Swap `environments/<pkg>` and `configs/<pkg>.toml` for any of the nine in `configs/`. The Swap `environments/<pkg>` and `configs/<pkg>.toml` for any of the eight in `configs/`. The
thinking-**on** sweep is `outputs/thinking-n32/run_all.sh` (gitignored, and it takes a thinking-**on** sweep is `outputs/thinking-n32/run_all.sh` (gitignored, and it takes a
`flock` — see below); the counters are `outputs/count_rewards.py` and `flock` — see below); the counters are `outputs/count_rewards.py` and
`outputs/summarise_run.py`. `outputs/summarise_run.py`.
@@ -109,7 +109,7 @@ for d in environments/*/; do printf "%-34s " "$d"
grep -h "DEFAULT_TIMEOUT = " $d/.venv/lib/python*/site-packages/verifiers/v1/clients/base.py grep -h "DEFAULT_TIMEOUT = " $d/.venv/lib/python*/site-packages/verifiers/v1/clients/base.py
done done
# environments/bot_detection/ DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=None, write=None, pool=None) # environments/bot_detection/ DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=None, write=None, pool=None)
# ... nine of nine # ... eight of eight
``` ```
`uv sync` rewrites each `environments/*/uv.lock`; none of them are tracked any more `uv sync` rewrites each `environments/*/uv.lock`; none of them are tracked any more
+3 -4
View File
@@ -115,11 +115,10 @@ ENVS = Path(__file__).parent / "environments"
def discover() -> dict[str, list[Path]]: def discover() -> dict[str, list[Path]]:
"""Taskset id -> the package directories that ship it, read off the manifests. """Taskset id -> the package directories that ship it, read off the manifests.
The seven names this file used to carry as a literal tuple are `environments/*/pyproject.toml` The names this file once carried as a literal tuple are `environments/*/pyproject.toml`
now: an environment is gated by existing, not by being remembered here. One directory may now: an environment is gated by existing, not by being remembered here. One directory may
declare more than one taskset — the Tera bridge ships four spatial tasksets out of a single declare more than one taskset through an optional `[tool.arena] tasksets = [...]`. Absent
package — through an optional `[tool.arena] tasksets = [...]`. Absent that key the taskset id that key the taskset id is the project name, which is what all eight public packages do.
is the project name, which is what all seven of the originals do.
Other sessions create directories under `environments/` while this runs, so a manifest that Other sessions create directories under `environments/` while this runs, so a manifest that
disappears between the glob and the read is skipped rather than fatal. A manifest that is disappears between the glob and the read is skipped rather than fatal. A manifest that is
+5 -9
View File
@@ -41,21 +41,17 @@ STEP_AT_ORACLE = frozenset({
("fault-localisation", "gate"), ("fault-localisation", "gate"),
}) })
# The environments that are allowed to carry no probe yet, because their taskset layer is not # The environments that are allowed to carry no probe yet. Everything discovered and NOT in
# written. Everything discovered and NOT in here MUST be gated. # here MUST be gated.
# #
# This list is the floor, and it is why it exists rather than a bare count: without it, both # This list is the floor, and it is why it exists rather than a bare count: without it, both
# loops below are vacuous when nothing is gated, and CI goes green while gating zero # loops below are vacuous when nothing is gated, and CI goes green while gating zero
# environments. Commenting out a single `@probes(...)` decorator used to pass — the degated # environments. Commenting out a single `@probes(...)` decorator used to pass — the degated
# name simply moved into the warning line, which already legitimately carries five names. # name simply moved into the warning line, which already legitimately carries five names.
# #
# It only ever shrinks. Delete a name here in the same commit that lands its probe. # It only ever shrinks. The former Tera entries moved to the private Tera repository with their
PENDING_PROBES = frozenset({ # simulator source, so every environment remaining in public Arena is gated today.
"tera-crow-nav", PENDING_PROBES: frozenset[str] = frozenset()
"tera-drive-101",
"tera-office-nav",
"tera-california-flight",
})
def discovered_tasksets() -> set[str]: def discovered_tasksets() -> set[str]: