docs: redact the reference node's address and hostname
arena-environments / validate (3.11) (push) Successful in 1m23s
arena-environments / validate (3.12) (push) Successful in 1m8s

The eval docs named spark-1 by fleet hostname and published its inference
endpoint (a tailnet address and port) plus the fact that it answers without
an API key. This repo is PUBLIC, so that was a standing disclosure of an
internal, unauthenticated service to anonymous readers.

Endpoints now come from $SPARK_BASE_URL, which the operator exports, and the
host is referred to as "the reference node" -- the convention the compute
repo's docs already use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AaUFYkUTsJn1fnJ89qbvW
This commit is contained in:
2026-09-15 12:15:37 -07:00
co-authored by Claude Opus 5
parent 3025c4cf85
commit ef01bde068
7 changed files with 33 additions and 30 deletions
+6 -6
View File
@@ -79,7 +79,7 @@ Stands and is high-value: the endpoint inventory (29 routes, not 31), the 23 dea
Corrected, blocking: Corrected, blocking:
1. **"Ingest the runs already on amd-server" is impossible — there are none.** One `config.toml`, zero episodes, and that config points at `api.pinference.ai` with `PRIME_API_KEY`, which is empty. **Phase 1 must contain "execute one eval to completion against spark-1" as an explicit prerequisite**, or Arena's dashboard ships empty. 1. **"Ingest the runs already on amd-server" is impossible — there are none.** One `config.toml`, zero episodes, and that config points at `api.pinference.ai` with `PRIME_API_KEY`, which is empty. **Phase 1 must contain "execute one eval to completion against the reference node" as an explicit prerequisite**, or Arena's dashboard ships empty.
2. **The leaderboard is operator-gameable and mis-specified.** (a) The server must recompute every aggregate in SQL from the `arena_episodes` it just inserted, never trust the pusher's numbers. (b) Ranking on a mean over *scored* episodes is Arena's own "score what survived" trap. Store `reward_mean_scored` **and** `reward_mean_attempted` (errors as 0), rank on attempted, refuse to rank below a scored-fraction floor. 2. **The leaderboard is operator-gameable and mis-specified.** (a) The server must recompute every aggregate in SQL from the `arena_episodes` it just inserted, never trust the pusher's numbers. (b) Ranking on a mean over *scored* episodes is Arena's own "score what survived" trap. Store `reward_mean_scored` **and** `reward_mean_attempted` (errors as 0), rank on attempted, refuse to rank below a scored-fraction floor.
3. **`arena_episodes` flattens a list.** `Episode.traces` is a list; `reward`/`rewards`/`stop_condition`/`last_reply` are per-**Trace**. Primary key must be `(run_id, episode_id, trace_id)` with an `agent_name` column — otherwise the schema is wrong for `grand-exchange-live` (simulator as second role) and for `redaction-pressure` (genuine two-agent). 3. **`arena_episodes` flattens a list.** `Episode.traces` is a list; `reward`/`rewards`/`stop_condition`/`last_reply` are per-**Trace**. Primary key must be `(run_id, episode_id, trace_id)` with an `agent_name` column — otherwise the schema is wrong for `grand-exchange-live` (simulator as second role) and for `redaction-pressure` (genuine two-agent).
4. **The trace viewer publishes the held-out slice.** `GET /api/arena/runs/:id/episodes/:eid` at `user` scope returns full transcripts, which contain held-out task data and, for oracle runs, the answers. One leaked invite publishes the held-out corpus. **Transcripts are admin-only or redacted.** This settles the spec's open question on read scope. 4. **The trace viewer publishes the held-out slice.** `GET /api/arena/runs/:id/episodes/:eid` at `user` scope returns full transcripts, which contain held-out task data and, for oracle runs, the answers. One leaked invite publishes the held-out corpus. **Transcripts are admin-only or redacted.** This settles the spec's open question on read scope.
@@ -115,7 +115,7 @@ Corrected:
### C-compute — rent to train, Spark to serve · feasible; **its headline correction is false** ### C-compute — rent to train, Spark to serve · feasible; **its headline correction is false**
Stands and is the most immediately actionable: the policy itself; the measured spark-1 tenancy (121 GB total / 84 GB used / 36 GB free, SGLang + Chatterbox + vox + ASR co-resident, Chatterbox's ~8.4 GB/hr leak having killed brain before); `prime login` as step zero because `api_key` is empty (verified); **Arena → prime-rl needs no adapter**`EnvConfig` is `SerializeAsAny[vf.EnvConfig]` and the taskset tree is the same shape, prefixed; the seat-name-is-a-TOML-key trap; Hosted Training being unavailable while E is parked (`rl.py:1522-1524` — bare v1 ids do not resolve through the Hub); the sm_121 lock contradiction in Forge; `TRAINERS = ("axolotl","trl")`; **no price is hardcoded anywhere and none should be invented.** Stands and is the most immediately actionable: the policy itself; the measured the reference node tenancy (121 GB total / 84 GB used / 36 GB free, SGLang + Chatterbox + vox + ASR co-resident, Chatterbox's ~8.4 GB/hr leak having killed brain before); `prime login` as step zero because `api_key` is empty (verified); **Arena → prime-rl needs no adapter**`EnvConfig` is `SerializeAsAny[vf.EnvConfig]` and the taskset tree is the same shape, prefixed; the seat-name-is-a-TOML-key trap; Hosted Training being unavailable while E is parked (`rl.py:1522-1524` — bare v1 ids do not resolve through the Hub); the sm_121 lock contradiction in Forge; `TRAINERS = ("axolotl","trl")`; **no price is hardcoded anywhere and none should be invented.**
Corrected, blocking: Corrected, blocking:
@@ -141,7 +141,7 @@ Corrected, blocking:
| **1** | **B1** — discovery de-hardcode + timeout + step name + `uv.lock` cleanup | `arena` | nothing | | **1** | **B1** — discovery de-hardcode + timeout + step name + `uv.lock` cleanup | `arena` | nothing |
| **2** | **A step 2**`live.py` engine + unit tests + **re-measure the ladder** | `arena` | nothing (own worktree) | | **2** | **A step 2**`live.py` engine + unit tests + **re-measure the ladder** | `arena` | nothing (own worktree) |
| **3** | **D step 1** — worker, vendored TS, hashes, `test_replay.py` | `arena` (new dir only) | nothing | | **3** | **D step 1** — worker, vendored TS, hashes, `test_replay.py` | `arena` (new dir only) | nothing |
| **4** | **Compute Policy step 0**`prime login`, catalogue + prices, doc the corrected spark-1 facts | none / `arena/docs` | nothing | | **4** | **Compute Policy step 0**`prime login`, catalogue + prices, doc the corrected the reference node facts | none / `arena/docs` | nothing |
| **5** | **P1** — Caddy host, host-aware routing, AppShell, Overview/Worlds/Org/Inbox | `lumbridge-v4` | nothing | | **5** | **P1** — Caddy host, host-aware routing, AppShell, Overview/Worlds/Org/Inbox | `lumbridge-v4` | nothing |
| **6** | **W step 0** — the five factual corrections + `eval-results.json` decision | `lumbridge-v4` | nothing | | **6** | **W step 0** — the five factual corrections + `eval-results.json` decision | `lumbridge-v4` | nothing |
@@ -209,7 +209,7 @@ Two edges that were wrong in the source specs and are now correct:
1. **B1** (arena): tomllib discovery, `@probes` registry, timeout 180, `uv.lock` untracking, step rename. *Gate: `probe.py` still prints seven `ok` rows; CI green with zero list edits.* 1. **B1** (arena): tomllib discovery, `@probes` registry, timeout 180, `uv.lock` untracking, step rename. *Gate: `probe.py` still prints seven `ok` rows; CI green with zero list edits.*
2. **A steps 1-2 + re-measurement** (arena, worktree): frozen generator copy + equality assertion; `LiveBook`/`Leg`/`Ledger` with unit tests; **then re-run the full policy ladder and publish the corrected numbers.** *Gate: the ladder reconciles with the reward formula; `horizon`, `FREEZE_RANGE`, `TARGET_SHARE`, `TURNS` are chosen from that measurement, not from the spec.* 2. **A steps 1-2 + re-measurement** (arena, worktree): frozen generator copy + equality assertion; `LiveBook`/`Leg`/`Ledger` with unit tests; **then re-run the full policy ladder and publish the corrected numbers.** *Gate: the ladder reconciles with the reward formula; `horizon`, `FREEZE_RANGE`, `TARGET_SHARE`, `TURNS` are chosen from that measurement, not from the spec.*
3. **D step 1** (arena, new dir): worker + vendored TS + `hashes.py` + `sync_tera.py` + `test_replay.py`. *Gate: 32/32 replay exact, tampering rejected, hash check green — and a decision on the 4 unpinned files.* 3. **D step 1** (arena, new dir): worker + vendored TS + `hashes.py` + `sync_tera.py` + `test_replay.py`. *Gate: 32/32 replay exact, tampering rejected, hash check green — and a decision on the 4 unpinned files.*
4. **Compute Policy step 0**: `prime login`; record real `availability` output and prices; write `arena/docs/COMPUTE_POLICY.md` with the corrected spark-1 facts (`brain-qwen38-dspark`, silent-wrong-model hazard, 36 GB headroom). 4. **Compute Policy step 0**: `prime login`; record real `availability` output and prices; write `arena/docs/COMPUTE_POLICY.md` with the corrected the reference node facts (`brain-qwen38-dspark`, silent-wrong-model hazard, 36 GB headroom).
5. **P1** (lumbridge-v4): Caddy `app.` block with `bind 10.0.0.2` and the non-empty-body check; host-aware `App.tsx`; `renderAppShell` + `sendShell` signature change + `seo.test.ts` + `spa-fallback.test.ts`; AppShell; Overview / Worlds / Org / Inbox; delete the 23 dead `api.ts` methods. 5. **P1** (lumbridge-v4): Caddy `app.` block with `bind 10.0.0.2` and the non-empty-body check; host-aware `App.tsx`; `renderAppShell` + `sendShell` signature change + `seo.test.ts` + `spa-fallback.test.ts`; AppShell; Overview / Worlds / Org / Inbox; delete the 23 dead `api.ts` methods.
6. **W step 0** (lumbridge-v4): the five factual corrections — `arena.tsx:330` "Nine tasks", `landing.tsx` card 02, the 4-row probe table → 7 rows, `office-jobs-v1` deleted, `arena.tsx:519` "no model scores published". Ship the Nemotron run into `eval-results.json`. 6. **W step 0** (lumbridge-v4): the five factual corrections — `arena.tsx:330` "Nine tasks", `landing.tsx` card 02, the 4-row probe table → 7 rows, `office-jobs-v1` deleted, `arena.tsx:519` "no model scores published". Ship the Nemotron run into `eval-results.json`.
@@ -218,7 +218,7 @@ Two edges that were wrong in the source specs and are now correct:
### Phase 2 ### Phase 2
7. **A steps 3-9**`grand-exchange-live` ships with its probe row (needs B1 merged). 7. **A steps 3-9**`grand-exchange-live` ships with its probe row (needs B1 merged).
8. **One real eval run** against spark-1, pinned to `brain-qwen38-dspark`, with the empty-rewards count checked. *This is a hard prerequisite for P2 and for any site number.* 8. **One real eval run** against the reference node, pinned to `brain-qwen38-dspark`, with the empty-rewards count checked. *This is a hard prerequisite for P2 and for any site number.*
9. **D crow-nav** end to end (role named `agent`, `vf.Taskset[...]` generic form, `oracle` op in the worker, `gate` as a multiplier). 9. **D crow-nav** end to end (role named `agent`, `vf.Taskset[...]` generic form, `oracle` op in the worker, `gate` as a multiplier).
10. **W /arena rebuild** + `/bench` on the new primitives. 10. **W /arena rebuild** + `/bench` on the new primitives.
11. **First rented run**: pod create with an attached disk, `install.sh`, wheel install, baseline eval, `uv run rl`, re-eval, before/after table. 11. **First rented run**: pod create with an attached disk, `install.sh`, wheel install, baseline eval, `uv run rl`, re-eval, before/after table.
@@ -263,7 +263,7 @@ Two edges that were wrong in the source specs and are now correct:
| **5** | **Silent wrong-model comparison.** `--model brain` returns 200 from a box serving Qwen3.8-27B. Re-running the Nemotron baseline today produces different-model numbers under an identical command, and they will be tabulated as comparable. | Pinning `brain-qwen38-dspark` (and the scene) in every command in `README.md`, `PLAN.md`, `MULTI_TURN.md` and `COMPUTE_POLICY.md`. Trivial; do it in Phase 1. | | **5** | **Silent wrong-model comparison.** `--model brain` returns 200 from a box serving Qwen3.8-27B. Re-running the Nemotron baseline today produces different-model numbers under an identical command, and they will be tabulated as comparable. | Pinning `brain-qwen38-dspark` (and the scene) in every command in `README.md`, `PLAN.md`, `MULTI_TURN.md` and `COMPUTE_POLICY.md`. Trivial; do it in Phase 1. |
| **6** | **`probe.py`, `tests/test_probe.py` and `ci.yml` are edited by five of six workstreams, in a repo that already took divergent concurrent pushes.** | B1, first, before anything else touches `arena`. | | **6** | **`probe.py`, `tests/test_probe.py` and `ci.yml` are edited by five of six workstreams, in a repo that already took divergent concurrent pushes.** | B1, first, before anything else touches `arena`. |
| **7** | **The reward denominator is editable with every `tera` gate green.** ⚠️ CORRECTED 2026-08-21: not `offices/sites.ts` (mutating it produces a bit-identical episode — refuted twice). The hole is **`src/arena/index.ts`**, which re-exports the scripted baselines that ARE the denominator. Demonstrated: a halved office-nav baseline changes return 4.744 → 4.098 while `arena:source-hashes` prints ok and exits 0. | Pinning `src/arena/index.ts` in `tera/scripts/check-arena-source-hashes.mjs` and regenerating `sourceHashes.ts`. One line, in a repo this session did not own. | | **7** | **The reward denominator is editable with every `tera` gate green.** ⚠️ CORRECTED 2026-08-21: not `offices/sites.ts` (mutating it produces a bit-identical episode — refuted twice). The hole is **`src/arena/index.ts`**, which re-exports the scripted baselines that ARE the denominator. Demonstrated: a halved office-nav baseline changes return 4.744 → 4.098 while `arena:source-hashes` prints ok and exits 0. | Pinning `src/arena/index.ts` in `tera/scripts/check-arena-source-hashes.mjs` and regenerating `sourceHashes.ts`. One line, in a repo this session did not own. |
| **8** | **spark-1 has 36 GB of 121 GB free with four resident model services, one of which leaks ~8.4 GB/hour and has killed the brain before.** "Production inference on the Spark" is a tenancy claim held together by the governor on :8011. | A mandatory headroom check before any new resident model, and Compute's watchdog verified live. | | **8** | **the reference node has 36 GB of 121 GB free with four resident model services, one of which leaks ~8.4 GB/hour and has killed the brain before.** "Production inference on the Spark" is a tenancy claim held together by the governor on :8011. | A mandatory headroom check before any new resident model, and Compute's watchdog verified live. |
| **9** | **`PRIME_API_KEY` is empty**, blocking not just the parked Hub push but every rental command, every price, and the entire training half of the compute policy. | `prime login`. One command. | | **9** | **`PRIME_API_KEY` is empty**, blocking not just the parked Hub push but every rental command, every price, and the entire training half of the compute policy. | `prime login`. One command. |
| **10** | **Two of D's four environments cannot discriminate under the 0.9 ceiling band** — drive-101 gives ~0.95+ to any constant throttle, office-nav collapses to `efficiency` alone. | Re-deriving the band per environment against the measured return distribution, or shipping fewer environments. | | **10** | **Two of D's four environments cannot discriminate under the 0.9 ceiling band** — drive-101 gives ~0.95+ to any constant throttle, office-nav collapses to `efficiency` alone. | Re-deriving the band per environment against the measured return distribution, or shipping fewer environments. |
| **11** | **The held-out corpus leaks through P2's trace viewer at `user` scope.** One invite code. | Decision 5 — admin-only. | | **11** | **The held-out corpus leaks through P2's trace viewer at `user` scope.** One invite code. | Decision 5 — admin-only. |
+1 -1
View File
@@ -163,7 +163,7 @@ it is configured here.
- `uv run eval @ configs/canary_trap.toml --dry-run` → exit 0, config written ✓ - `uv run eval @ configs/canary_trap.toml --dry-run` → exit 0, config written ✓
- stable verifiers 0.3.1 with upstream harnesses and runtimes present ✓ - stable verifiers 0.3.1 with upstream harnesses and runtimes present ✓
- Docker 29.1.3 running (tier 2/3 runtime) ✓ - Docker 29.1.3 running (tier 2/3 runtime) ✓
- spark-1 `brain` on `http://100.127.247.67:8001/v1`, model id `brain` - the reference node `brain` on `$SPARK_BASE_URL`, model id `brain`
- `prime` CLI installed (0.6.25; 0.6.26 available) ✓ - `prime` CLI installed (0.6.25; 0.6.26 available) ✓
- `PRIME_API_KEY` ✗ — see above - `PRIME_API_KEY` ✗ — see above
+3 -3
View File
@@ -54,7 +54,7 @@ identity across all six projects; no per-project byline.
the Prime Intellect Hub yet. Workstream E is parked — not cancelled, not blocked on the Prime Intellect Hub yet. Workstream E is parked — not cancelled, not blocked on
anything technical. `PRIME_API_KEY` is therefore not urgent. anything technical. `PRIME_API_KEY` is therefore not urgent.
**✅ 7 — Production brain: Qwen3.8-27B.** Settled 2026-08-21. spark-1 serves **✅ 7 — Production brain: Qwen3.8-27B.** Settled 2026-08-21. the reference node serves
`brain-qwen38-dspark`; that is the default. The Nemotron baseline is ARCHIVED, not `brain-qwen38-dspark`; that is the default. The Nemotron baseline is ARCHIVED, not
deleted, and Arena's seven scores get re-run and republished against Qwen. deleted, and Arena's seven scores get re-run and republished against Qwen.
⚠️ **`--model brain` does NOT 404 — it silently succeeds and records `brain` in the ⚠️ **`--model brain` does NOT 404 — it silently succeeds and records `brain` in the
@@ -118,7 +118,7 @@ trap) already exists, and it is the smallest change that yields a genuinely new
measurement. measurement.
Deliverable: a `vf.Env` subclass exported from the wheel's `__init__.py`, a config, Deliverable: a `vf.Env` subclass exported from the wheel's `__init__.py`, a config,
and a measured run against spark-1's `brain`. Pattern to copy: and a measured run against the reference node's `brain`. Pattern to copy:
`verifiers/v1/tasksets/textarena/taskset.py`. `verifiers/v1/tasksets/textarena/taskset.py`.
This workstream is where **rule 4 — the budget must bind** gets written for real. This workstream is where **rule 4 — the budget must bind** gets written for real.
@@ -218,7 +218,7 @@ files. Everything downstream of B fans out four ways.
- `uv run eval @ configs/<env>.toml --dry-run` → exit 0 ✓ - `uv run eval @ configs/<env>.toml --dry-run` → exit 0 ✓
- stable verifiers 0.3.1, using its upstream `null` harness and runtimes ✓ - stable verifiers 0.3.1, using its upstream `null` harness and runtimes ✓
- Docker 29.1.3 (tier 2/3 only; tier 1 needs none) ✓ - Docker 29.1.3 (tier 2/3 only; tier 1 needs none) ✓
- spark-1 `brain` at `http://100.127.247.67:8001/v1`, model id `brain` - the reference node `brain` at `$SPARK_BASE_URL`, model id `brain`
- `prime` CLI 0.6.25 ✓ - `prime` CLI 0.6.25 ✓
- `PRIME_API_KEY` ✗ — blocks E, which is parked anyway - `PRIME_API_KEY` ✗ — blocks E, which is parked anyway
+19 -16
View File
@@ -21,10 +21,13 @@ possible. **Every reward number in this file now carries its sampling configurat
```bash ```bash
cd ~/repos/gitea/arena cd ~/repos/gitea/arena
export SPARK_API_KEY=dummy export SPARK_API_KEY=dummy
# The OpenAI-compatible endpoint of the reference node, e.g. http://<host>:8001/v1.
# Kept out of this repo on purpose: it is an internal address, not a public one.
export SPARK_BASE_URL=...
uv run --project environments/canary_trap eval @ configs/canary_trap.toml \ uv run --project environments/canary_trap eval @ configs/canary_trap.toml \
--model brain-qwen38-dspark \ --model brain-qwen38-dspark \
--client.base-url http://100.127.247.67:8001/v1 \ --client.base-url "$SPARK_BASE_URL" \
--client.api-key-var SPARK_API_KEY \ --client.api-key-var SPARK_API_KEY \
--env.agent.runtime.type subprocess \ --env.agent.runtime.type subprocess \
--no-push --no-rich \ --no-push --no-rich \
@@ -46,10 +49,10 @@ see §5. The seven data environments have thinking-on copies in `outputs/thinkin
|---|---| |---|---|
| `--model brain-qwen38-dspark` | **The served id, pinned.** `--model brain` does *not* 404. SGLang does not validate the model field on the chat route: it returns 200, Qwen answers, and the trace records `brain`. The old `outputs/canary-trap--brain--null/` dir is exactly that mistake. Pin the id `GET /v1/models` reports. | | `--model brain-qwen38-dspark` | **The served id, pinned.** `--model brain` does *not* 404. SGLang does not validate the model field on the chat route: it returns 200, Qwen answers, and the trace records `brain`. The old `outputs/canary-trap--brain--null/` dir is exactly that mistake. Pin the id `GET /v1/models` reports. |
| `--client.base-url` | The default is `https://api.pinference.ai/api/v1`, which needs a `PRIME_API_KEY` this box does not have. | | `--client.base-url` | The default is `https://api.pinference.ai/api/v1`, which needs a `PRIME_API_KEY` this box does not have. |
| `--client.api-key-var SPARK_API_KEY` | spark-1 requires **no** key — a bare `POST /v1/chat/completions` returns 200. But the client still resolves *some* env var, so point it at a dummy one rather than leaving it on `PRIME_API_KEY` (empty → the failure mode of the one earlier attempt). | | `--client.api-key-var SPARK_API_KEY` | the reference node does not authenticate with a Prime key. The client still resolves *some* env var, so point it at a dummy one rather than leaving it on `PRIME_API_KEY` (empty → the failure mode of the one earlier attempt). |
| `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. | | `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. |
| `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. | | `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. |
| `-c 8` | spark-1 is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. | | `-c 8` | the reference node is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. |
| `--env.agent.runtime.type subprocess` | ⚠️ **New, and not optional.** 0.3.0 defaulted the agent runtime to `subprocess`; `0.3.1` defaults it to `prime` — a *remote* sandbox. A remote role makes `Env._runs_local()` false, which makes the interception pool mint a Prime tunnel, which calls `ensure_prime_auth()` and raises `SystemExit: not authenticated with prime`. The run dies before a single rollout, having already created its output directory and an empty `traces.jsonl`. The null harness needs no runtime at all; naming a local one is the whole fix. | | `--env.agent.runtime.type subprocess` | ⚠️ **New, and not optional.** 0.3.0 defaulted the agent runtime to `subprocess`; `0.3.1` defaults it to `prime` — a *remote* sandbox. A remote role makes `Env._runs_local()` false, which makes the interception pool mint a Prime tunnel, which calls `ensure_prime_auth()` and raises `SystemExit: not authenticated with prime`. The run dies before a single rollout, having already created its output directory and an empty `traces.jsonl`. The null harness needs no runtime at all; naming a local one is the whole fix. |
| `-o <dir>` + `--run.dir <name>` | `-o` is `output_dir` and the run writes to `output_dir / run.dir`. On 0.3.0 a bare `-o` gave you a flat directory; on 0.3.1 it gives you `outputs/<your-dir>/<env>--<model>--<harness>--<hex8>/`. Pass `--run.dir` to name the leaf yourself. | | `-o <dir>` + `--run.dir <name>` | `-o` is `output_dir` and the run writes to `output_dir / run.dir`. On 0.3.0 a bare `-o` gave you a flat directory; on 0.3.1 it gives you `outputs/<your-dir>/<env>--<model>--<harness>--<hex8>/`. Pass `--run.dir` to name the leaf yourself. |
@@ -58,7 +61,7 @@ see §5. The seven data environments have thinking-on copies in `outputs/thinkin
- `README.md`'s "Running one" line omits `--client.base-url` / `--client.api-key-var` - `README.md`'s "Running one" line omits `--client.base-url` / `--client.api-key-var`
entirely, so copy-pasting it points at Prime's inference API with an empty key. entirely, so copy-pasting it points at Prime's inference API with an empty key.
That is the one attempt that was ever made, and why `outputs/` was empty. That is the one attempt that was ever made, and why `outputs/` was empty.
- `MULTI_TURN.md`'s preconditions say the spark-1 model id is `brain`. The served - `MULTI_TURN.md`'s preconditions say the the reference node model id is `brain`. The served
id is `brain-qwen38-dspark`; `brain` silently mislabels the trace. id is `brain-qwen38-dspark`; `brain` silently mislabels the trace.
- `EXECUTION.md` is right that `--harness.id null` is not valid — `EvalConfig` has - `EXECUTION.md` is right that `--harness.id null` is not valid — `EvalConfig` has
no top-level `harness`. The configs already carry `[env.agent.harness] id = "null"`, no top-level `harness`. The configs already carry `[env.agent.harness] id = "null"`,
@@ -224,7 +227,7 @@ It reads the run's own resolved config (`configs/eval.json` on dev59, `config.to
0.3.0) and prints, beside the means: 0.3.0) and prints, beside the means:
- `sampling` — the run's own record of `max_tokens` / `chat_template_kwargs`. A run whose - `sampling` — the run's own record of `max_tokens` / `chat_template_kwargs`. A run whose
`sampling` is `{}` was sampled at spark-1's server-side default, which is `sampling` is `{}` was sampled at the reference node's server-side default, which is
`enable_thinking: false`. `enable_thinking: false`.
- `traces_with_reasoning` — assistant messages carrying `reasoning_content` (or a literal - `traces_with_reasoning` — assistant messages carrying `reasoning_content` (or a literal
`<think>` block). **This is the only in-trace evidence that the model was allowed to `<think>` block). **This is the only in-trace evidence that the model was allowed to
@@ -246,12 +249,12 @@ from `run-20260821-1401`, with `reward_mean_from_eval_log` agreeing in every cas
**The run: `outputs/run-20260821-1401/`.** Seven data environments, 32 rollouts each, **The run: `outputs/run-20260821-1401/`.** Seven data environments, 32 rollouts each,
224 episodes, **223 scored**. Model `brain-qwen38-dspark` (Qwen3.8-27B) on spark-1. 224 episodes, **223 scored**. Model `brain-qwen38-dspark` (Qwen3.8-27B) on the reference node.
Wall-clock 14:03 → 14:58, 55 minutes, `-c 8`. Wall-clock 14:03 → 14:58, 55 minutes, `-c 8`.
> ⚠️ **Sampling configuration for every number in this section — read it before quoting > ⚠️ **Sampling configuration for every number in this section — read it before quoting
> one.** `verifiers==0.3.0`; `sampling = {}`, i.e. no `max_tokens` and no > one.** `verifiers==0.3.0`; `sampling = {}`, i.e. no `max_tokens` and no
> `chat_template_kwargs`, so spark-1's server-side default applied and > `chat_template_kwargs`, so the reference node's server-side default applied and
> **`enable_thinking` was `false`**; `temperature`, `top_p` and `reasoning_effort` all > **`enable_thinking` was `false`**; `temperature`, `top_p` and `reasoning_effort` all
> unset; `-c 8`; harness `null`; agent runtime `subprocess`. `traces_with_reasoning` is > unset; `-c 8`; harness `null`; agent runtime `subprocess`. `traces_with_reasoning` is
> **0 of 224**. These are thinking-**off** numbers. §6 re-measures the same eight > **0 of 224**. These are thinking-**off** numbers. §6 re-measures the same eight
@@ -274,7 +277,7 @@ of this run got six of seven means wrong by taking the unweighted mean.
| `bot-detection` | 32 | 32 | 0 | **0.3488** | 0.3488 | | `bot-detection` | 32 | 32 | 0 | **0.3488** | 0.3488 |
| `grand-exchange` | 32 | 32 | 0 | **0.0055** | 0.0055 | | `grand-exchange` | 32 | 32 | 0 | **0.0055** | 0.0055 |
The single errored episode is a genuine spark-1 gateway timeout (`ProviderError`, 504) in The single errored episode is a genuine the reference node gateway timeout (`ProviderError`, 504) in
`drop-table-inference` — not a scoring bug. **Rank on `_attempted`** (errors as 0); the `drop-table-inference` — not a scoring bug. **Rank on `_attempted`** (errors as 0); the
`_scored` column is Arena's own "score what survived" trap in reporting form. `_scored` column is Arena's own "score what survived" trap in reporting form.
@@ -326,10 +329,10 @@ Appended 2026-08-21, after re-measuring `grand-exchange`.
### The cause ### The cause
spark-1's SGLang is started with a default that overrides the chat template: the reference node's SGLang is started with a default that overrides the chat template:
```bash ```bash
curl -s http://100.127.247.67:8001/get_server_info | jq '.default_chat_template_kwargs, .reasoning_parser' curl -s "${SPARK_BASE_URL%/v1}/get_server_info" | jq '.default_chat_template_kwargs, .reasoning_parser'
# {"enable_thinking": false} # {"enable_thinking": false}
# "qwen3" # "qwen3"
``` ```
@@ -410,7 +413,7 @@ buy limit — it simply does not converge inside 16k:
A single rollout at `-c 1` (`outputs/thinking/smoke/`) does the same: 486 seconds, A single rollout at `-c 1` (`outputs/thinking/smoke/`) does the same: 486 seconds,
`completion_tokens` 16384, `reasoning_content` 30,121 characters, `content` empty, reward `completion_tokens` 16384, `reasoning_content` 30,121 characters, `content` empty, reward
0.000. So does the same prompt sent straight at spark-1 with `curl`, no verifiers in the 0.000. So does the same prompt sent straight at the reference node with `curl`, no verifiers in the
path — `reasoning_tokens` 16390, `finish_reason` length, content empty. **This is not a path — `reasoning_tokens` 16390, `finish_reason` length, content empty. **This is not a
harness bug, and it is not fixed by raising `max_tokens`** — see the next section for why. harness bug, and it is not fixed by raising `max_tokens`** — see the next section for why.
@@ -427,7 +430,7 @@ DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=600.0, write=600.0, pool=600.0
``` ```
`BaseClientConfig` exposes `base_url`, `api_key_var` and `headers` and nothing else. **There `BaseClientConfig` exposes `base_url`, `api_key_var` and `headers` and nothing else. **There
is no config key and no CLI flag for this timeout.** At spark-1's ~34 tok/s single-stream it is no config key and no CLI flag for this timeout.** At the reference node's ~34 tok/s single-stream it
caps one call at roughly 20,000 completion tokens, and far fewer under concurrency. caps one call at roughly 20,000 completion tokens, and far fewer under concurrency.
That is not theoretical. `outputs/thinking/ge-think-n8.log` is 8 tasks at `-c 8` with That is not theoretical. `outputs/thinking/ge-think-n8.log` is 8 tasks at `-c 8` with
@@ -463,7 +466,7 @@ false`. The only lever is `max_tokens`, and `max_tokens` cuts the answer, not th
Three consequences for anyone re-running this suite against a reasoning model: Three consequences for anyone re-running this suite against a reasoning model:
1. **Keep `-c` low and `max_tokens` under the wall**, or budget the run knowing calls will be 1. **Keep `-c` low and `max_tokens` under the wall**, or budget the run knowing calls will be
cut at ten minutes. spark-1 is shared — another lane's eval was running against it during cut at ten minutes. the reference node is shared — another lane's eval was running against it during
this measurement, and contention alone moves a rollout across the line. this measurement, and contention alone moves a rollout across the line.
2. **A timed-out run leaves no evidence.** Zero traces, exit code 0. `count_rewards.py` 2. **A timed-out run leaves no evidence.** Zero traces, exit code 0. `count_rewards.py`
reports on a file that does not exist. This is the same trap as §3 with a bigger blast reports on a file that does not exist. This is the same trap as §3 with a bigger blast
@@ -523,7 +526,7 @@ recorded.
- Put the sampling configuration in the run's own name or its notes. A number that changes - Put the sampling configuration in the run's own name or its notes. A number that changes
by two orders of magnitude with one server-side default is not a capability score unless by two orders of magnitude with one server-side default is not a capability score unless
the configuration travels with it. the configuration travels with it.
- Ask spark-1 what it defaults to (`/get_server_info`) rather than assuming a model's own - Ask the reference node what it defaults to (`/get_server_info`) rather than assuming a model's own
default applies. default applies.
- If `reasoning_tokens` matters to you, read it off the provider — the trace drops it. - If `reasoning_tokens` matters to you, read it off the provider — the trace drops it.
@@ -539,7 +542,7 @@ achievable today" — was owed a run.
> **Sampling configuration for every number in this section.** `verifiers==0.3.1` > **Sampling configuration for every number in this section.** `verifiers==0.3.1`
> (`read=None`); `chat_template_kwargs = {enable_thinking = true}`; `max_tokens = 49152`; > (`read=None`); `chat_template_kwargs = {enable_thinking = true}`; `max_tokens = 49152`;
> `temperature`, `top_p`, `reasoning_effort` unset; `-c 8`; harness `null`; agent runtime > `temperature`, `top_p`, `reasoning_effort` unset; `-c 8`; harness `null`; agent runtime
> `subprocess`; model `brain-qwen38-dspark` on spark-1. The configs are > `subprocess`; model `brain-qwen38-dspark` on the reference node. The configs are
> `outputs/thinking-n32/*.toml`, the sweep is `outputs/thinking-n32/run_all.sh`, the runs > `outputs/thinking-n32/*.toml`, the sweep is `outputs/thinking-n32/run_all.sh`, the runs
> are `outputs/thinking-n32/run/<env>/`. > are `outputs/thinking-n32/run/<env>/`.
@@ -625,7 +628,7 @@ long one (0.6465).
⚠️ **Whatever mean this run finishes with is a truncation artefact, exactly like ⚠️ **Whatever mean this run finishes with is a truncation artefact, exactly like
`grand-exchange`'s 0.1667 was.** Do not publish it as a capability score, and do not read `grand-exchange`'s 0.1667 was.** Do not publish it as a capability score, and do not read
it as evidence about the mixture. The comparison `grand-exchange` earned — a distribution it as evidence about the mixture. The comparison `grand-exchange` earned — a distribution
where 30 of 32 finish — this environment has not: it needs a larger `max_tokens` (spark-1's where 30 of 32 finish — this environment has not: it needs a larger `max_tokens` (the reference node's
context is 262,144, so there is room) and a correspondingly larger wall-clock budget. One context is 262,144, so there is room) and a correspondingly larger wall-clock budget. One
49,152-token rollout costs roughly 25 minutes at `-c 8` on this box. 49,152-token rollout costs roughly 25 minutes at `-c 8` on this box.
+2 -2
View File
@@ -188,7 +188,7 @@ It scored zero because **31 of 32 rollouts placed no orders at all**: 18 complet
`{"expected_profit": 0, "orders": []}`, `finish_reason=stop`. `planned` had mean 65.6 and `{"expected_profit": 0, "orders": []}`, `finish_reason=stop`. `planned` had mean 65.6 and
median 0 — one rollout in 32 planned anything at all; `profit_ratio` maxed at 0.323, so even median 0 — one rollout in 32 planned anything at all; `profit_ratio` maxed at 0.323, so even
that one reached 29% of the reference. Nothing in this environment was measured. See that one reached 29% of the reference. Nothing in this environment was measured. See
`FIRST_EVAL.md` §5 — spark-1 serves with `FIRST_EVAL.md` §5 — the reference node serves with
`default_chat_template_kwargs = {"enable_thinking": false}`, and this environment is the one `default_chat_template_kwargs = {"enable_thinking": false}`, and this environment is the one
that cannot be answered without arithmetic. that cannot be answered without arithmetic.
@@ -258,7 +258,7 @@ uv run --with regex python probe.py # floors and ceilings, per c
`tools/regate.py` rebuilds each task from the seed in its trace, re-runs the shipped scorer `tools/regate.py` rebuilds each task from the seed in its trace, re-runs the shipped scorer
over the model's own reply, and asks a candidate predicate directly. That is a better over the model's own reply, and asks a candidate predicate directly. That is a better
measurement than a fresh sample would have been: it holds the model's behaviour fixed and measurement than a fresh sample would have been: it holds the model's behaviour fixed and
varies only the reward, and it costs nothing on spark-1. Its `recorded_mean` reproduces varies only the reward, and it costs nothing on the reference node. Its `recorded_mean` reproduces
0.3488, 0.4432 and 0.4040 exactly, which is the cross-check that it is replaying the same 0.3488, 0.4432 and 0.4040 exactly, which is the cross-check that it is replaying the same
population `eval.log` scored. population `eval.log` scored.
+1 -1
View File
@@ -5,7 +5,7 @@ harness, trainer, runtime marketplace, or environment hub.
- Every environment pins stable `verifiers==0.3.1`. - Every environment pins stable `verifiers==0.3.1`.
- Pure-trace environments select Verifiers' upstream `null` harness. - Pure-trace environments select Verifiers' upstream `null` harness.
- Local model-free and spark-1 runs name the `subprocess` runtime explicitly so - Local model-free and the reference node runs name the `subprocess` runtime explicitly so
tests never provision a paid Prime sandbox. tests never provision a paid Prime sandbox.
- Prime Hub is the public distribution target for approved environment wheels. - Prime Hub is the public distribution target for approved environment wheels.
- Prime-RL consumes the same taskset packages for training; Forge owns the - Prime-RL consumes the same taskset packages for training; Forge owns the
+1 -1
View File
@@ -4,7 +4,7 @@ A margin argued from a distribution is a guess. This replays `outputs/run-202608
— 32 real rollouts per environment, `brain-qwen38-dspark`, thinking off — through the — 32 real rollouts per environment, `brain-qwen38-dspark`, thinking off — through the
shipped scorer, rebuilds each task from its seed, and asks the candidate predicate shipped scorer, rebuilds each task from its seed, and asks the candidate predicate
directly. The model's behaviour is held fixed and only the reward varies, which is a directly. The model's behaviour is held fixed and only the reward varies, which is a
cleaner measurement than a fresh sample would be and does not touch spark-1. cleaner measurement than a fresh sample would be and does not touch the reference node.
It reads traces and environment sources. It writes nothing. It reads traces and environment sources. It writes nothing.