docs: redact the reference node's address and hostname
The eval docs named spark-1 by fleet hostname and published its inference endpoint (a tailnet address and port) plus the fact that it answers without an API key. This repo is PUBLIC, so that was a standing disclosure of an internal, unauthenticated service to anonymous readers. Endpoints now come from $SPARK_BASE_URL, which the operator exports, and the host is referred to as "the reference node" -- the convention the compute repo's docs already use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012AaUFYkUTsJn1fnJ89qbvW
This commit is contained in:
+6
-6
@@ -79,7 +79,7 @@ Stands and is high-value: the endpoint inventory (29 routes, not 31), the 23 dea
|
|||||||
|
|
||||||
Corrected, blocking:
|
Corrected, blocking:
|
||||||
|
|
||||||
1. **"Ingest the runs already on amd-server" is impossible — there are none.** One `config.toml`, zero episodes, and that config points at `api.pinference.ai` with `PRIME_API_KEY`, which is empty. **Phase 1 must contain "execute one eval to completion against spark-1" as an explicit prerequisite**, or Arena's dashboard ships empty.
|
1. **"Ingest the runs already on amd-server" is impossible — there are none.** One `config.toml`, zero episodes, and that config points at `api.pinference.ai` with `PRIME_API_KEY`, which is empty. **Phase 1 must contain "execute one eval to completion against the reference node" as an explicit prerequisite**, or Arena's dashboard ships empty.
|
||||||
2. **The leaderboard is operator-gameable and mis-specified.** (a) The server must recompute every aggregate in SQL from the `arena_episodes` it just inserted, never trust the pusher's numbers. (b) Ranking on a mean over *scored* episodes is Arena's own "score what survived" trap. Store `reward_mean_scored` **and** `reward_mean_attempted` (errors as 0), rank on attempted, refuse to rank below a scored-fraction floor.
|
2. **The leaderboard is operator-gameable and mis-specified.** (a) The server must recompute every aggregate in SQL from the `arena_episodes` it just inserted, never trust the pusher's numbers. (b) Ranking on a mean over *scored* episodes is Arena's own "score what survived" trap. Store `reward_mean_scored` **and** `reward_mean_attempted` (errors as 0), rank on attempted, refuse to rank below a scored-fraction floor.
|
||||||
3. **`arena_episodes` flattens a list.** `Episode.traces` is a list; `reward`/`rewards`/`stop_condition`/`last_reply` are per-**Trace**. Primary key must be `(run_id, episode_id, trace_id)` with an `agent_name` column — otherwise the schema is wrong for `grand-exchange-live` (simulator as second role) and for `redaction-pressure` (genuine two-agent).
|
3. **`arena_episodes` flattens a list.** `Episode.traces` is a list; `reward`/`rewards`/`stop_condition`/`last_reply` are per-**Trace**. Primary key must be `(run_id, episode_id, trace_id)` with an `agent_name` column — otherwise the schema is wrong for `grand-exchange-live` (simulator as second role) and for `redaction-pressure` (genuine two-agent).
|
||||||
4. **The trace viewer publishes the held-out slice.** `GET /api/arena/runs/:id/episodes/:eid` at `user` scope returns full transcripts, which contain held-out task data and, for oracle runs, the answers. One leaked invite publishes the held-out corpus. **Transcripts are admin-only or redacted.** This settles the spec's open question on read scope.
|
4. **The trace viewer publishes the held-out slice.** `GET /api/arena/runs/:id/episodes/:eid` at `user` scope returns full transcripts, which contain held-out task data and, for oracle runs, the answers. One leaked invite publishes the held-out corpus. **Transcripts are admin-only or redacted.** This settles the spec's open question on read scope.
|
||||||
@@ -115,7 +115,7 @@ Corrected:
|
|||||||
|
|
||||||
### C-compute — rent to train, Spark to serve · feasible; **its headline correction is false**
|
### C-compute — rent to train, Spark to serve · feasible; **its headline correction is false**
|
||||||
|
|
||||||
Stands and is the most immediately actionable: the policy itself; the measured spark-1 tenancy (121 GB total / 84 GB used / 36 GB free, SGLang + Chatterbox + vox + ASR co-resident, Chatterbox's ~8.4 GB/hr leak having killed brain before); `prime login` as step zero because `api_key` is empty (verified); **Arena → prime-rl needs no adapter** — `EnvConfig` is `SerializeAsAny[vf.EnvConfig]` and the taskset tree is the same shape, prefixed; the seat-name-is-a-TOML-key trap; Hosted Training being unavailable while E is parked (`rl.py:1522-1524` — bare v1 ids do not resolve through the Hub); the sm_121 lock contradiction in Forge; `TRAINERS = ("axolotl","trl")`; **no price is hardcoded anywhere and none should be invented.**
|
Stands and is the most immediately actionable: the policy itself; the measured the reference node tenancy (121 GB total / 84 GB used / 36 GB free, SGLang + Chatterbox + vox + ASR co-resident, Chatterbox's ~8.4 GB/hr leak having killed brain before); `prime login` as step zero because `api_key` is empty (verified); **Arena → prime-rl needs no adapter** — `EnvConfig` is `SerializeAsAny[vf.EnvConfig]` and the taskset tree is the same shape, prefixed; the seat-name-is-a-TOML-key trap; Hosted Training being unavailable while E is parked (`rl.py:1522-1524` — bare v1 ids do not resolve through the Hub); the sm_121 lock contradiction in Forge; `TRAINERS = ("axolotl","trl")`; **no price is hardcoded anywhere and none should be invented.**
|
||||||
|
|
||||||
Corrected, blocking:
|
Corrected, blocking:
|
||||||
|
|
||||||
@@ -141,7 +141,7 @@ Corrected, blocking:
|
|||||||
| **1** | **B1** — discovery de-hardcode + timeout + step name + `uv.lock` cleanup | `arena` | nothing |
|
| **1** | **B1** — discovery de-hardcode + timeout + step name + `uv.lock` cleanup | `arena` | nothing |
|
||||||
| **2** | **A step 2** — `live.py` engine + unit tests + **re-measure the ladder** | `arena` | nothing (own worktree) |
|
| **2** | **A step 2** — `live.py` engine + unit tests + **re-measure the ladder** | `arena` | nothing (own worktree) |
|
||||||
| **3** | **D step 1** — worker, vendored TS, hashes, `test_replay.py` | `arena` (new dir only) | nothing |
|
| **3** | **D step 1** — worker, vendored TS, hashes, `test_replay.py` | `arena` (new dir only) | nothing |
|
||||||
| **4** | **Compute Policy step 0** — `prime login`, catalogue + prices, doc the corrected spark-1 facts | none / `arena/docs` | nothing |
|
| **4** | **Compute Policy step 0** — `prime login`, catalogue + prices, doc the corrected the reference node facts | none / `arena/docs` | nothing |
|
||||||
| **5** | **P1** — Caddy host, host-aware routing, AppShell, Overview/Worlds/Org/Inbox | `lumbridge-v4` | nothing |
|
| **5** | **P1** — Caddy host, host-aware routing, AppShell, Overview/Worlds/Org/Inbox | `lumbridge-v4` | nothing |
|
||||||
| **6** | **W step 0** — the five factual corrections + `eval-results.json` decision | `lumbridge-v4` | nothing |
|
| **6** | **W step 0** — the five factual corrections + `eval-results.json` decision | `lumbridge-v4` | nothing |
|
||||||
|
|
||||||
@@ -209,7 +209,7 @@ Two edges that were wrong in the source specs and are now correct:
|
|||||||
1. **B1** (arena): tomllib discovery, `@probes` registry, timeout 180, `uv.lock` untracking, step rename. *Gate: `probe.py` still prints seven `ok` rows; CI green with zero list edits.*
|
1. **B1** (arena): tomllib discovery, `@probes` registry, timeout 180, `uv.lock` untracking, step rename. *Gate: `probe.py` still prints seven `ok` rows; CI green with zero list edits.*
|
||||||
2. **A steps 1-2 + re-measurement** (arena, worktree): frozen generator copy + equality assertion; `LiveBook`/`Leg`/`Ledger` with unit tests; **then re-run the full policy ladder and publish the corrected numbers.** *Gate: the ladder reconciles with the reward formula; `horizon`, `FREEZE_RANGE`, `TARGET_SHARE`, `TURNS` are chosen from that measurement, not from the spec.*
|
2. **A steps 1-2 + re-measurement** (arena, worktree): frozen generator copy + equality assertion; `LiveBook`/`Leg`/`Ledger` with unit tests; **then re-run the full policy ladder and publish the corrected numbers.** *Gate: the ladder reconciles with the reward formula; `horizon`, `FREEZE_RANGE`, `TARGET_SHARE`, `TURNS` are chosen from that measurement, not from the spec.*
|
||||||
3. **D step 1** (arena, new dir): worker + vendored TS + `hashes.py` + `sync_tera.py` + `test_replay.py`. *Gate: 32/32 replay exact, tampering rejected, hash check green — and a decision on the 4 unpinned files.*
|
3. **D step 1** (arena, new dir): worker + vendored TS + `hashes.py` + `sync_tera.py` + `test_replay.py`. *Gate: 32/32 replay exact, tampering rejected, hash check green — and a decision on the 4 unpinned files.*
|
||||||
4. **Compute Policy step 0**: `prime login`; record real `availability` output and prices; write `arena/docs/COMPUTE_POLICY.md` with the corrected spark-1 facts (`brain-qwen38-dspark`, silent-wrong-model hazard, 36 GB headroom).
|
4. **Compute Policy step 0**: `prime login`; record real `availability` output and prices; write `arena/docs/COMPUTE_POLICY.md` with the corrected the reference node facts (`brain-qwen38-dspark`, silent-wrong-model hazard, 36 GB headroom).
|
||||||
5. **P1** (lumbridge-v4): Caddy `app.` block with `bind 10.0.0.2` and the non-empty-body check; host-aware `App.tsx`; `renderAppShell` + `sendShell` signature change + `seo.test.ts` + `spa-fallback.test.ts`; AppShell; Overview / Worlds / Org / Inbox; delete the 23 dead `api.ts` methods.
|
5. **P1** (lumbridge-v4): Caddy `app.` block with `bind 10.0.0.2` and the non-empty-body check; host-aware `App.tsx`; `renderAppShell` + `sendShell` signature change + `seo.test.ts` + `spa-fallback.test.ts`; AppShell; Overview / Worlds / Org / Inbox; delete the 23 dead `api.ts` methods.
|
||||||
6. **W step 0** (lumbridge-v4): the five factual corrections — `arena.tsx:330` "Nine tasks", `landing.tsx` card 02, the 4-row probe table → 7 rows, `office-jobs-v1` deleted, `arena.tsx:519` "no model scores published". Ship the Nemotron run into `eval-results.json`.
|
6. **W step 0** (lumbridge-v4): the five factual corrections — `arena.tsx:330` "Nine tasks", `landing.tsx` card 02, the 4-row probe table → 7 rows, `office-jobs-v1` deleted, `arena.tsx:519` "no model scores published". Ship the Nemotron run into `eval-results.json`.
|
||||||
|
|
||||||
@@ -218,7 +218,7 @@ Two edges that were wrong in the source specs and are now correct:
|
|||||||
### Phase 2
|
### Phase 2
|
||||||
|
|
||||||
7. **A steps 3-9** → `grand-exchange-live` ships with its probe row (needs B1 merged).
|
7. **A steps 3-9** → `grand-exchange-live` ships with its probe row (needs B1 merged).
|
||||||
8. **One real eval run** against spark-1, pinned to `brain-qwen38-dspark`, with the empty-rewards count checked. *This is a hard prerequisite for P2 and for any site number.*
|
8. **One real eval run** against the reference node, pinned to `brain-qwen38-dspark`, with the empty-rewards count checked. *This is a hard prerequisite for P2 and for any site number.*
|
||||||
9. **D crow-nav** end to end (role named `agent`, `vf.Taskset[...]` generic form, `oracle` op in the worker, `gate` as a multiplier).
|
9. **D crow-nav** end to end (role named `agent`, `vf.Taskset[...]` generic form, `oracle` op in the worker, `gate` as a multiplier).
|
||||||
10. **W /arena rebuild** + `/bench` on the new primitives.
|
10. **W /arena rebuild** + `/bench` on the new primitives.
|
||||||
11. **First rented run**: pod create with an attached disk, `install.sh`, wheel install, baseline eval, `uv run rl`, re-eval, before/after table.
|
11. **First rented run**: pod create with an attached disk, `install.sh`, wheel install, baseline eval, `uv run rl`, re-eval, before/after table.
|
||||||
@@ -263,7 +263,7 @@ Two edges that were wrong in the source specs and are now correct:
|
|||||||
| **5** | **Silent wrong-model comparison.** `--model brain` returns 200 from a box serving Qwen3.8-27B. Re-running the Nemotron baseline today produces different-model numbers under an identical command, and they will be tabulated as comparable. | Pinning `brain-qwen38-dspark` (and the scene) in every command in `README.md`, `PLAN.md`, `MULTI_TURN.md` and `COMPUTE_POLICY.md`. Trivial; do it in Phase 1. |
|
| **5** | **Silent wrong-model comparison.** `--model brain` returns 200 from a box serving Qwen3.8-27B. Re-running the Nemotron baseline today produces different-model numbers under an identical command, and they will be tabulated as comparable. | Pinning `brain-qwen38-dspark` (and the scene) in every command in `README.md`, `PLAN.md`, `MULTI_TURN.md` and `COMPUTE_POLICY.md`. Trivial; do it in Phase 1. |
|
||||||
| **6** | **`probe.py`, `tests/test_probe.py` and `ci.yml` are edited by five of six workstreams, in a repo that already took divergent concurrent pushes.** | B1, first, before anything else touches `arena`. |
|
| **6** | **`probe.py`, `tests/test_probe.py` and `ci.yml` are edited by five of six workstreams, in a repo that already took divergent concurrent pushes.** | B1, first, before anything else touches `arena`. |
|
||||||
| **7** | **The reward denominator is editable with every `tera` gate green.** ⚠️ CORRECTED 2026-08-21: not `offices/sites.ts` (mutating it produces a bit-identical episode — refuted twice). The hole is **`src/arena/index.ts`**, which re-exports the scripted baselines that ARE the denominator. Demonstrated: a halved office-nav baseline changes return 4.744 → 4.098 while `arena:source-hashes` prints ok and exits 0. | Pinning `src/arena/index.ts` in `tera/scripts/check-arena-source-hashes.mjs` and regenerating `sourceHashes.ts`. One line, in a repo this session did not own. |
|
| **7** | **The reward denominator is editable with every `tera` gate green.** ⚠️ CORRECTED 2026-08-21: not `offices/sites.ts` (mutating it produces a bit-identical episode — refuted twice). The hole is **`src/arena/index.ts`**, which re-exports the scripted baselines that ARE the denominator. Demonstrated: a halved office-nav baseline changes return 4.744 → 4.098 while `arena:source-hashes` prints ok and exits 0. | Pinning `src/arena/index.ts` in `tera/scripts/check-arena-source-hashes.mjs` and regenerating `sourceHashes.ts`. One line, in a repo this session did not own. |
|
||||||
| **8** | **spark-1 has 36 GB of 121 GB free with four resident model services, one of which leaks ~8.4 GB/hour and has killed the brain before.** "Production inference on the Spark" is a tenancy claim held together by the governor on :8011. | A mandatory headroom check before any new resident model, and Compute's watchdog verified live. |
|
| **8** | **the reference node has 36 GB of 121 GB free with four resident model services, one of which leaks ~8.4 GB/hour and has killed the brain before.** "Production inference on the Spark" is a tenancy claim held together by the governor on :8011. | A mandatory headroom check before any new resident model, and Compute's watchdog verified live. |
|
||||||
| **9** | **`PRIME_API_KEY` is empty**, blocking not just the parked Hub push but every rental command, every price, and the entire training half of the compute policy. | `prime login`. One command. |
|
| **9** | **`PRIME_API_KEY` is empty**, blocking not just the parked Hub push but every rental command, every price, and the entire training half of the compute policy. | `prime login`. One command. |
|
||||||
| **10** | **Two of D's four environments cannot discriminate under the 0.9 ceiling band** — drive-101 gives ~0.95+ to any constant throttle, office-nav collapses to `efficiency` alone. | Re-deriving the band per environment against the measured return distribution, or shipping fewer environments. |
|
| **10** | **Two of D's four environments cannot discriminate under the 0.9 ceiling band** — drive-101 gives ~0.95+ to any constant throttle, office-nav collapses to `efficiency` alone. | Re-deriving the band per environment against the measured return distribution, or shipping fewer environments. |
|
||||||
| **11** | **The held-out corpus leaks through P2's trace viewer at `user` scope.** One invite code. | Decision 5 — admin-only. |
|
| **11** | **The held-out corpus leaks through P2's trace viewer at `user` scope.** One invite code. | Decision 5 — admin-only. |
|
||||||
|
|||||||
+1
-1
@@ -163,7 +163,7 @@ it is configured here.
|
|||||||
- `uv run eval @ configs/canary_trap.toml --dry-run` → exit 0, config written ✓
|
- `uv run eval @ configs/canary_trap.toml --dry-run` → exit 0, config written ✓
|
||||||
- stable verifiers 0.3.1 with upstream harnesses and runtimes present ✓
|
- stable verifiers 0.3.1 with upstream harnesses and runtimes present ✓
|
||||||
- Docker 29.1.3 running (tier 2/3 runtime) ✓
|
- Docker 29.1.3 running (tier 2/3 runtime) ✓
|
||||||
- spark-1 `brain` on `http://100.127.247.67:8001/v1`, model id `brain` ✓
|
- the reference node `brain` on `$SPARK_BASE_URL`, model id `brain` ✓
|
||||||
- `prime` CLI installed (0.6.25; 0.6.26 available) ✓
|
- `prime` CLI installed (0.6.25; 0.6.26 available) ✓
|
||||||
- `PRIME_API_KEY` ✗ — see above
|
- `PRIME_API_KEY` ✗ — see above
|
||||||
|
|
||||||
|
|||||||
@@ -54,7 +54,7 @@ identity across all six projects; no per-project byline.
|
|||||||
the Prime Intellect Hub yet. Workstream E is parked — not cancelled, not blocked on
|
the Prime Intellect Hub yet. Workstream E is parked — not cancelled, not blocked on
|
||||||
anything technical. `PRIME_API_KEY` is therefore not urgent.
|
anything technical. `PRIME_API_KEY` is therefore not urgent.
|
||||||
|
|
||||||
**✅ 7 — Production brain: Qwen3.8-27B.** Settled 2026-08-21. spark-1 serves
|
**✅ 7 — Production brain: Qwen3.8-27B.** Settled 2026-08-21. the reference node serves
|
||||||
`brain-qwen38-dspark`; that is the default. The Nemotron baseline is ARCHIVED, not
|
`brain-qwen38-dspark`; that is the default. The Nemotron baseline is ARCHIVED, not
|
||||||
deleted, and Arena's seven scores get re-run and republished against Qwen.
|
deleted, and Arena's seven scores get re-run and republished against Qwen.
|
||||||
⚠️ **`--model brain` does NOT 404 — it silently succeeds and records `brain` in the
|
⚠️ **`--model brain` does NOT 404 — it silently succeeds and records `brain` in the
|
||||||
@@ -118,7 +118,7 @@ trap) already exists, and it is the smallest change that yields a genuinely new
|
|||||||
measurement.
|
measurement.
|
||||||
|
|
||||||
Deliverable: a `vf.Env` subclass exported from the wheel's `__init__.py`, a config,
|
Deliverable: a `vf.Env` subclass exported from the wheel's `__init__.py`, a config,
|
||||||
and a measured run against spark-1's `brain`. Pattern to copy:
|
and a measured run against the reference node's `brain`. Pattern to copy:
|
||||||
`verifiers/v1/tasksets/textarena/taskset.py`.
|
`verifiers/v1/tasksets/textarena/taskset.py`.
|
||||||
|
|
||||||
This workstream is where **rule 4 — the budget must bind** gets written for real.
|
This workstream is where **rule 4 — the budget must bind** gets written for real.
|
||||||
@@ -218,7 +218,7 @@ files. Everything downstream of B fans out four ways.
|
|||||||
- `uv run eval @ configs/<env>.toml --dry-run` → exit 0 ✓
|
- `uv run eval @ configs/<env>.toml --dry-run` → exit 0 ✓
|
||||||
- stable verifiers 0.3.1, using its upstream `null` harness and runtimes ✓
|
- stable verifiers 0.3.1, using its upstream `null` harness and runtimes ✓
|
||||||
- Docker 29.1.3 (tier 2/3 only; tier 1 needs none) ✓
|
- Docker 29.1.3 (tier 2/3 only; tier 1 needs none) ✓
|
||||||
- spark-1 `brain` at `http://100.127.247.67:8001/v1`, model id `brain` ✓
|
- the reference node `brain` at `$SPARK_BASE_URL`, model id `brain` ✓
|
||||||
- `prime` CLI 0.6.25 ✓
|
- `prime` CLI 0.6.25 ✓
|
||||||
- `PRIME_API_KEY` ✗ — blocks E, which is parked anyway
|
- `PRIME_API_KEY` ✗ — blocks E, which is parked anyway
|
||||||
|
|
||||||
|
|||||||
+19
-16
@@ -21,10 +21,13 @@ possible. **Every reward number in this file now carries its sampling configurat
|
|||||||
```bash
|
```bash
|
||||||
cd ~/repos/gitea/arena
|
cd ~/repos/gitea/arena
|
||||||
export SPARK_API_KEY=dummy
|
export SPARK_API_KEY=dummy
|
||||||
|
# The OpenAI-compatible endpoint of the reference node, e.g. http://<host>:8001/v1.
|
||||||
|
# Kept out of this repo on purpose: it is an internal address, not a public one.
|
||||||
|
export SPARK_BASE_URL=...
|
||||||
|
|
||||||
uv run --project environments/canary_trap eval @ configs/canary_trap.toml \
|
uv run --project environments/canary_trap eval @ configs/canary_trap.toml \
|
||||||
--model brain-qwen38-dspark \
|
--model brain-qwen38-dspark \
|
||||||
--client.base-url http://100.127.247.67:8001/v1 \
|
--client.base-url "$SPARK_BASE_URL" \
|
||||||
--client.api-key-var SPARK_API_KEY \
|
--client.api-key-var SPARK_API_KEY \
|
||||||
--env.agent.runtime.type subprocess \
|
--env.agent.runtime.type subprocess \
|
||||||
--no-push --no-rich \
|
--no-push --no-rich \
|
||||||
@@ -46,10 +49,10 @@ see §5. The seven data environments have thinking-on copies in `outputs/thinkin
|
|||||||
|---|---|
|
|---|---|
|
||||||
| `--model brain-qwen38-dspark` | **The served id, pinned.** `--model brain` does *not* 404. SGLang does not validate the model field on the chat route: it returns 200, Qwen answers, and the trace records `brain`. The old `outputs/canary-trap--brain--null/` dir is exactly that mistake. Pin the id `GET /v1/models` reports. |
|
| `--model brain-qwen38-dspark` | **The served id, pinned.** `--model brain` does *not* 404. SGLang does not validate the model field on the chat route: it returns 200, Qwen answers, and the trace records `brain`. The old `outputs/canary-trap--brain--null/` dir is exactly that mistake. Pin the id `GET /v1/models` reports. |
|
||||||
| `--client.base-url` | The default is `https://api.pinference.ai/api/v1`, which needs a `PRIME_API_KEY` this box does not have. |
|
| `--client.base-url` | The default is `https://api.pinference.ai/api/v1`, which needs a `PRIME_API_KEY` this box does not have. |
|
||||||
| `--client.api-key-var SPARK_API_KEY` | spark-1 requires **no** key — a bare `POST /v1/chat/completions` returns 200. But the client still resolves *some* env var, so point it at a dummy one rather than leaving it on `PRIME_API_KEY` (empty → the failure mode of the one earlier attempt). |
|
| `--client.api-key-var SPARK_API_KEY` | the reference node does not authenticate with a Prime key. The client still resolves *some* env var, so point it at a dummy one rather than leaving it on `PRIME_API_KEY` (empty → the failure mode of the one earlier attempt). |
|
||||||
| `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. |
|
| `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. |
|
||||||
| `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. |
|
| `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. |
|
||||||
| `-c 8` | spark-1 is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. |
|
| `-c 8` | the reference node is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. |
|
||||||
| `--env.agent.runtime.type subprocess` | ⚠️ **New, and not optional.** 0.3.0 defaulted the agent runtime to `subprocess`; `0.3.1` defaults it to `prime` — a *remote* sandbox. A remote role makes `Env._runs_local()` false, which makes the interception pool mint a Prime tunnel, which calls `ensure_prime_auth()` and raises `SystemExit: not authenticated with prime`. The run dies before a single rollout, having already created its output directory and an empty `traces.jsonl`. The null harness needs no runtime at all; naming a local one is the whole fix. |
|
| `--env.agent.runtime.type subprocess` | ⚠️ **New, and not optional.** 0.3.0 defaulted the agent runtime to `subprocess`; `0.3.1` defaults it to `prime` — a *remote* sandbox. A remote role makes `Env._runs_local()` false, which makes the interception pool mint a Prime tunnel, which calls `ensure_prime_auth()` and raises `SystemExit: not authenticated with prime`. The run dies before a single rollout, having already created its output directory and an empty `traces.jsonl`. The null harness needs no runtime at all; naming a local one is the whole fix. |
|
||||||
| `-o <dir>` + `--run.dir <name>` | `-o` is `output_dir` and the run writes to `output_dir / run.dir`. On 0.3.0 a bare `-o` gave you a flat directory; on 0.3.1 it gives you `outputs/<your-dir>/<env>--<model>--<harness>--<hex8>/`. Pass `--run.dir` to name the leaf yourself. |
|
| `-o <dir>` + `--run.dir <name>` | `-o` is `output_dir` and the run writes to `output_dir / run.dir`. On 0.3.0 a bare `-o` gave you a flat directory; on 0.3.1 it gives you `outputs/<your-dir>/<env>--<model>--<harness>--<hex8>/`. Pass `--run.dir` to name the leaf yourself. |
|
||||||
|
|
||||||
@@ -58,7 +61,7 @@ see §5. The seven data environments have thinking-on copies in `outputs/thinkin
|
|||||||
- `README.md`'s "Running one" line omits `--client.base-url` / `--client.api-key-var`
|
- `README.md`'s "Running one" line omits `--client.base-url` / `--client.api-key-var`
|
||||||
entirely, so copy-pasting it points at Prime's inference API with an empty key.
|
entirely, so copy-pasting it points at Prime's inference API with an empty key.
|
||||||
That is the one attempt that was ever made, and why `outputs/` was empty.
|
That is the one attempt that was ever made, and why `outputs/` was empty.
|
||||||
- `MULTI_TURN.md`'s preconditions say the spark-1 model id is `brain`. The served
|
- `MULTI_TURN.md`'s preconditions say the the reference node model id is `brain`. The served
|
||||||
id is `brain-qwen38-dspark`; `brain` silently mislabels the trace.
|
id is `brain-qwen38-dspark`; `brain` silently mislabels the trace.
|
||||||
- `EXECUTION.md` is right that `--harness.id null` is not valid — `EvalConfig` has
|
- `EXECUTION.md` is right that `--harness.id null` is not valid — `EvalConfig` has
|
||||||
no top-level `harness`. The configs already carry `[env.agent.harness] id = "null"`,
|
no top-level `harness`. The configs already carry `[env.agent.harness] id = "null"`,
|
||||||
@@ -224,7 +227,7 @@ It reads the run's own resolved config (`configs/eval.json` on dev59, `config.to
|
|||||||
0.3.0) and prints, beside the means:
|
0.3.0) and prints, beside the means:
|
||||||
|
|
||||||
- `sampling` — the run's own record of `max_tokens` / `chat_template_kwargs`. A run whose
|
- `sampling` — the run's own record of `max_tokens` / `chat_template_kwargs`. A run whose
|
||||||
`sampling` is `{}` was sampled at spark-1's server-side default, which is
|
`sampling` is `{}` was sampled at the reference node's server-side default, which is
|
||||||
`enable_thinking: false`.
|
`enable_thinking: false`.
|
||||||
- `traces_with_reasoning` — assistant messages carrying `reasoning_content` (or a literal
|
- `traces_with_reasoning` — assistant messages carrying `reasoning_content` (or a literal
|
||||||
`<think>` block). **This is the only in-trace evidence that the model was allowed to
|
`<think>` block). **This is the only in-trace evidence that the model was allowed to
|
||||||
@@ -246,12 +249,12 @@ from `run-20260821-1401`, with `reward_mean_from_eval_log` agreeing in every cas
|
|||||||
|
|
||||||
|
|
||||||
**The run: `outputs/run-20260821-1401/`.** Seven data environments, 32 rollouts each,
|
**The run: `outputs/run-20260821-1401/`.** Seven data environments, 32 rollouts each,
|
||||||
224 episodes, **223 scored**. Model `brain-qwen38-dspark` (Qwen3.8-27B) on spark-1.
|
224 episodes, **223 scored**. Model `brain-qwen38-dspark` (Qwen3.8-27B) on the reference node.
|
||||||
Wall-clock 14:03 → 14:58, 55 minutes, `-c 8`.
|
Wall-clock 14:03 → 14:58, 55 minutes, `-c 8`.
|
||||||
|
|
||||||
> ⚠️ **Sampling configuration for every number in this section — read it before quoting
|
> ⚠️ **Sampling configuration for every number in this section — read it before quoting
|
||||||
> one.** `verifiers==0.3.0`; `sampling = {}`, i.e. no `max_tokens` and no
|
> one.** `verifiers==0.3.0`; `sampling = {}`, i.e. no `max_tokens` and no
|
||||||
> `chat_template_kwargs`, so spark-1's server-side default applied and
|
> `chat_template_kwargs`, so the reference node's server-side default applied and
|
||||||
> **`enable_thinking` was `false`**; `temperature`, `top_p` and `reasoning_effort` all
|
> **`enable_thinking` was `false`**; `temperature`, `top_p` and `reasoning_effort` all
|
||||||
> unset; `-c 8`; harness `null`; agent runtime `subprocess`. `traces_with_reasoning` is
|
> unset; `-c 8`; harness `null`; agent runtime `subprocess`. `traces_with_reasoning` is
|
||||||
> **0 of 224**. These are thinking-**off** numbers. §6 re-measures the same eight
|
> **0 of 224**. These are thinking-**off** numbers. §6 re-measures the same eight
|
||||||
@@ -274,7 +277,7 @@ of this run got six of seven means wrong by taking the unweighted mean.
|
|||||||
| `bot-detection` | 32 | 32 | 0 | **0.3488** | 0.3488 |
|
| `bot-detection` | 32 | 32 | 0 | **0.3488** | 0.3488 |
|
||||||
| `grand-exchange` | 32 | 32 | 0 | **0.0055** | 0.0055 |
|
| `grand-exchange` | 32 | 32 | 0 | **0.0055** | 0.0055 |
|
||||||
|
|
||||||
The single errored episode is a genuine spark-1 gateway timeout (`ProviderError`, 504) in
|
The single errored episode is a genuine the reference node gateway timeout (`ProviderError`, 504) in
|
||||||
`drop-table-inference` — not a scoring bug. **Rank on `_attempted`** (errors as 0); the
|
`drop-table-inference` — not a scoring bug. **Rank on `_attempted`** (errors as 0); the
|
||||||
`_scored` column is Arena's own "score what survived" trap in reporting form.
|
`_scored` column is Arena's own "score what survived" trap in reporting form.
|
||||||
|
|
||||||
@@ -326,10 +329,10 @@ Appended 2026-08-21, after re-measuring `grand-exchange`.
|
|||||||
|
|
||||||
### The cause
|
### The cause
|
||||||
|
|
||||||
spark-1's SGLang is started with a default that overrides the chat template:
|
the reference node's SGLang is started with a default that overrides the chat template:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -s http://100.127.247.67:8001/get_server_info | jq '.default_chat_template_kwargs, .reasoning_parser'
|
curl -s "${SPARK_BASE_URL%/v1}/get_server_info" | jq '.default_chat_template_kwargs, .reasoning_parser'
|
||||||
# {"enable_thinking": false}
|
# {"enable_thinking": false}
|
||||||
# "qwen3"
|
# "qwen3"
|
||||||
```
|
```
|
||||||
@@ -410,7 +413,7 @@ buy limit — it simply does not converge inside 16k:
|
|||||||
|
|
||||||
A single rollout at `-c 1` (`outputs/thinking/smoke/`) does the same: 486 seconds,
|
A single rollout at `-c 1` (`outputs/thinking/smoke/`) does the same: 486 seconds,
|
||||||
`completion_tokens` 16384, `reasoning_content` 30,121 characters, `content` empty, reward
|
`completion_tokens` 16384, `reasoning_content` 30,121 characters, `content` empty, reward
|
||||||
0.000. So does the same prompt sent straight at spark-1 with `curl`, no verifiers in the
|
0.000. So does the same prompt sent straight at the reference node with `curl`, no verifiers in the
|
||||||
path — `reasoning_tokens` 16390, `finish_reason` length, content empty. **This is not a
|
path — `reasoning_tokens` 16390, `finish_reason` length, content empty. **This is not a
|
||||||
harness bug, and it is not fixed by raising `max_tokens`** — see the next section for why.
|
harness bug, and it is not fixed by raising `max_tokens`** — see the next section for why.
|
||||||
|
|
||||||
@@ -427,7 +430,7 @@ DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=600.0, write=600.0, pool=600.0
|
|||||||
```
|
```
|
||||||
|
|
||||||
`BaseClientConfig` exposes `base_url`, `api_key_var` and `headers` and nothing else. **There
|
`BaseClientConfig` exposes `base_url`, `api_key_var` and `headers` and nothing else. **There
|
||||||
is no config key and no CLI flag for this timeout.** At spark-1's ~34 tok/s single-stream it
|
is no config key and no CLI flag for this timeout.** At the reference node's ~34 tok/s single-stream it
|
||||||
caps one call at roughly 20,000 completion tokens, and far fewer under concurrency.
|
caps one call at roughly 20,000 completion tokens, and far fewer under concurrency.
|
||||||
|
|
||||||
That is not theoretical. `outputs/thinking/ge-think-n8.log` is 8 tasks at `-c 8` with
|
That is not theoretical. `outputs/thinking/ge-think-n8.log` is 8 tasks at `-c 8` with
|
||||||
@@ -463,7 +466,7 @@ false`. The only lever is `max_tokens`, and `max_tokens` cuts the answer, not th
|
|||||||
Three consequences for anyone re-running this suite against a reasoning model:
|
Three consequences for anyone re-running this suite against a reasoning model:
|
||||||
|
|
||||||
1. **Keep `-c` low and `max_tokens` under the wall**, or budget the run knowing calls will be
|
1. **Keep `-c` low and `max_tokens` under the wall**, or budget the run knowing calls will be
|
||||||
cut at ten minutes. spark-1 is shared — another lane's eval was running against it during
|
cut at ten minutes. the reference node is shared — another lane's eval was running against it during
|
||||||
this measurement, and contention alone moves a rollout across the line.
|
this measurement, and contention alone moves a rollout across the line.
|
||||||
2. **A timed-out run leaves no evidence.** Zero traces, exit code 0. `count_rewards.py`
|
2. **A timed-out run leaves no evidence.** Zero traces, exit code 0. `count_rewards.py`
|
||||||
reports on a file that does not exist. This is the same trap as §3 with a bigger blast
|
reports on a file that does not exist. This is the same trap as §3 with a bigger blast
|
||||||
@@ -523,7 +526,7 @@ recorded.
|
|||||||
- Put the sampling configuration in the run's own name or its notes. A number that changes
|
- Put the sampling configuration in the run's own name or its notes. A number that changes
|
||||||
by two orders of magnitude with one server-side default is not a capability score unless
|
by two orders of magnitude with one server-side default is not a capability score unless
|
||||||
the configuration travels with it.
|
the configuration travels with it.
|
||||||
- Ask spark-1 what it defaults to (`/get_server_info`) rather than assuming a model's own
|
- Ask the reference node what it defaults to (`/get_server_info`) rather than assuming a model's own
|
||||||
default applies.
|
default applies.
|
||||||
- If `reasoning_tokens` matters to you, read it off the provider — the trace drops it.
|
- If `reasoning_tokens` matters to you, read it off the provider — the trace drops it.
|
||||||
|
|
||||||
@@ -539,7 +542,7 @@ achievable today" — was owed a run.
|
|||||||
> **Sampling configuration for every number in this section.** `verifiers==0.3.1`
|
> **Sampling configuration for every number in this section.** `verifiers==0.3.1`
|
||||||
> (`read=None`); `chat_template_kwargs = {enable_thinking = true}`; `max_tokens = 49152`;
|
> (`read=None`); `chat_template_kwargs = {enable_thinking = true}`; `max_tokens = 49152`;
|
||||||
> `temperature`, `top_p`, `reasoning_effort` unset; `-c 8`; harness `null`; agent runtime
|
> `temperature`, `top_p`, `reasoning_effort` unset; `-c 8`; harness `null`; agent runtime
|
||||||
> `subprocess`; model `brain-qwen38-dspark` on spark-1. The configs are
|
> `subprocess`; model `brain-qwen38-dspark` on the reference node. The configs are
|
||||||
> `outputs/thinking-n32/*.toml`, the sweep is `outputs/thinking-n32/run_all.sh`, the runs
|
> `outputs/thinking-n32/*.toml`, the sweep is `outputs/thinking-n32/run_all.sh`, the runs
|
||||||
> are `outputs/thinking-n32/run/<env>/`.
|
> are `outputs/thinking-n32/run/<env>/`.
|
||||||
|
|
||||||
@@ -625,7 +628,7 @@ long one (0.6465).
|
|||||||
⚠️ **Whatever mean this run finishes with is a truncation artefact, exactly like
|
⚠️ **Whatever mean this run finishes with is a truncation artefact, exactly like
|
||||||
`grand-exchange`'s 0.1667 was.** Do not publish it as a capability score, and do not read
|
`grand-exchange`'s 0.1667 was.** Do not publish it as a capability score, and do not read
|
||||||
it as evidence about the mixture. The comparison `grand-exchange` earned — a distribution
|
it as evidence about the mixture. The comparison `grand-exchange` earned — a distribution
|
||||||
where 30 of 32 finish — this environment has not: it needs a larger `max_tokens` (spark-1's
|
where 30 of 32 finish — this environment has not: it needs a larger `max_tokens` (the reference node's
|
||||||
context is 262,144, so there is room) and a correspondingly larger wall-clock budget. One
|
context is 262,144, so there is room) and a correspondingly larger wall-clock budget. One
|
||||||
49,152-token rollout costs roughly 25 minutes at `-c 8` on this box.
|
49,152-token rollout costs roughly 25 minutes at `-c 8` on this box.
|
||||||
|
|
||||||
|
|||||||
@@ -188,7 +188,7 @@ It scored zero because **31 of 32 rollouts placed no orders at all**: 18 complet
|
|||||||
`{"expected_profit": 0, "orders": []}`, `finish_reason=stop`. `planned` had mean 65.6 and
|
`{"expected_profit": 0, "orders": []}`, `finish_reason=stop`. `planned` had mean 65.6 and
|
||||||
median 0 — one rollout in 32 planned anything at all; `profit_ratio` maxed at 0.323, so even
|
median 0 — one rollout in 32 planned anything at all; `profit_ratio` maxed at 0.323, so even
|
||||||
that one reached 29% of the reference. Nothing in this environment was measured. See
|
that one reached 29% of the reference. Nothing in this environment was measured. See
|
||||||
`FIRST_EVAL.md` §5 — spark-1 serves with
|
`FIRST_EVAL.md` §5 — the reference node serves with
|
||||||
`default_chat_template_kwargs = {"enable_thinking": false}`, and this environment is the one
|
`default_chat_template_kwargs = {"enable_thinking": false}`, and this environment is the one
|
||||||
that cannot be answered without arithmetic.
|
that cannot be answered without arithmetic.
|
||||||
|
|
||||||
@@ -258,7 +258,7 @@ uv run --with regex python probe.py # floors and ceilings, per c
|
|||||||
`tools/regate.py` rebuilds each task from the seed in its trace, re-runs the shipped scorer
|
`tools/regate.py` rebuilds each task from the seed in its trace, re-runs the shipped scorer
|
||||||
over the model's own reply, and asks a candidate predicate directly. That is a better
|
over the model's own reply, and asks a candidate predicate directly. That is a better
|
||||||
measurement than a fresh sample would have been: it holds the model's behaviour fixed and
|
measurement than a fresh sample would have been: it holds the model's behaviour fixed and
|
||||||
varies only the reward, and it costs nothing on spark-1. Its `recorded_mean` reproduces
|
varies only the reward, and it costs nothing on the reference node. Its `recorded_mean` reproduces
|
||||||
0.3488, 0.4432 and 0.4040 exactly, which is the cross-check that it is replaying the same
|
0.3488, 0.4432 and 0.4040 exactly, which is the cross-check that it is replaying the same
|
||||||
population `eval.log` scored.
|
population `eval.log` scored.
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -5,7 +5,7 @@ harness, trainer, runtime marketplace, or environment hub.
|
|||||||
|
|
||||||
- Every environment pins stable `verifiers==0.3.1`.
|
- Every environment pins stable `verifiers==0.3.1`.
|
||||||
- Pure-trace environments select Verifiers' upstream `null` harness.
|
- Pure-trace environments select Verifiers' upstream `null` harness.
|
||||||
- Local model-free and spark-1 runs name the `subprocess` runtime explicitly so
|
- Local model-free and the reference node runs name the `subprocess` runtime explicitly so
|
||||||
tests never provision a paid Prime sandbox.
|
tests never provision a paid Prime sandbox.
|
||||||
- Prime Hub is the public distribution target for approved environment wheels.
|
- Prime Hub is the public distribution target for approved environment wheels.
|
||||||
- Prime-RL consumes the same taskset packages for training; Forge owns the
|
- Prime-RL consumes the same taskset packages for training; Forge owns the
|
||||||
|
|||||||
+1
-1
@@ -4,7 +4,7 @@ A margin argued from a distribution is a guess. This replays `outputs/run-202608
|
|||||||
— 32 real rollouts per environment, `brain-qwen38-dspark`, thinking off — through the
|
— 32 real rollouts per environment, `brain-qwen38-dspark`, thinking off — through the
|
||||||
shipped scorer, rebuilds each task from its seed, and asks the candidate predicate
|
shipped scorer, rebuilds each task from its seed, and asks the candidate predicate
|
||||||
directly. The model's behaviour is held fixed and only the reward varies, which is a
|
directly. The model's behaviour is held fixed and only the reward varies, which is a
|
||||||
cleaner measurement than a fresh sample would be and does not touch spark-1.
|
cleaner measurement than a fresh sample would be and does not touch the reference node.
|
||||||
|
|
||||||
It reads traces and environment sources. It writes nothing.
|
It reads traces and environment sources. It writes nothing.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user