diff --git a/EXECUTION.md b/EXECUTION.md index 44e7e42..2c27fc1 100644 --- a/EXECUTION.md +++ b/EXECUTION.md @@ -263,7 +263,7 @@ Two edges that were wrong in the source specs and are now correct: | **5** | **Silent wrong-model comparison.** `--model brain` returns 200 from a box serving Qwen3.8-27B. Re-running the Nemotron baseline today produces different-model numbers under an identical command, and they will be tabulated as comparable. | Pinning `brain-qwen38-dspark` (and the scene) in every command in `README.md`, `PLAN.md`, `MULTI_TURN.md` and `COMPUTE_POLICY.md`. Trivial; do it in Phase 1. | | **6** | **`probe.py`, `tests/test_probe.py` and `ci.yml` are edited by five of six workstreams, in a repo that already took divergent concurrent pushes.** | B1, first, before anything else touches `arena`. | | **7** | **The reward denominator is editable with every `tera` gate green.** ⚠️ CORRECTED 2026-08-21: not `offices/sites.ts` (mutating it produces a bit-identical episode — refuted twice). The hole is **`src/arena/index.ts`**, which re-exports the scripted baselines that ARE the denominator. Demonstrated: a halved office-nav baseline changes return 4.744 → 4.098 while `arena:source-hashes` prints ok and exits 0. | Pinning `src/arena/index.ts` in `tera/scripts/check-arena-source-hashes.mjs` and regenerating `sourceHashes.ts`. One line, in a repo this session did not own. | -| **8** | **the reference node has 36 GB of 121 GB free with four resident model services, one of which leaks ~8.4 GB/hour and has killed the brain before.** "Production inference on the Spark" is a tenancy claim held together by the governor on :8011. | A mandatory headroom check before any new resident model, and Compute's watchdog verified live. | +| **8** | **The reference node has 36 GB of 121 GB free with four resident model services, one of which leaks ~8.4 GB/hour and has killed the brain before.** "Production inference on the Spark" is a tenancy claim held together by the governor on :8011. | A mandatory headroom check before any new resident model, and Compute's watchdog verified live. | | **9** | **`PRIME_API_KEY` is empty**, blocking not just the parked Hub push but every rental command, every price, and the entire training half of the compute policy. | `prime login`. One command. | | **10** | **Two of D's four environments cannot discriminate under the 0.9 ceiling band** — drive-101 gives ~0.95+ to any constant throttle, office-nav collapses to `efficiency` alone. | Re-deriving the band per environment against the measured return distribution, or shipping fewer environments. | | **11** | **The held-out corpus leaks through P2's trace viewer at `user` scope.** One invite code. | Decision 5 — admin-only. | diff --git a/PLAN.md b/PLAN.md index b828247..fbfbcb3 100644 --- a/PLAN.md +++ b/PLAN.md @@ -54,7 +54,7 @@ identity across all six projects; no per-project byline. the Prime Intellect Hub yet. Workstream E is parked — not cancelled, not blocked on anything technical. `PRIME_API_KEY` is therefore not urgent. -**✅ 7 — Production brain: Qwen3.8-27B.** Settled 2026-08-21. the reference node serves +**✅ 7 — Production brain: Qwen3.8-27B.** Settled 2026-08-21. The reference node serves `brain-qwen38-dspark`; that is the default. The Nemotron baseline is ARCHIVED, not deleted, and Arena's seven scores get re-run and republished against Qwen. ⚠️ **`--model brain` does NOT 404 — it silently succeeds and records `brain` in the diff --git a/docs/FIRST_EVAL.md b/docs/FIRST_EVAL.md index 3c8dffe..4425cb4 100644 --- a/docs/FIRST_EVAL.md +++ b/docs/FIRST_EVAL.md @@ -49,10 +49,10 @@ see §5. The seven data environments have thinking-on copies in `outputs/thinkin |---|---| | `--model brain-qwen38-dspark` | **The served id, pinned.** `--model brain` does *not* 404. SGLang does not validate the model field on the chat route: it returns 200, Qwen answers, and the trace records `brain`. The old `outputs/canary-trap--brain--null/` dir is exactly that mistake. Pin the id `GET /v1/models` reports. | | `--client.base-url` | The default is `https://api.pinference.ai/api/v1`, which needs a `PRIME_API_KEY` this box does not have. | -| `--client.api-key-var SPARK_API_KEY` | the reference node does not authenticate with a Prime key. The client still resolves *some* env var, so point it at a dummy one rather than leaving it on `PRIME_API_KEY` (empty → the failure mode of the one earlier attempt). | +| `--client.api-key-var SPARK_API_KEY` | The reference node does not authenticate with a Prime key. The client still resolves *some* env var, so point it at a dummy one rather than leaving it on `PRIME_API_KEY` (empty → the failure mode of the one earlier attempt). | | `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. | | `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. | -| `-c 8` | the reference node is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. | +| `-c 8` | The reference node is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. | | `--env.agent.runtime.type subprocess` | ⚠️ **New, and not optional.** 0.3.0 defaulted the agent runtime to `subprocess`; `0.3.1` defaults it to `prime` — a *remote* sandbox. A remote role makes `Env._runs_local()` false, which makes the interception pool mint a Prime tunnel, which calls `ensure_prime_auth()` and raises `SystemExit: not authenticated with prime`. The run dies before a single rollout, having already created its output directory and an empty `traces.jsonl`. The null harness needs no runtime at all; naming a local one is the whole fix. | | `-o ` + `--run.dir ` | `-o` is `output_dir` and the run writes to `output_dir / run.dir`. On 0.3.0 a bare `-o` gave you a flat directory; on 0.3.1 it gives you `outputs//------/`. Pass `--run.dir` to name the leaf yourself. | @@ -329,7 +329,7 @@ Appended 2026-08-21, after re-measuring `grand-exchange`. ### The cause -the reference node's SGLang is started with a default that overrides the chat template: +The reference node's SGLang is started with a default that overrides the chat template: ```bash curl -s "${SPARK_BASE_URL%/v1}/get_server_info" | jq '.default_chat_template_kwargs, .reasoning_parser' @@ -466,7 +466,7 @@ false`. The only lever is `max_tokens`, and `max_tokens` cuts the answer, not th Three consequences for anyone re-running this suite against a reasoning model: 1. **Keep `-c` low and `max_tokens` under the wall**, or budget the run knowing calls will be - cut at ten minutes. the reference node is shared — another lane's eval was running against it during + cut at ten minutes. The reference node is shared — another lane's eval was running against it during this measurement, and contention alone moves a rollout across the line. 2. **A timed-out run leaves no evidence.** Zero traces, exit code 0. `count_rewards.py` reports on a file that does not exist. This is the same trap as §3 with a bigger blast