From ef01bde068ed684620610408f607e108d5f75c61 Mon Sep 17 00:00:00 2001 From: Karti Tripathi Date: Tue, 15 Sep 2026 12:15:37 -0700 Subject: [PATCH] docs: redact the reference node's address and hostname The eval docs named spark-1 by fleet hostname and published its inference endpoint (a tailnet address and port) plus the fact that it answers without an API key. This repo is PUBLIC, so that was a standing disclosure of an internal, unauthenticated service to anonymous readers. Endpoints now come from $SPARK_BASE_URL, which the operator exports, and the host is referred to as "the reference node" -- the convention the compute repo's docs already use. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_012AaUFYkUTsJn1fnJ89qbvW --- EXECUTION.md | 12 ++++++------ MULTI_TURN.md | 2 +- PLAN.md | 6 +++--- docs/FIRST_EVAL.md | 35 +++++++++++++++++++---------------- docs/GATE_DIAGNOSIS.md | 4 ++-- docs/PRIME_FIRST.md | 2 +- tools/regate.py | 2 +- 7 files changed, 33 insertions(+), 30 deletions(-) diff --git a/EXECUTION.md b/EXECUTION.md index f8329d6..44e7e42 100644 --- a/EXECUTION.md +++ b/EXECUTION.md @@ -79,7 +79,7 @@ Stands and is high-value: the endpoint inventory (29 routes, not 31), the 23 dea Corrected, blocking: -1. **"Ingest the runs already on amd-server" is impossible — there are none.** One `config.toml`, zero episodes, and that config points at `api.pinference.ai` with `PRIME_API_KEY`, which is empty. **Phase 1 must contain "execute one eval to completion against spark-1" as an explicit prerequisite**, or Arena's dashboard ships empty. +1. **"Ingest the runs already on amd-server" is impossible — there are none.** One `config.toml`, zero episodes, and that config points at `api.pinference.ai` with `PRIME_API_KEY`, which is empty. **Phase 1 must contain "execute one eval to completion against the reference node" as an explicit prerequisite**, or Arena's dashboard ships empty. 2. **The leaderboard is operator-gameable and mis-specified.** (a) The server must recompute every aggregate in SQL from the `arena_episodes` it just inserted, never trust the pusher's numbers. (b) Ranking on a mean over *scored* episodes is Arena's own "score what survived" trap. Store `reward_mean_scored` **and** `reward_mean_attempted` (errors as 0), rank on attempted, refuse to rank below a scored-fraction floor. 3. **`arena_episodes` flattens a list.** `Episode.traces` is a list; `reward`/`rewards`/`stop_condition`/`last_reply` are per-**Trace**. Primary key must be `(run_id, episode_id, trace_id)` with an `agent_name` column — otherwise the schema is wrong for `grand-exchange-live` (simulator as second role) and for `redaction-pressure` (genuine two-agent). 4. **The trace viewer publishes the held-out slice.** `GET /api/arena/runs/:id/episodes/:eid` at `user` scope returns full transcripts, which contain held-out task data and, for oracle runs, the answers. One leaked invite publishes the held-out corpus. **Transcripts are admin-only or redacted.** This settles the spec's open question on read scope. @@ -115,7 +115,7 @@ Corrected: ### C-compute — rent to train, Spark to serve · feasible; **its headline correction is false** -Stands and is the most immediately actionable: the policy itself; the measured spark-1 tenancy (121 GB total / 84 GB used / 36 GB free, SGLang + Chatterbox + vox + ASR co-resident, Chatterbox's ~8.4 GB/hr leak having killed brain before); `prime login` as step zero because `api_key` is empty (verified); **Arena → prime-rl needs no adapter** — `EnvConfig` is `SerializeAsAny[vf.EnvConfig]` and the taskset tree is the same shape, prefixed; the seat-name-is-a-TOML-key trap; Hosted Training being unavailable while E is parked (`rl.py:1522-1524` — bare v1 ids do not resolve through the Hub); the sm_121 lock contradiction in Forge; `TRAINERS = ("axolotl","trl")`; **no price is hardcoded anywhere and none should be invented.** +Stands and is the most immediately actionable: the policy itself; the measured the reference node tenancy (121 GB total / 84 GB used / 36 GB free, SGLang + Chatterbox + vox + ASR co-resident, Chatterbox's ~8.4 GB/hr leak having killed brain before); `prime login` as step zero because `api_key` is empty (verified); **Arena → prime-rl needs no adapter** — `EnvConfig` is `SerializeAsAny[vf.EnvConfig]` and the taskset tree is the same shape, prefixed; the seat-name-is-a-TOML-key trap; Hosted Training being unavailable while E is parked (`rl.py:1522-1524` — bare v1 ids do not resolve through the Hub); the sm_121 lock contradiction in Forge; `TRAINERS = ("axolotl","trl")`; **no price is hardcoded anywhere and none should be invented.** Corrected, blocking: @@ -141,7 +141,7 @@ Corrected, blocking: | **1** | **B1** — discovery de-hardcode + timeout + step name + `uv.lock` cleanup | `arena` | nothing | | **2** | **A step 2** — `live.py` engine + unit tests + **re-measure the ladder** | `arena` | nothing (own worktree) | | **3** | **D step 1** — worker, vendored TS, hashes, `test_replay.py` | `arena` (new dir only) | nothing | -| **4** | **Compute Policy step 0** — `prime login`, catalogue + prices, doc the corrected spark-1 facts | none / `arena/docs` | nothing | +| **4** | **Compute Policy step 0** — `prime login`, catalogue + prices, doc the corrected the reference node facts | none / `arena/docs` | nothing | | **5** | **P1** — Caddy host, host-aware routing, AppShell, Overview/Worlds/Org/Inbox | `lumbridge-v4` | nothing | | **6** | **W step 0** — the five factual corrections + `eval-results.json` decision | `lumbridge-v4` | nothing | @@ -209,7 +209,7 @@ Two edges that were wrong in the source specs and are now correct: 1. **B1** (arena): tomllib discovery, `@probes` registry, timeout 180, `uv.lock` untracking, step rename. *Gate: `probe.py` still prints seven `ok` rows; CI green with zero list edits.* 2. **A steps 1-2 + re-measurement** (arena, worktree): frozen generator copy + equality assertion; `LiveBook`/`Leg`/`Ledger` with unit tests; **then re-run the full policy ladder and publish the corrected numbers.** *Gate: the ladder reconciles with the reward formula; `horizon`, `FREEZE_RANGE`, `TARGET_SHARE`, `TURNS` are chosen from that measurement, not from the spec.* 3. **D step 1** (arena, new dir): worker + vendored TS + `hashes.py` + `sync_tera.py` + `test_replay.py`. *Gate: 32/32 replay exact, tampering rejected, hash check green — and a decision on the 4 unpinned files.* -4. **Compute Policy step 0**: `prime login`; record real `availability` output and prices; write `arena/docs/COMPUTE_POLICY.md` with the corrected spark-1 facts (`brain-qwen38-dspark`, silent-wrong-model hazard, 36 GB headroom). +4. **Compute Policy step 0**: `prime login`; record real `availability` output and prices; write `arena/docs/COMPUTE_POLICY.md` with the corrected the reference node facts (`brain-qwen38-dspark`, silent-wrong-model hazard, 36 GB headroom). 5. **P1** (lumbridge-v4): Caddy `app.` block with `bind 10.0.0.2` and the non-empty-body check; host-aware `App.tsx`; `renderAppShell` + `sendShell` signature change + `seo.test.ts` + `spa-fallback.test.ts`; AppShell; Overview / Worlds / Org / Inbox; delete the 23 dead `api.ts` methods. 6. **W step 0** (lumbridge-v4): the five factual corrections — `arena.tsx:330` "Nine tasks", `landing.tsx` card 02, the 4-row probe table → 7 rows, `office-jobs-v1` deleted, `arena.tsx:519` "no model scores published". Ship the Nemotron run into `eval-results.json`. @@ -218,7 +218,7 @@ Two edges that were wrong in the source specs and are now correct: ### Phase 2 7. **A steps 3-9** → `grand-exchange-live` ships with its probe row (needs B1 merged). -8. **One real eval run** against spark-1, pinned to `brain-qwen38-dspark`, with the empty-rewards count checked. *This is a hard prerequisite for P2 and for any site number.* +8. **One real eval run** against the reference node, pinned to `brain-qwen38-dspark`, with the empty-rewards count checked. *This is a hard prerequisite for P2 and for any site number.* 9. **D crow-nav** end to end (role named `agent`, `vf.Taskset[...]` generic form, `oracle` op in the worker, `gate` as a multiplier). 10. **W /arena rebuild** + `/bench` on the new primitives. 11. **First rented run**: pod create with an attached disk, `install.sh`, wheel install, baseline eval, `uv run rl`, re-eval, before/after table. @@ -263,7 +263,7 @@ Two edges that were wrong in the source specs and are now correct: | **5** | **Silent wrong-model comparison.** `--model brain` returns 200 from a box serving Qwen3.8-27B. Re-running the Nemotron baseline today produces different-model numbers under an identical command, and they will be tabulated as comparable. | Pinning `brain-qwen38-dspark` (and the scene) in every command in `README.md`, `PLAN.md`, `MULTI_TURN.md` and `COMPUTE_POLICY.md`. Trivial; do it in Phase 1. | | **6** | **`probe.py`, `tests/test_probe.py` and `ci.yml` are edited by five of six workstreams, in a repo that already took divergent concurrent pushes.** | B1, first, before anything else touches `arena`. | | **7** | **The reward denominator is editable with every `tera` gate green.** ⚠️ CORRECTED 2026-08-21: not `offices/sites.ts` (mutating it produces a bit-identical episode — refuted twice). The hole is **`src/arena/index.ts`**, which re-exports the scripted baselines that ARE the denominator. Demonstrated: a halved office-nav baseline changes return 4.744 → 4.098 while `arena:source-hashes` prints ok and exits 0. | Pinning `src/arena/index.ts` in `tera/scripts/check-arena-source-hashes.mjs` and regenerating `sourceHashes.ts`. One line, in a repo this session did not own. | -| **8** | **spark-1 has 36 GB of 121 GB free with four resident model services, one of which leaks ~8.4 GB/hour and has killed the brain before.** "Production inference on the Spark" is a tenancy claim held together by the governor on :8011. | A mandatory headroom check before any new resident model, and Compute's watchdog verified live. | +| **8** | **the reference node has 36 GB of 121 GB free with four resident model services, one of which leaks ~8.4 GB/hour and has killed the brain before.** "Production inference on the Spark" is a tenancy claim held together by the governor on :8011. | A mandatory headroom check before any new resident model, and Compute's watchdog verified live. | | **9** | **`PRIME_API_KEY` is empty**, blocking not just the parked Hub push but every rental command, every price, and the entire training half of the compute policy. | `prime login`. One command. | | **10** | **Two of D's four environments cannot discriminate under the 0.9 ceiling band** — drive-101 gives ~0.95+ to any constant throttle, office-nav collapses to `efficiency` alone. | Re-deriving the band per environment against the measured return distribution, or shipping fewer environments. | | **11** | **The held-out corpus leaks through P2's trace viewer at `user` scope.** One invite code. | Decision 5 — admin-only. | diff --git a/MULTI_TURN.md b/MULTI_TURN.md index 888ab09..430bd1e 100644 --- a/MULTI_TURN.md +++ b/MULTI_TURN.md @@ -163,7 +163,7 @@ it is configured here. - `uv run eval @ configs/canary_trap.toml --dry-run` → exit 0, config written ✓ - stable verifiers 0.3.1 with upstream harnesses and runtimes present ✓ - Docker 29.1.3 running (tier 2/3 runtime) ✓ -- spark-1 `brain` on `http://100.127.247.67:8001/v1`, model id `brain` ✓ +- the reference node `brain` on `$SPARK_BASE_URL`, model id `brain` ✓ - `prime` CLI installed (0.6.25; 0.6.26 available) ✓ - `PRIME_API_KEY` ✗ — see above diff --git a/PLAN.md b/PLAN.md index 9d109c0..b828247 100644 --- a/PLAN.md +++ b/PLAN.md @@ -54,7 +54,7 @@ identity across all six projects; no per-project byline. the Prime Intellect Hub yet. Workstream E is parked — not cancelled, not blocked on anything technical. `PRIME_API_KEY` is therefore not urgent. -**✅ 7 — Production brain: Qwen3.8-27B.** Settled 2026-08-21. spark-1 serves +**✅ 7 — Production brain: Qwen3.8-27B.** Settled 2026-08-21. the reference node serves `brain-qwen38-dspark`; that is the default. The Nemotron baseline is ARCHIVED, not deleted, and Arena's seven scores get re-run and republished against Qwen. ⚠️ **`--model brain` does NOT 404 — it silently succeeds and records `brain` in the @@ -118,7 +118,7 @@ trap) already exists, and it is the smallest change that yields a genuinely new measurement. Deliverable: a `vf.Env` subclass exported from the wheel's `__init__.py`, a config, -and a measured run against spark-1's `brain`. Pattern to copy: +and a measured run against the reference node's `brain`. Pattern to copy: `verifiers/v1/tasksets/textarena/taskset.py`. This workstream is where **rule 4 — the budget must bind** gets written for real. @@ -218,7 +218,7 @@ files. Everything downstream of B fans out four ways. - `uv run eval @ configs/.toml --dry-run` → exit 0 ✓ - stable verifiers 0.3.1, using its upstream `null` harness and runtimes ✓ - Docker 29.1.3 (tier 2/3 only; tier 1 needs none) ✓ -- spark-1 `brain` at `http://100.127.247.67:8001/v1`, model id `brain` ✓ +- the reference node `brain` at `$SPARK_BASE_URL`, model id `brain` ✓ - `prime` CLI 0.6.25 ✓ - `PRIME_API_KEY` ✗ — blocks E, which is parked anyway diff --git a/docs/FIRST_EVAL.md b/docs/FIRST_EVAL.md index a5e421b..3c8dffe 100644 --- a/docs/FIRST_EVAL.md +++ b/docs/FIRST_EVAL.md @@ -21,10 +21,13 @@ possible. **Every reward number in this file now carries its sampling configurat ```bash cd ~/repos/gitea/arena export SPARK_API_KEY=dummy +# The OpenAI-compatible endpoint of the reference node, e.g. http://:8001/v1. +# Kept out of this repo on purpose: it is an internal address, not a public one. +export SPARK_BASE_URL=... uv run --project environments/canary_trap eval @ configs/canary_trap.toml \ --model brain-qwen38-dspark \ - --client.base-url http://100.127.247.67:8001/v1 \ + --client.base-url "$SPARK_BASE_URL" \ --client.api-key-var SPARK_API_KEY \ --env.agent.runtime.type subprocess \ --no-push --no-rich \ @@ -46,10 +49,10 @@ see §5. The seven data environments have thinking-on copies in `outputs/thinkin |---|---| | `--model brain-qwen38-dspark` | **The served id, pinned.** `--model brain` does *not* 404. SGLang does not validate the model field on the chat route: it returns 200, Qwen answers, and the trace records `brain`. The old `outputs/canary-trap--brain--null/` dir is exactly that mistake. Pin the id `GET /v1/models` reports. | | `--client.base-url` | The default is `https://api.pinference.ai/api/v1`, which needs a `PRIME_API_KEY` this box does not have. | -| `--client.api-key-var SPARK_API_KEY` | spark-1 requires **no** key — a bare `POST /v1/chat/completions` returns 200. But the client still resolves *some* env var, so point it at a dummy one rather than leaving it on `PRIME_API_KEY` (empty → the failure mode of the one earlier attempt). | +| `--client.api-key-var SPARK_API_KEY` | the reference node does not authenticate with a Prime key. The client still resolves *some* env var, so point it at a dummy one rather than leaving it on `PRIME_API_KEY` (empty → the failure mode of the one earlier attempt). | | `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. | | `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. | -| `-c 8` | spark-1 is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. | +| `-c 8` | the reference node is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. | | `--env.agent.runtime.type subprocess` | ⚠️ **New, and not optional.** 0.3.0 defaulted the agent runtime to `subprocess`; `0.3.1` defaults it to `prime` — a *remote* sandbox. A remote role makes `Env._runs_local()` false, which makes the interception pool mint a Prime tunnel, which calls `ensure_prime_auth()` and raises `SystemExit: not authenticated with prime`. The run dies before a single rollout, having already created its output directory and an empty `traces.jsonl`. The null harness needs no runtime at all; naming a local one is the whole fix. | | `-o ` + `--run.dir ` | `-o` is `output_dir` and the run writes to `output_dir / run.dir`. On 0.3.0 a bare `-o` gave you a flat directory; on 0.3.1 it gives you `outputs//------/`. Pass `--run.dir` to name the leaf yourself. | @@ -58,7 +61,7 @@ see §5. The seven data environments have thinking-on copies in `outputs/thinkin - `README.md`'s "Running one" line omits `--client.base-url` / `--client.api-key-var` entirely, so copy-pasting it points at Prime's inference API with an empty key. That is the one attempt that was ever made, and why `outputs/` was empty. -- `MULTI_TURN.md`'s preconditions say the spark-1 model id is `brain`. The served +- `MULTI_TURN.md`'s preconditions say the the reference node model id is `brain`. The served id is `brain-qwen38-dspark`; `brain` silently mislabels the trace. - `EXECUTION.md` is right that `--harness.id null` is not valid — `EvalConfig` has no top-level `harness`. The configs already carry `[env.agent.harness] id = "null"`, @@ -224,7 +227,7 @@ It reads the run's own resolved config (`configs/eval.json` on dev59, `config.to 0.3.0) and prints, beside the means: - `sampling` — the run's own record of `max_tokens` / `chat_template_kwargs`. A run whose - `sampling` is `{}` was sampled at spark-1's server-side default, which is + `sampling` is `{}` was sampled at the reference node's server-side default, which is `enable_thinking: false`. - `traces_with_reasoning` — assistant messages carrying `reasoning_content` (or a literal `` block). **This is the only in-trace evidence that the model was allowed to @@ -246,12 +249,12 @@ from `run-20260821-1401`, with `reward_mean_from_eval_log` agreeing in every cas **The run: `outputs/run-20260821-1401/`.** Seven data environments, 32 rollouts each, -224 episodes, **223 scored**. Model `brain-qwen38-dspark` (Qwen3.8-27B) on spark-1. +224 episodes, **223 scored**. Model `brain-qwen38-dspark` (Qwen3.8-27B) on the reference node. Wall-clock 14:03 → 14:58, 55 minutes, `-c 8`. > ⚠️ **Sampling configuration for every number in this section — read it before quoting > one.** `verifiers==0.3.0`; `sampling = {}`, i.e. no `max_tokens` and no -> `chat_template_kwargs`, so spark-1's server-side default applied and +> `chat_template_kwargs`, so the reference node's server-side default applied and > **`enable_thinking` was `false`**; `temperature`, `top_p` and `reasoning_effort` all > unset; `-c 8`; harness `null`; agent runtime `subprocess`. `traces_with_reasoning` is > **0 of 224**. These are thinking-**off** numbers. §6 re-measures the same eight @@ -274,7 +277,7 @@ of this run got six of seven means wrong by taking the unweighted mean. | `bot-detection` | 32 | 32 | 0 | **0.3488** | 0.3488 | | `grand-exchange` | 32 | 32 | 0 | **0.0055** | 0.0055 | -The single errored episode is a genuine spark-1 gateway timeout (`ProviderError`, 504) in +The single errored episode is a genuine the reference node gateway timeout (`ProviderError`, 504) in `drop-table-inference` — not a scoring bug. **Rank on `_attempted`** (errors as 0); the `_scored` column is Arena's own "score what survived" trap in reporting form. @@ -326,10 +329,10 @@ Appended 2026-08-21, after re-measuring `grand-exchange`. ### The cause -spark-1's SGLang is started with a default that overrides the chat template: +the reference node's SGLang is started with a default that overrides the chat template: ```bash -curl -s http://100.127.247.67:8001/get_server_info | jq '.default_chat_template_kwargs, .reasoning_parser' +curl -s "${SPARK_BASE_URL%/v1}/get_server_info" | jq '.default_chat_template_kwargs, .reasoning_parser' # {"enable_thinking": false} # "qwen3" ``` @@ -410,7 +413,7 @@ buy limit — it simply does not converge inside 16k: A single rollout at `-c 1` (`outputs/thinking/smoke/`) does the same: 486 seconds, `completion_tokens` 16384, `reasoning_content` 30,121 characters, `content` empty, reward -0.000. So does the same prompt sent straight at spark-1 with `curl`, no verifiers in the +0.000. So does the same prompt sent straight at the reference node with `curl`, no verifiers in the path — `reasoning_tokens` 16390, `finish_reason` length, content empty. **This is not a harness bug, and it is not fixed by raising `max_tokens`** — see the next section for why. @@ -427,7 +430,7 @@ DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=600.0, write=600.0, pool=600.0 ``` `BaseClientConfig` exposes `base_url`, `api_key_var` and `headers` and nothing else. **There -is no config key and no CLI flag for this timeout.** At spark-1's ~34 tok/s single-stream it +is no config key and no CLI flag for this timeout.** At the reference node's ~34 tok/s single-stream it caps one call at roughly 20,000 completion tokens, and far fewer under concurrency. That is not theoretical. `outputs/thinking/ge-think-n8.log` is 8 tasks at `-c 8` with @@ -463,7 +466,7 @@ false`. The only lever is `max_tokens`, and `max_tokens` cuts the answer, not th Three consequences for anyone re-running this suite against a reasoning model: 1. **Keep `-c` low and `max_tokens` under the wall**, or budget the run knowing calls will be - cut at ten minutes. spark-1 is shared — another lane's eval was running against it during + cut at ten minutes. the reference node is shared — another lane's eval was running against it during this measurement, and contention alone moves a rollout across the line. 2. **A timed-out run leaves no evidence.** Zero traces, exit code 0. `count_rewards.py` reports on a file that does not exist. This is the same trap as §3 with a bigger blast @@ -523,7 +526,7 @@ recorded. - Put the sampling configuration in the run's own name or its notes. A number that changes by two orders of magnitude with one server-side default is not a capability score unless the configuration travels with it. -- Ask spark-1 what it defaults to (`/get_server_info`) rather than assuming a model's own +- Ask the reference node what it defaults to (`/get_server_info`) rather than assuming a model's own default applies. - If `reasoning_tokens` matters to you, read it off the provider — the trace drops it. @@ -539,7 +542,7 @@ achievable today" — was owed a run. > **Sampling configuration for every number in this section.** `verifiers==0.3.1` > (`read=None`); `chat_template_kwargs = {enable_thinking = true}`; `max_tokens = 49152`; > `temperature`, `top_p`, `reasoning_effort` unset; `-c 8`; harness `null`; agent runtime -> `subprocess`; model `brain-qwen38-dspark` on spark-1. The configs are +> `subprocess`; model `brain-qwen38-dspark` on the reference node. The configs are > `outputs/thinking-n32/*.toml`, the sweep is `outputs/thinking-n32/run_all.sh`, the runs > are `outputs/thinking-n32/run//`. @@ -625,7 +628,7 @@ long one (0.6465). ⚠️ **Whatever mean this run finishes with is a truncation artefact, exactly like `grand-exchange`'s 0.1667 was.** Do not publish it as a capability score, and do not read it as evidence about the mixture. The comparison `grand-exchange` earned — a distribution -where 30 of 32 finish — this environment has not: it needs a larger `max_tokens` (spark-1's +where 30 of 32 finish — this environment has not: it needs a larger `max_tokens` (the reference node's context is 262,144, so there is room) and a correspondingly larger wall-clock budget. One 49,152-token rollout costs roughly 25 minutes at `-c 8` on this box. diff --git a/docs/GATE_DIAGNOSIS.md b/docs/GATE_DIAGNOSIS.md index 1beef72..597dc91 100644 --- a/docs/GATE_DIAGNOSIS.md +++ b/docs/GATE_DIAGNOSIS.md @@ -188,7 +188,7 @@ It scored zero because **31 of 32 rollouts placed no orders at all**: 18 complet `{"expected_profit": 0, "orders": []}`, `finish_reason=stop`. `planned` had mean 65.6 and median 0 — one rollout in 32 planned anything at all; `profit_ratio` maxed at 0.323, so even that one reached 29% of the reference. Nothing in this environment was measured. See -`FIRST_EVAL.md` §5 — spark-1 serves with +`FIRST_EVAL.md` §5 — the reference node serves with `default_chat_template_kwargs = {"enable_thinking": false}`, and this environment is the one that cannot be answered without arithmetic. @@ -258,7 +258,7 @@ uv run --with regex python probe.py # floors and ceilings, per c `tools/regate.py` rebuilds each task from the seed in its trace, re-runs the shipped scorer over the model's own reply, and asks a candidate predicate directly. That is a better measurement than a fresh sample would have been: it holds the model's behaviour fixed and -varies only the reward, and it costs nothing on spark-1. Its `recorded_mean` reproduces +varies only the reward, and it costs nothing on the reference node. Its `recorded_mean` reproduces 0.3488, 0.4432 and 0.4040 exactly, which is the cross-check that it is replaying the same population `eval.log` scored. diff --git a/docs/PRIME_FIRST.md b/docs/PRIME_FIRST.md index c20762d..80e08b6 100644 --- a/docs/PRIME_FIRST.md +++ b/docs/PRIME_FIRST.md @@ -5,7 +5,7 @@ harness, trainer, runtime marketplace, or environment hub. - Every environment pins stable `verifiers==0.3.1`. - Pure-trace environments select Verifiers' upstream `null` harness. -- Local model-free and spark-1 runs name the `subprocess` runtime explicitly so +- Local model-free and the reference node runs name the `subprocess` runtime explicitly so tests never provision a paid Prime sandbox. - Prime Hub is the public distribution target for approved environment wheels. - Prime-RL consumes the same taskset packages for training; Forge owns the diff --git a/tools/regate.py b/tools/regate.py index af2d33a..16d9e4a 100644 --- a/tools/regate.py +++ b/tools/regate.py @@ -4,7 +4,7 @@ A margin argued from a distribution is a guess. This replays `outputs/run-202608 — 32 real rollouts per environment, `brain-qwen38-dspark`, thinking off — through the shipped scorer, rebuilds each task from its seed, and asks the candidate predicate directly. The model's behaviour is held fixed and only the reward varies, which is a -cleaner measurement than a fresh sample would be and does not touch spark-1. +cleaner measurement than a fresh sample would be and does not touch the reference node. It reads traces and environment sources. It writes nothing.