docs: redact the reference node's address and hostname
The eval docs named spark-1 by fleet hostname and published its inference endpoint (a tailnet address and port) plus the fact that it answers without an API key. This repo is PUBLIC, so that was a standing disclosure of an internal, unauthenticated service to anonymous readers. Endpoints now come from $SPARK_BASE_URL, which the operator exports, and the host is referred to as "the reference node" -- the convention the compute repo's docs already use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012AaUFYkUTsJn1fnJ89qbvW
This commit is contained in:
+19
-16
@@ -21,10 +21,13 @@ possible. **Every reward number in this file now carries its sampling configurat
|
||||
```bash
|
||||
cd ~/repos/gitea/arena
|
||||
export SPARK_API_KEY=dummy
|
||||
# The OpenAI-compatible endpoint of the reference node, e.g. http://<host>:8001/v1.
|
||||
# Kept out of this repo on purpose: it is an internal address, not a public one.
|
||||
export SPARK_BASE_URL=...
|
||||
|
||||
uv run --project environments/canary_trap eval @ configs/canary_trap.toml \
|
||||
--model brain-qwen38-dspark \
|
||||
--client.base-url http://100.127.247.67:8001/v1 \
|
||||
--client.base-url "$SPARK_BASE_URL" \
|
||||
--client.api-key-var SPARK_API_KEY \
|
||||
--env.agent.runtime.type subprocess \
|
||||
--no-push --no-rich \
|
||||
@@ -46,10 +49,10 @@ see §5. The seven data environments have thinking-on copies in `outputs/thinkin
|
||||
|---|---|
|
||||
| `--model brain-qwen38-dspark` | **The served id, pinned.** `--model brain` does *not* 404. SGLang does not validate the model field on the chat route: it returns 200, Qwen answers, and the trace records `brain`. The old `outputs/canary-trap--brain--null/` dir is exactly that mistake. Pin the id `GET /v1/models` reports. |
|
||||
| `--client.base-url` | The default is `https://api.pinference.ai/api/v1`, which needs a `PRIME_API_KEY` this box does not have. |
|
||||
| `--client.api-key-var SPARK_API_KEY` | spark-1 requires **no** key — a bare `POST /v1/chat/completions` returns 200. But the client still resolves *some* env var, so point it at a dummy one rather than leaving it on `PRIME_API_KEY` (empty → the failure mode of the one earlier attempt). |
|
||||
| `--client.api-key-var SPARK_API_KEY` | the reference node does not authenticate with a Prime key. The client still resolves *some* env var, so point it at a dummy one rather than leaving it on `PRIME_API_KEY` (empty → the failure mode of the one earlier attempt). |
|
||||
| `--no-push` | Hub push is parked (PLAN.md). Push is **on by default** and needs `$PRIME_API_KEY`. |
|
||||
| `--no-rich` | The live dashboard is in-process only and unreadable in a log. Without it you get one `rollout done: … reward=…` line per rollout. |
|
||||
| `-c 8` | spark-1 is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. |
|
||||
| `-c 8` | the reference node is shared with Chatterbox, vox and ASR. The config's own `max_concurrent = 128` will hammer it. |
|
||||
| `--env.agent.runtime.type subprocess` | ⚠️ **New, and not optional.** 0.3.0 defaulted the agent runtime to `subprocess`; `0.3.1` defaults it to `prime` — a *remote* sandbox. A remote role makes `Env._runs_local()` false, which makes the interception pool mint a Prime tunnel, which calls `ensure_prime_auth()` and raises `SystemExit: not authenticated with prime`. The run dies before a single rollout, having already created its output directory and an empty `traces.jsonl`. The null harness needs no runtime at all; naming a local one is the whole fix. |
|
||||
| `-o <dir>` + `--run.dir <name>` | `-o` is `output_dir` and the run writes to `output_dir / run.dir`. On 0.3.0 a bare `-o` gave you a flat directory; on 0.3.1 it gives you `outputs/<your-dir>/<env>--<model>--<harness>--<hex8>/`. Pass `--run.dir` to name the leaf yourself. |
|
||||
|
||||
@@ -58,7 +61,7 @@ see §5. The seven data environments have thinking-on copies in `outputs/thinkin
|
||||
- `README.md`'s "Running one" line omits `--client.base-url` / `--client.api-key-var`
|
||||
entirely, so copy-pasting it points at Prime's inference API with an empty key.
|
||||
That is the one attempt that was ever made, and why `outputs/` was empty.
|
||||
- `MULTI_TURN.md`'s preconditions say the spark-1 model id is `brain`. The served
|
||||
- `MULTI_TURN.md`'s preconditions say the the reference node model id is `brain`. The served
|
||||
id is `brain-qwen38-dspark`; `brain` silently mislabels the trace.
|
||||
- `EXECUTION.md` is right that `--harness.id null` is not valid — `EvalConfig` has
|
||||
no top-level `harness`. The configs already carry `[env.agent.harness] id = "null"`,
|
||||
@@ -224,7 +227,7 @@ It reads the run's own resolved config (`configs/eval.json` on dev59, `config.to
|
||||
0.3.0) and prints, beside the means:
|
||||
|
||||
- `sampling` — the run's own record of `max_tokens` / `chat_template_kwargs`. A run whose
|
||||
`sampling` is `{}` was sampled at spark-1's server-side default, which is
|
||||
`sampling` is `{}` was sampled at the reference node's server-side default, which is
|
||||
`enable_thinking: false`.
|
||||
- `traces_with_reasoning` — assistant messages carrying `reasoning_content` (or a literal
|
||||
`<think>` block). **This is the only in-trace evidence that the model was allowed to
|
||||
@@ -246,12 +249,12 @@ from `run-20260821-1401`, with `reward_mean_from_eval_log` agreeing in every cas
|
||||
|
||||
|
||||
**The run: `outputs/run-20260821-1401/`.** Seven data environments, 32 rollouts each,
|
||||
224 episodes, **223 scored**. Model `brain-qwen38-dspark` (Qwen3.8-27B) on spark-1.
|
||||
224 episodes, **223 scored**. Model `brain-qwen38-dspark` (Qwen3.8-27B) on the reference node.
|
||||
Wall-clock 14:03 → 14:58, 55 minutes, `-c 8`.
|
||||
|
||||
> ⚠️ **Sampling configuration for every number in this section — read it before quoting
|
||||
> one.** `verifiers==0.3.0`; `sampling = {}`, i.e. no `max_tokens` and no
|
||||
> `chat_template_kwargs`, so spark-1's server-side default applied and
|
||||
> `chat_template_kwargs`, so the reference node's server-side default applied and
|
||||
> **`enable_thinking` was `false`**; `temperature`, `top_p` and `reasoning_effort` all
|
||||
> unset; `-c 8`; harness `null`; agent runtime `subprocess`. `traces_with_reasoning` is
|
||||
> **0 of 224**. These are thinking-**off** numbers. §6 re-measures the same eight
|
||||
@@ -274,7 +277,7 @@ of this run got six of seven means wrong by taking the unweighted mean.
|
||||
| `bot-detection` | 32 | 32 | 0 | **0.3488** | 0.3488 |
|
||||
| `grand-exchange` | 32 | 32 | 0 | **0.0055** | 0.0055 |
|
||||
|
||||
The single errored episode is a genuine spark-1 gateway timeout (`ProviderError`, 504) in
|
||||
The single errored episode is a genuine the reference node gateway timeout (`ProviderError`, 504) in
|
||||
`drop-table-inference` — not a scoring bug. **Rank on `_attempted`** (errors as 0); the
|
||||
`_scored` column is Arena's own "score what survived" trap in reporting form.
|
||||
|
||||
@@ -326,10 +329,10 @@ Appended 2026-08-21, after re-measuring `grand-exchange`.
|
||||
|
||||
### The cause
|
||||
|
||||
spark-1's SGLang is started with a default that overrides the chat template:
|
||||
the reference node's SGLang is started with a default that overrides the chat template:
|
||||
|
||||
```bash
|
||||
curl -s http://100.127.247.67:8001/get_server_info | jq '.default_chat_template_kwargs, .reasoning_parser'
|
||||
curl -s "${SPARK_BASE_URL%/v1}/get_server_info" | jq '.default_chat_template_kwargs, .reasoning_parser'
|
||||
# {"enable_thinking": false}
|
||||
# "qwen3"
|
||||
```
|
||||
@@ -410,7 +413,7 @@ buy limit — it simply does not converge inside 16k:
|
||||
|
||||
A single rollout at `-c 1` (`outputs/thinking/smoke/`) does the same: 486 seconds,
|
||||
`completion_tokens` 16384, `reasoning_content` 30,121 characters, `content` empty, reward
|
||||
0.000. So does the same prompt sent straight at spark-1 with `curl`, no verifiers in the
|
||||
0.000. So does the same prompt sent straight at the reference node with `curl`, no verifiers in the
|
||||
path — `reasoning_tokens` 16390, `finish_reason` length, content empty. **This is not a
|
||||
harness bug, and it is not fixed by raising `max_tokens`** — see the next section for why.
|
||||
|
||||
@@ -427,7 +430,7 @@ DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=600.0, write=600.0, pool=600.0
|
||||
```
|
||||
|
||||
`BaseClientConfig` exposes `base_url`, `api_key_var` and `headers` and nothing else. **There
|
||||
is no config key and no CLI flag for this timeout.** At spark-1's ~34 tok/s single-stream it
|
||||
is no config key and no CLI flag for this timeout.** At the reference node's ~34 tok/s single-stream it
|
||||
caps one call at roughly 20,000 completion tokens, and far fewer under concurrency.
|
||||
|
||||
That is not theoretical. `outputs/thinking/ge-think-n8.log` is 8 tasks at `-c 8` with
|
||||
@@ -463,7 +466,7 @@ false`. The only lever is `max_tokens`, and `max_tokens` cuts the answer, not th
|
||||
Three consequences for anyone re-running this suite against a reasoning model:
|
||||
|
||||
1. **Keep `-c` low and `max_tokens` under the wall**, or budget the run knowing calls will be
|
||||
cut at ten minutes. spark-1 is shared — another lane's eval was running against it during
|
||||
cut at ten minutes. the reference node is shared — another lane's eval was running against it during
|
||||
this measurement, and contention alone moves a rollout across the line.
|
||||
2. **A timed-out run leaves no evidence.** Zero traces, exit code 0. `count_rewards.py`
|
||||
reports on a file that does not exist. This is the same trap as §3 with a bigger blast
|
||||
@@ -523,7 +526,7 @@ recorded.
|
||||
- Put the sampling configuration in the run's own name or its notes. A number that changes
|
||||
by two orders of magnitude with one server-side default is not a capability score unless
|
||||
the configuration travels with it.
|
||||
- Ask spark-1 what it defaults to (`/get_server_info`) rather than assuming a model's own
|
||||
- Ask the reference node what it defaults to (`/get_server_info`) rather than assuming a model's own
|
||||
default applies.
|
||||
- If `reasoning_tokens` matters to you, read it off the provider — the trace drops it.
|
||||
|
||||
@@ -539,7 +542,7 @@ achievable today" — was owed a run.
|
||||
> **Sampling configuration for every number in this section.** `verifiers==0.3.1`
|
||||
> (`read=None`); `chat_template_kwargs = {enable_thinking = true}`; `max_tokens = 49152`;
|
||||
> `temperature`, `top_p`, `reasoning_effort` unset; `-c 8`; harness `null`; agent runtime
|
||||
> `subprocess`; model `brain-qwen38-dspark` on spark-1. The configs are
|
||||
> `subprocess`; model `brain-qwen38-dspark` on the reference node. The configs are
|
||||
> `outputs/thinking-n32/*.toml`, the sweep is `outputs/thinking-n32/run_all.sh`, the runs
|
||||
> are `outputs/thinking-n32/run/<env>/`.
|
||||
|
||||
@@ -625,7 +628,7 @@ long one (0.6465).
|
||||
⚠️ **Whatever mean this run finishes with is a truncation artefact, exactly like
|
||||
`grand-exchange`'s 0.1667 was.** Do not publish it as a capability score, and do not read
|
||||
it as evidence about the mixture. The comparison `grand-exchange` earned — a distribution
|
||||
where 30 of 32 finish — this environment has not: it needs a larger `max_tokens` (spark-1's
|
||||
where 30 of 32 finish — this environment has not: it needs a larger `max_tokens` (the reference node's
|
||||
context is 262,144, so there is room) and a correspondingly larger wall-clock budget. One
|
||||
49,152-token rollout costs roughly 25 minutes at `-c 8` on this box.
|
||||
|
||||
|
||||
@@ -188,7 +188,7 @@ It scored zero because **31 of 32 rollouts placed no orders at all**: 18 complet
|
||||
`{"expected_profit": 0, "orders": []}`, `finish_reason=stop`. `planned` had mean 65.6 and
|
||||
median 0 — one rollout in 32 planned anything at all; `profit_ratio` maxed at 0.323, so even
|
||||
that one reached 29% of the reference. Nothing in this environment was measured. See
|
||||
`FIRST_EVAL.md` §5 — spark-1 serves with
|
||||
`FIRST_EVAL.md` §5 — the reference node serves with
|
||||
`default_chat_template_kwargs = {"enable_thinking": false}`, and this environment is the one
|
||||
that cannot be answered without arithmetic.
|
||||
|
||||
@@ -258,7 +258,7 @@ uv run --with regex python probe.py # floors and ceilings, per c
|
||||
`tools/regate.py` rebuilds each task from the seed in its trace, re-runs the shipped scorer
|
||||
over the model's own reply, and asks a candidate predicate directly. That is a better
|
||||
measurement than a fresh sample would have been: it holds the model's behaviour fixed and
|
||||
varies only the reward, and it costs nothing on spark-1. Its `recorded_mean` reproduces
|
||||
varies only the reward, and it costs nothing on the reference node. Its `recorded_mean` reproduces
|
||||
0.3488, 0.4432 and 0.4040 exactly, which is the cross-check that it is replaying the same
|
||||
population `eval.log` scored.
|
||||
|
||||
|
||||
+1
-1
@@ -5,7 +5,7 @@ harness, trainer, runtime marketplace, or environment hub.
|
||||
|
||||
- Every environment pins stable `verifiers==0.3.1`.
|
||||
- Pure-trace environments select Verifiers' upstream `null` harness.
|
||||
- Local model-free and spark-1 runs name the `subprocess` runtime explicitly so
|
||||
- Local model-free and the reference node runs name the `subprocess` runtime explicitly so
|
||||
tests never provision a paid Prime sandbox.
|
||||
- Prime Hub is the public distribution target for approved environment wheels.
|
||||
- Prime-RL consumes the same taskset packages for training; Forge owns the
|
||||
|
||||
Reference in New Issue
Block a user