new-environment pipeline: scripted arms required, model arms captured when spark-1 is up
ci / web (push) Failing after 1m53s
ci / python (push) Failing after 7s

An environment ships on its scripted agents alone — they are what the reward
flip is built on and they need no model. A recorded model attempt is captured
once from spark-1 for the Watch tab when it is serving, and the environment does
not block on it when it is not.
This commit is contained in:
karti-ai
2026-08-28 18:32:29 -07:00
parent 1a6bbf3309
commit 1239dc7034
2 changed files with 383 additions and 2 deletions
+8 -2
View File
@@ -154,11 +154,17 @@ const captured = await agent(`${BASE}
## YOUR STAGE: real rollouts.
Extend \`envs/capture.py\` for taskset "${slug}" and capture seeds ${seeds} for at least three arms: a weak one, a strong one, and a generated reference policy.
Extend \`envs/capture.py\` for taskset "${slug}".
**Two kinds of arm, with different rules.**
*Scripted arms are REQUIRED* — the reference policy and at least one weaker scripted policy (the probe ladder's rungs are the obvious candidates). They are generated by code, need no model, and are what the Reward tab's ranking flip is built on. An environment ships with these alone.
*Model arms are an ENHANCEMENT* — a real model's recorded attempt for the Watch tab, captured once from spark-1. Check it first: \`curl http://100.127.247.67:8001/v1/models\`. If it is serving, capture seeds ${seeds} for a thinking-off and a thinking-on arm. If it is not, ship without them, say so plainly in your report, and leave a note in \`envs/${pkg}/README.md\` that model arms are still to be captured — do NOT block the environment on it and do NOT invent fixtures.
⚠️ **spark-1 serves one model and is single-stream — run the arms one after another, never concurrently.** Check it is up first: \`curl http://100.127.247.67:8001/v1/models\`. A thinking-enabled arm can take minutes per seed; run it in the background, redirect output to a FILE (piping to head/tail block-buffers Python's stdout and a working run looks hung), and poll the written fixtures rather than the log.
Include at least one run the agent loses. Then \`uv run python envs/build_manifest.py\`.
Include at least one run an agent loses. Then \`uv run python envs/build_manifest.py\`.
**Your gate:** \`uv run python envs/verify_fixtures.py\` — every fixture replays and reproduces its own recorded rewards at delta 0.