new-environment pipeline: scripted arms required, model arms captured when spark-1 is up
An environment ships on its scripted agents alone — they are what the reward flip is built on and they need no model. A recorded model attempt is captured once from spark-1 for the Watch tab when it is serving, and the environment does not block on it when it is not.
This commit is contained in:
@@ -154,11 +154,17 @@ const captured = await agent(`${BASE}
|
||||
|
||||
## YOUR STAGE: real rollouts.
|
||||
|
||||
Extend \`envs/capture.py\` for taskset "${slug}" and capture seeds ${seeds} for at least three arms: a weak one, a strong one, and a generated reference policy.
|
||||
Extend \`envs/capture.py\` for taskset "${slug}".
|
||||
|
||||
**Two kinds of arm, with different rules.**
|
||||
|
||||
*Scripted arms are REQUIRED* — the reference policy and at least one weaker scripted policy (the probe ladder's rungs are the obvious candidates). They are generated by code, need no model, and are what the Reward tab's ranking flip is built on. An environment ships with these alone.
|
||||
|
||||
*Model arms are an ENHANCEMENT* — a real model's recorded attempt for the Watch tab, captured once from spark-1. Check it first: \`curl http://100.127.247.67:8001/v1/models\`. If it is serving, capture seeds ${seeds} for a thinking-off and a thinking-on arm. If it is not, ship without them, say so plainly in your report, and leave a note in \`envs/${pkg}/README.md\` that model arms are still to be captured — do NOT block the environment on it and do NOT invent fixtures.
|
||||
|
||||
⚠️ **spark-1 serves one model and is single-stream — run the arms one after another, never concurrently.** Check it is up first: \`curl http://100.127.247.67:8001/v1/models\`. A thinking-enabled arm can take minutes per seed; run it in the background, redirect output to a FILE (piping to head/tail block-buffers Python's stdout and a working run looks hung), and poll the written fixtures rather than the log.
|
||||
|
||||
Include at least one run the agent loses. Then \`uv run python envs/build_manifest.py\`.
|
||||
Include at least one run an agent loses. Then \`uv run python envs/build_manifest.py\`.
|
||||
|
||||
**Your gate:** \`uv run python envs/verify_fixtures.py\` — every fixture replays and reproduces its own recorded rewards at delta 0.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user