arena/MULTI_TURN.md

# Scope: the multi-turn gap

Status: scoping document, 2026-08-21. Nothing here is built yet.

Every environment in this repository sets `[env.agent.harness] id = "null"` — one
completion, no tools, no runtime. That is the right choice for a pure-trace data
environment and it is also the whole of what Arena currently measures. Nothing here
scores an agent *acting*: choosing what to look at next, paying for a turn, or
revising after seeing a consequence.

This document scopes closing that gap. The headline finding is that it is three
separable tiers, not one, and the cheapest tier needs no new infrastructure at all.

## What upstream actually requires

Read from stable `verifiers==0.3.1` and the matching checkout under
`~/vendor/prime-intellect/verifiers`, not
from blog posts.

**Multi-turn does not imply a container.** `vf.Env.run(task, agents)` programs the
control flow host-side. `agents.<role>.interaction(task)` opens a rollout and holds
it open; `await interaction.turn(observation)` sends one user turn, runs one harness
segment, and returns a `Segment` with `.last_reply` and `.terminated`. The simulator
runs in-process, as the user. The harness can stay `null` throughout.

The reference implementation is `verifiers/v1/tasksets/textarena/taskset.py` — 90
lines, drives a real game engine, scores off the engine directly. Read it before
writing anything.

**An Env ships inside the taskset's own wheel.** `EnvConfig.id` empty means "the
taskset's own env, else `SingleAgentEnv`". Export the `Env` subclass from the
package `__init__.py`, exactly as `environments/wordle` exports `TextArenaEnv`, and
`[env.taskset] id = "..."` picks it up with no extra config.

**Available in the installed version** — confirmed, not assumed:

- harnesses: `null`, `bash`, `browser_use`, `claude_code`, `codex`, `hermes_agent`,
  `kimi_code`, `mini_swe_agent`, `openclaw`, `pi`, `pool`, `rlm`, `terminus_2`
- runtimes: `subprocess` (debug only — rollouts share side effects), `docker`,
  `modal`, `prime`
- prebuilt envs: `single_agent`, `best_of_n`, `agentic_judge`,
  `shared_agentic_judge`, `user_sim`

**The interception server** sits between harness and provider. It speaks each
harness's native dialect (Anthropic Messages for Claude Code, OpenAI Responses for
Codex), builds the trace live, forces sampling params the harness will not expose,
and can rewrite tool results and web-search results to block reward hacks. It is
mandatory for tiers 2 and 3 and free for tier 1.

## The three tiers

| | Mechanism | Harness | Runtime | New infra |
|---|---|---|---|---|
| **1. Interactive** | `vf.Env.run` + `interaction.turn`, simulator in-process | `null` | none | **none** |
| **2. Tool-using** | Taskset exports a `Toolset`, served as MCP | `bash` | `subprocess` / `docker` | image per env |
| **3. Agentic** | Full coding agent in a sandbox | `claude_code`, `codex`, `openclaw`, `pi` | `docker` / `prime` | image + interception + budget |

Tier 1 is the one to build first, and it is where our existing simulators already
sit. Tiers 2 and 3 are real work with real cost and should follow evidence from
tier 1, not precede it.

## What we already have that plugs in

**Six of the seven environments here own a simulator that is already steppable.**
They are one-shot only because the taskset hands the model the whole observation up
front. Each has an obvious interactive form:

| environment | one-shot today | interactive form | tier |
|---|---|---|---|
| `grand-exchange` | read 30 ticks of history, place one basket of orders | quote, see fills and the next tick, re-quote, over a turn budget — inventory and unfilled orders carry | 1 |
| `drop-table-inference` | read a fixed kill log, estimate the table | **buy** observations: each turn spends budget on more kills, then answers — measures when to stop looking | 1 |
| `fault-localisation` | read the whole incident log, name the cause | the log is *not* given: query it by service, time window and level under a query budget | 1 → 2 |
| `redaction-pressure` | write rules, scored by a scanner | two agents: a redactor and an extractor scored on what it recovers — the counterweight becomes an opponent | 1 (multi-agent) |
| `schema-migration` | emit a migration, executed against held-out rows | a real SQLite in a container the agent runs SQL against, with rollback | 3 |
| `bot-detection` | classify from six labelled examples | request labels one at a time under a budget (active learning) | 1 |
| `canary-trap` | write probes | no natural interactive form — leave one-shot | — |

**Tera's four spatial environments are already multi-turn — in the wrong language.**
`~/repos/gitea/tera/src/arena/` implements `tera.arena/v1`: Gym-style
`reset(seed, scenario)` / `step(action)` / `snapshot` / `restore` / `trace` /
`replay`, FNV-1a-64 checksums, SHA-256 source pins, train/dev splits, counterweighted
rewards, and a CI gate that proves inaction finishes below zero and a scripted
baseline finishes positive. That is a stricter contract than most of what is on the
Hub — and none of it is reachable from `uv run eval`, because it is TypeScript with
its own contract rather than a verifiers taskset.

The work there is a **bridge**, not a simulator: a Python `vf.Env` whose `run()`
drives the TS package over a subprocess or a small local HTTP loop, turning
`step()` observations into `interaction.turn()` messages and the existing component
rewards into `trace.record_reward` calls. The replay checksum makes the bridge
verifiable — a bridged episode must replay identically in the TS package or the
bridge is wrong. That is a rare property and worth building the bridge around.

⚠️ **RESOLVED 2026-08-21, 15:50 — `office-jobs-v1` EXISTS.** It was absent at 13:00 and
present at 15:50, because another session fast-forward-merged `feat/tera-ultracode-integration`
into `~/repos/gitea/tera` mid-session (reflog `8738367`). It is committed (`e378a03`,
`120eac8`, dated 2026-08-19), is in `ARENA_MANIFESTS`, is documented in `tera/ARENA.md`, and
passes `npm run arena:source-hashes`. **The true counts are twelve environments, five spatial,
twenty scenarios** — measured, not inferred. The site's original copy was right; the
correction to eleven/four/sixteen was right when made and wrong an hour later, and has been
reverted. Lesson: `tera` is a shared tree with live concurrent sessions — re-check its HEAD
before acting on a claim about what it contains.

~~⚠️ **`office-jobs-v1` does not exist.** `lumbridgecorp.com/arena` and
`services/api/src/seo.ts` both advertise five spatial environments; `ARENA_MANIFESTS`
in `src/arena/index.ts` holds four (`drive-101-v1`, `office-nav-v1`, `crow-nav-v1`,
`california-flight-v1`) and there is no `officeJobs.ts`. The real total is eleven,
not twelve. Decide whether to build it or correct the page before publishing
anything that cites the count.~~ (superseded — see above)

## The house rules under multi-turn

The three rules were written for one-shot scoring. Two survive unchanged and one
does not, and there is a fourth the interactive tier forces.

1. **Graded on what the model did not see** — unchanged and easier: a stepped
   simulator generates its held-out continuation by construction.
2. **No single-sided reward** — *harder*. Multi-turn adds degenerate strategies that
   have no one-shot analogue: stalling to burn the clock when the score is
   time-averaged, spamming cheap turns when observations are free, refusing to act
   when acting risks a penalty. Every new reward needs a counterweight in the *turn*
   dimension, not just the answer dimension.
3. **A zero floor and a reachable ceiling, demonstrated** — unchanged, but the floor
   now has to hold across a whole trajectory, not one reply.
4. **New: the budget must bind.** An interactive environment where the optimal
   policy is "use every turn" is measuring nothing the one-shot version did not.
   Observations must cost something, and the oracle must be shown to stop early.

**`probe.py` must grow degenerate *policies*, not just degenerate answers.** Today
it scores four fixed strings per environment. The interactive tier needs at minimum:
the staller (never acts), the spammer (acts every turn at random), the impatient
(answers on turn 1 with no observation), and the exhaustive (spends the entire
budget then answers). The gate stays the same shape — inaction 0.000, oracle 1.000,
exit 1 otherwise — plus a new assertion that the exhaustive policy scores *below*
the oracle. If it does not, rule 4 is violated and the environment does not ship.

This is the differentiator. Nothing in `prime-envs` (88 environments) or
`community-environments` (109) ships a degenerate-policy gate.

## Dual publishing

Both targets take the same wheel; the repo layout already mirrors verifiers' own.

- **Prime Intellect Hub**, our own namespace: `prime env push <name> -v PUBLIC`.
  We keep authorship and versioning, and the listing can link back to
  lumbridgecorp.com/arena.
- **`community-environments`**: a PR to that repo, published under the
  `primeintellect` org by their CI on a version bump. Extra distribution, but it
  hands the namespace to them. Secondary, not primary.
- **lumbridgecorp.com/arena**: unchanged — the page documents the same wheel.

⚠️ Its README still documents the **legacy** flow (`load_environment`, `vf-eval`,
`uv pip install -e .`). Ignore it; follow `verifiers/docs/v1/`.

⚠️ **Blocker: `PRIME_API_KEY` is not set on amd-server.** `prime config view` shows
`API Key: Not set`. The `pit_` key lives in karti's memory note on cloud-1
(`.claude/projects/-home-ubuntu/memory/prime-intellect.md`) and is the same
credential for `api.pinference.ai` and `api.primeintellect.ai`. Nothing pushes until
it is configured here.

## Preconditions, verified on this box

- `uv run eval @ configs/canary_trap.toml --dry-run` → exit 0, config written ✓
- stable verifiers 0.3.1 with upstream harnesses and runtimes present ✓
- Docker 29.1.3 running (tier 2/3 runtime) ✓
- spark-1 `brain` on `http://100.127.247.67:8001/v1`, model id `brain` ✓
- `prime` CLI installed (0.6.25; 0.6.26 available) ✓
- `PRIME_API_KEY` ✗ — see above

## Proposed sequence

**Phase 1 — one interactive environment, end to end.** `grand-exchange` is the
candidate: the market simulation already steps, the counterweight (liquidity, the
random-walk trap) already exists, and turning one basket into a quote/fill loop is
the smallest change that produces a genuinely new measurement. Deliverable: a
`vf.Env` in the existing wheel, a second config, and probe coverage for the four
degenerate policies. This is where rule 4 gets written for real.

**Phase 2 — extend `probe.py` to policies and re-gate everything.** Whatever
phase 1 teaches about degenerate trajectories applies to every later environment.
Do this before building a second one, not after. Note that `tests/test_probe.py`
and the wheel-build loop in `.gitea/workflows/ci.yml` hardcode the environment
list — both need editing or CI goes red.

**Phase 3 — the Tera bridge.** One Python `vf.Env` that drives the existing TS
package, validated by replaying every bridged episode through `replay()` and
requiring an identical checksum. Lands four environments at once and makes the
strongest existing asset publishable. Larger and riskier than phase 1; the
process boundary and the action encoding are the unknowns.

**Phase 4 — first Hub push**, once phases 1–3 give three to four interactive
environments alongside the seven one-shot ones. Needs the API key.

**Phase 5 — tier 2, then tier 3.** `fault-localisation` as a query-budgeted
tool-using environment, then `schema-migration` against a real database in a
container. Only after tier 1 has produced measured scores worth extending, and
with the concurrency cost of a container per rollout understood first.

## Open questions

- Does a bridged Tera episode replay to an identical checksum through a process
  boundary? If not, the bridge cannot be verified and phase 3 gets much more
  expensive.
- What is the per-rollout wall-clock cost of the `docker` runtime here, at the
  concurrency an eval sweep needs? Unmeasured. It gates tier 3 entirely.
- Should the interactive form of an environment be a **separate** taskset id
  (`grand-exchange-live`) or a config flag on the existing one? Separate ids keep
  the one-shot scores comparable across time; a flag keeps the wheel count down.
  Leaning separate.