Import upstream Verifiers episodes into Bench

2026-08-25 13:08:02 -07:00
parent 006feee0f7
commit 68a436d7ea
5 changed files with 496 additions and 2 deletions
@@ -198,3 +198,33 @@ reverse-engineer the private set (D3) by probing it one sample at a time.
 **Why.** The product does not need a public code-execution service. Manual approval is the intended workflow and keeps ownership of scarce hardware and private evaluation data clear. The Lumbridge control plane stores the inert review record with status `suggested`; only an owner can advance its lifecycle.

 **Supersedes D8 for product behavior.** The sandbox described in D8 is no longer a launch dependency because no public suggestion is automatically executed. If automated third-party execution is ever proposed again, it requires a new explicit decision and the full D8 isolation boundary first.
+
+---
+
+## D11 — Import Verifiers episodes; do not add another harness
+
+**Decision.** Inspect AI remains Bench's native execution engine. Online Tera
+and Prime-RL evaluations run in upstream Prime Verifiers and enter Bench through
+the durable Verifiers v1 `traces.jsonl` artifact. Bench parses that artifact
+into its existing `QualityResult` and `SampleOutcome` fields; the scorecard
+schema is not changed.
+
+One Verifiers episode becomes one Bench sample. Only trainable traces contribute
+to its score, so a modeled user or judge seat does not dilute the assistant's
+result. By default the score is the mean summed weighted reward of those traces;
+a named trace metric can be selected explicitly. The card stores episode and
+trace ids, task identity hashes, reward names, pass/fail and at most the normal
+400-character assistant excerpt. It never copies task data, prompts, tool
+arguments, error messages or the full conversation.
+
+**Why.** Re-running a Verifiers environment through an Inspect-shaped adapter
+would duplicate tool-loop and multi-agent behavior and could change the thing
+being measured. The completed episode is already the evidence boundary. Bench's
+unique work is longitudinal comparison against an exact model/host target, not
+owning every runner.
+
+**Cost.** Inspect and Verifiers logs have different notions of a sample and a
+headline score. The import command therefore requires a target and task name,
+records its pass threshold and source-file digest, and labels sampling as
+Verifiers rather than pretending Inspect produced it. Comparability still
+requires the same target snapshot, task fingerprint, score source and threshold.