Import upstream Verifiers episodes into Bench
CI / verify (push) Successful in 40s
CI / deploy (push) Failing after 1m21s

This commit is contained in:
2026-08-25 13:08:02 -07:00
parent 006feee0f7
commit 68a436d7ea
5 changed files with 496 additions and 2 deletions
+9
View File
@@ -38,6 +38,7 @@ kbench snapshot your-node # live serving flags, for models.yaml
kbench perf brain # perf sweep only (no task authoring needed)
kbench run brain --limit 50 # quality + perf -> results/*.json
kbench import-verifiers brain <run-dir> --task <environment-name>
kbench results # every score card recorded so far
kbench card results/<file> # re-render one
```
@@ -48,6 +49,13 @@ Raw Inspect logs land in `logs/` (gitignored). View them with:
.venv/bin/inspect view --log-dir logs
```
Prime Verifiers v1 `traces.jsonl` can enter the same scorecard with
`kbench import-verifiers`. Bench reads the upstream episode artifact; it does
not execute or copy Verifiers' harness. The importer stores episode scores,
task-content fingerprints and short assistant excerpts, never task data, tool
arguments or full conversations. Inspect AI remains the native runner for
Bench tasks.
## Layout
```
@@ -58,6 +66,7 @@ kbench/
perf.py async load generator, any OpenAI-compatible endpoint
results.py the committed result schema
run.py orchestration: target + tasks + perf -> score card
verifiers_import.py upstream Verifiers episode JSONL -> QualityResult
data/
public/ a few example samples, shown on the site
private/ the real holdout — never leaves this repo