Import upstream Verifiers episodes into Bench
This commit is contained in:
@@ -38,6 +38,7 @@ kbench snapshot your-node # live serving flags, for models.yaml
|
||||
|
||||
kbench perf brain # perf sweep only (no task authoring needed)
|
||||
kbench run brain --limit 50 # quality + perf -> results/*.json
|
||||
kbench import-verifiers brain <run-dir> --task <environment-name>
|
||||
kbench results # every score card recorded so far
|
||||
kbench card results/<file> # re-render one
|
||||
```
|
||||
@@ -48,6 +49,13 @@ Raw Inspect logs land in `logs/` (gitignored). View them with:
|
||||
.venv/bin/inspect view --log-dir logs
|
||||
```
|
||||
|
||||
Prime Verifiers v1 `traces.jsonl` can enter the same scorecard with
|
||||
`kbench import-verifiers`. Bench reads the upstream episode artifact; it does
|
||||
not execute or copy Verifiers' harness. The importer stores episode scores,
|
||||
task-content fingerprints and short assistant excerpts, never task data, tool
|
||||
arguments or full conversations. Inspect AI remains the native runner for
|
||||
Bench tasks.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
@@ -58,6 +66,7 @@ kbench/
|
||||
perf.py async load generator, any OpenAI-compatible endpoint
|
||||
results.py the committed result schema
|
||||
run.py orchestration: target + tasks + perf -> score card
|
||||
verifiers_import.py upstream Verifiers episode JSONL -> QualityResult
|
||||
data/
|
||||
public/ a few example samples, shown on the site
|
||||
private/ the real holdout — never leaves this repo
|
||||
|
||||
Reference in New Issue
Block a user