Karti Tripathi
107 lines
4.4 KiB
Markdown
107 lines
4.4 KiB
Markdown
# Lumbridge Bench
|
||
|
||
Private evaluation and serving-performance evidence for models running on Lumbridge Compute.
|
||
|
||
Two questions, one score card:
|
||
|
||
1. **Is this model good enough** on the work we actually do?
|
||
2. **Is it fast enough** on our own hardware to put in front of that work?
|
||
|
||
Most eval frameworks answer only the first. A self-hosting decision needs
|
||
both, so every run produces quality scores *and* a serving perf profile for
|
||
the same `(model × host × quant × serving config)` target.
|
||
|
||
## Place in Lumbridge
|
||
|
||
Lumbridge Compute owns nodes, model artifacts, Scenes, and workload state. Lumbridge Bench owns measurement truth: versioned Evals, exact target snapshots, immutable runs, comparisons, and selected public score cards. The `kbench` command remains the compatible runner name.
|
||
|
||
## Why this exists
|
||
|
||
`your-node` currently serves `Qwen3.6-35B-A3B-NVFP4` behind the `brain` and
|
||
`local-moe` aliases — in production, with nothing measuring whether
|
||
that was the right call. This bench exists to answer that, and to be the eval
|
||
half of the loop when we fine-tune our own model.
|
||
|
||
## Install
|
||
|
||
```bash
|
||
uv venv --python 3.12
|
||
uv pip install -e ".[reference]"
|
||
```
|
||
|
||
## Use
|
||
|
||
```bash
|
||
kbench targets # what can be benchmarked
|
||
kbench tasks # what it gets benchmarked on
|
||
kbench snapshot your-node # live serving flags, for models.yaml
|
||
|
||
kbench perf brain # perf sweep only (no task authoring needed)
|
||
kbench run brain --limit 50 # quality + perf -> results/*.json
|
||
kbench results # every score card recorded so far
|
||
kbench card results/<file> # re-render one
|
||
```
|
||
|
||
Raw Inspect logs land in `logs/` (gitignored). View them with:
|
||
|
||
```bash
|
||
.venv/bin/inspect view --log-dir logs
|
||
```
|
||
|
||
## Layout
|
||
|
||
```
|
||
kbench/
|
||
registry/ targets: (model × host × quant × serving config × checkpoint)
|
||
tasks/ task catalog — reference tier and signal tier
|
||
scorers/ custom + model-graded scorers
|
||
perf.py async load generator, any OpenAI-compatible endpoint
|
||
results.py the committed result schema
|
||
run.py orchestration: target + tasks + perf -> score card
|
||
data/
|
||
public/ a few example samples, shown on the site
|
||
private/ the real holdout — never leaves this repo
|
||
canary/ GUID-tagged samples for proving contamination later
|
||
results/ one JSON per run, committed. the git history IS the record.
|
||
docs/ DECISIONS.md — why any of this is shaped the way it is
|
||
```
|
||
|
||
## Two tiers of task
|
||
|
||
| tier | source | purpose |
|
||
|---|---|---|
|
||
| **reference** | `inspect_evals`, unmodified | calibration + smoke test. Public, likely contaminated. Not a ranking. |
|
||
| **signal** | ours, private | the actual measurement |
|
||
|
||
If a reference score lands far from published values, the harness is broken,
|
||
not the model. That is what it is for.
|
||
|
||
## Rules that are not negotiable
|
||
|
||
- **The private eval set never becomes public.** Not a secrets concern
|
||
primarily — a public test set gets scraped into the next training corpus and
|
||
stops measuring anything. See `docs/DECISIONS.md#d3`.
|
||
- **Scrub samples at authoring time.** No keys, tokens, tailscale addresses,
|
||
or customer data. Repo privacy is defence in depth, not the control.
|
||
- **Never publish a capped run as a full run.** `--limit` is recorded in the
|
||
result and surfaced on the card.
|
||
- **Results are only comparable across identical serving flags.** Use
|
||
`kbench snapshot` rather than trusting memory.
|
||
|
||
## Manual model suggestions
|
||
|
||
The public site accepts suggestions, not executable submissions. A suggestion records a Hugging Face reference and optional notes in the Lumbridge control plane for Karti to review. Nothing downloads or runs automatically; Karti verifies the model card, license, files, compatibility, footprint, and serving path before manually scheduling any run.
|
||
|
||
Built on [Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai) as a
|
||
dependency, not a fork — see `docs/DECISIONS.md#d1`.
|
||
|
||
## License
|
||
|
||
Apache-2.0 — see [LICENSE](LICENSE) and [NOTICE](NOTICE).
|
||
|
||
The private signal set, the contamination canary, and the sample seeder are **not**
|
||
published. Publishing an eval destroys it: the signal set is worthless once it has been
|
||
crawled, and a published canary GUID is reproduced by every model that ever read the
|
||
repo, so the detector would report contamination for everyone forever. `kbench` ships
|
||
with the public example family; you author your own private families locally.
|