Lumbridge Bench

AGENTS.md

@@ -0,0 +1,80 @@
+# Working guide — Lumbridge Bench
+
+Read `docs/DECISIONS.md` before changing anything structural. It records why
+the shape is the shape; the code does not.
+
+## What this repo is
+
+A Lumbridge evidence lab that answers "should we self-host this model?" — quality on our real
+work, plus serving performance on our own boxes, in one score card per
+`(model × host × quant × serving config × checkpoint)` target.
+
+Built **on** Inspect AI as a dependency. It is not a fork and must not become
+one (D1). Adding a benchmark means adding a task, not patching the harness.
+
+## Ground rules
+
+1. **Never commit anything to `data/private/` that was not scrubbed.** No API
+   keys, tokens, tailscale hostnames/IPs, customer data, or personal
+   information. Samples derived from real tool calls get sanitised at
+   authoring time. The content will be sent to third-party APIs (judges,
+   baselines) regardless of how private this repo is.
+
+2. **Never publish or export `data/private/`.** Not in the site build, not in
+   an artifact, not in a share card, not in a bug report. A public test set is
+   a dead test set (D3).
+
+3. **Never present a capped run as a full run.** `--limit` is recorded in the
+   result JSON and rendered on the card. Keep it that way.
+
+4. **Do not change `results/` schema fields in place.** Old result files must
+   stay readable. Bump `SCHEMA_VERSION` and add fields; never repurpose one.
+
+5. **Results are only comparable across identical serving flags.** Before
+   recording a run, confirm the target's flags with `kbench snapshot <host>`.
+   If they drifted, update `models.yaml` first — a result carries its own
+   snapshot, so a stale registry silently produces incomparable numbers.
+
+## Adding a reference task
+
+Reference tasks come from `inspect_evals` unmodified. Add an entry to
+`CATALOG` in `kbench/tasks/__init__.py` with its `primary_metric` (the headline
+key in the scorer's metrics) and, if sample scores are dict-valued,
+`primary_sample_key`. Do not modify the upstream task.
+
+## Adding a signal task
+
+Signal tasks are ours. Samples live in `data/private/<family>.jsonl`, one JSON
+object per line. See `data/README.md` for the format. Use `kbench add` rather
+than hand-editing so the canary tag and split assignment are applied.
+
+Assign `split` at authoring time and never move a sample between splits —
+`test` samples must never be seen during fine-tuning, or every number after
+that point is a lie you cannot detect.
+
+## Perf measurement
+
+`kbench/perf.py` deliberately defeats prefix caching (unique nonce per
+request) and fixes output length (`ignore_eos`). If you touch it, preserve
+both — without them the numbers look better and mean nothing (D6).
+
+Concurrency-1 numbers describe what a single user feels. Aggregate throughput
+at higher concurrency describes what the box can serve. Report both; they can
+differ by more than 20x on the Spark.
+
+## Environment
+
+- Runner: EQ (this box). It only needs HTTP to the serving host.
+- `your-node` serves vLLM on `:8001`, OpenAI-compatible.
+- Inspect addresses it as `openai-api/spark/<model>` with `SPARK_BASE_URL` /
+  `SPARK_API_KEY`, both derived from the registry — do not hardcode them.
+
+## Deploy
+
+The standalone site is **retired** (2026-08-01): `bench.karti.ai` permanently redirects to
+`lumbridgecorp.com/bench`, and `bench-karti` on web-host :8909 is stopped and disabled. Bench ships as
+part of Lumbridge, not as its own destination — see [docs/DEPLOY.md](docs/DEPLOY.md) before standing
+any UI back up. Supabase stores auth and the private manual-suggestion inbox in schema `bench`; it is
+not an execution queue. The `kbench` CLI runner is unaffected.
+
+Public model suggestions never execute code. The suggestion endpoint may record a reference for owner review, but it must never download, import, schedule, or run a model. D10 supersedes D8 for product behavior; every run begins with explicit owner approval.