Lumbridge Bench
This commit is contained in:
Karti Tripathi
@@ -0,0 +1,200 @@
|
||||
# Decisions
|
||||
|
||||
Why this repo is shaped the way it is. The reasoning is not recoverable from
|
||||
the code, which is the only reason this file exists. Append, don't rewrite --
|
||||
a superseded decision is more useful with its replacement attached than
|
||||
deleted.
|
||||
|
||||
---
|
||||
|
||||
## D1 — Depend on `inspect_ai`, do not fork it
|
||||
|
||||
**Decision.** `bench.karti.ai` pip-installs `inspect-ai` and `inspect-evals`.
|
||||
It is not a fork.
|
||||
|
||||
**Why.** The obvious move is to fork [inspect_ai][1] and grow from there. The
|
||||
numbers say otherwise: 407 MB, ~2 commits/day, pushed daily. Forking means
|
||||
either abandoning that stream of fixes or rebasing against it forever, and we
|
||||
would be paying that for nothing — we are not changing the execution engine,
|
||||
the scorer protocol, the sandbox, or the viewer. We are adding tasks, and
|
||||
tasks are the documented extension point.
|
||||
|
||||
The decisive evidence is upstream's own layout: `inspect_evals` is a
|
||||
**separate package that depends on `inspect_ai`**, holding 200+ evals, none of
|
||||
them in a fork of core. The maintainers already built the seam. This repo is
|
||||
simply our private `inspect_evals`.
|
||||
|
||||
**When to revisit.** If we ever need to change core execution behaviour. Then
|
||||
fork from a known-good tag, for a stated reason. Not speculatively.
|
||||
|
||||
**Rejected alternative — build our own harness.** Reimplements sandboxed tool
|
||||
execution, retry/rate-limit handling, parallel rollouts, log format, viewer,
|
||||
and judge protocol, to arrive somewhere functionally identical. All of this
|
||||
bench's value is in the task data; none is in the runner.
|
||||
|
||||
[1]: https://github.com/UKGovernmentBEIS/inspect_ai
|
||||
|
||||
---
|
||||
|
||||
## D2 — A target is `(model × host × quant × serving config × checkpoint)`
|
||||
|
||||
**Decision.** The registry is keyed on the full tuple, not on model name.
|
||||
See `kbench/registry/__init__.py`.
|
||||
|
||||
**Why.** On unified-memory boxes the serving config moves throughput more than
|
||||
a model swap does. The live your-node server runs `--enforce-eager` (no CUDA
|
||||
graphs) and `--gpu-memory-utilization 0.55` (45% of unified memory unused,
|
||||
capping KV cache and therefore the concurrency ceiling). A result recorded
|
||||
against "Qwen3.6-35B-A3B" without those flags is not reproducible and not
|
||||
comparable to anything.
|
||||
|
||||
`checkpoint` is in the key from day one because once we fine-tune, we will
|
||||
compare `karti-7b@step2000` against `karti-7b@step4000` far more often than we
|
||||
compare Qwen against Llama. Adding it later would invalidate every result
|
||||
recorded before it.
|
||||
|
||||
**Consequence.** Every run snapshots the full target into its result file.
|
||||
Referencing a registry id would go stale the first time `models.yaml` is
|
||||
edited.
|
||||
|
||||
---
|
||||
|
||||
## D3 — Three-tier data split, and the eval set stays private
|
||||
|
||||
**Decision.** `data/public/` (a handful of shown examples), `data/private/`
|
||||
(the real holdout), `data/canary/` (GUID strings embedded in samples).
|
||||
|
||||
**Why private — contamination, not secrets.** The instinctive reason to keep
|
||||
eval data private is that samples derived from real tool calls might contain
|
||||
sensitive material. That is true but secondary, and it has its own fix (see
|
||||
below). The primary reason is that **a public test set gets scraped into the
|
||||
next training corpus and stops measuring capability**. That is how MMLU,
|
||||
GSM8K, and HumanEval died. Every benchmark with signal left in 2026 keeps a
|
||||
private holdout. If our samples ever go public, the bench's useful life is
|
||||
about one training cycle.
|
||||
|
||||
**On secrets specifically.** A private repo is not a secret-management
|
||||
strategy. Anything derived from real workloads is scrubbed **at authoring
|
||||
time** — no keys, no tokens, no tailscale addresses, no customer data —
|
||||
because that content would also leak through the model we send it to (often
|
||||
someone else's API) and through any generated share card. Repo privacy is
|
||||
defence in depth, not the control.
|
||||
|
||||
**Canary tier.** Samples carry embedded GUIDs. Later we can grep a candidate
|
||||
model for those strings and *prove* contamination rather than suspect it.
|
||||
Costs nothing now; impossible to add retroactively.
|
||||
|
||||
---
|
||||
|
||||
## D4 — Per-sample outcomes in git; full outputs stay local
|
||||
|
||||
**Decision.** `results/*.json` stores each sample's score, pass/fail, and a
|
||||
400-char excerpt. Full model outputs stay in the raw Inspect `.eval` log,
|
||||
which is gitignored.
|
||||
|
||||
**Why.** An aggregate score tells you a checkpoint got worse. Per-sample
|
||||
diffs tell you *which capability broke* — the difference between a leaderboard
|
||||
and a debugging tool, and the whole reason this is usable for fine-tuning.
|
||||
Storing complete transcripts, though, would bloat the repo (thousands of
|
||||
samples × multi-KB reasoning traces × hundreds of runs) and would persist
|
||||
sensitive echoes of signal-tier prompts into git history forever. Git carries
|
||||
the shape of the failure; the local log carries the transcript.
|
||||
|
||||
---
|
||||
|
||||
## D5 — Two task tiers: reference and signal
|
||||
|
||||
**Decision.** `reference` tasks come from `inspect_evals` unmodified.
|
||||
`signal` tasks are ours and private. Both appear on the leaderboard, visually
|
||||
distinguished, with reference scores captioned *public, likely contaminated,
|
||||
calibration only*.
|
||||
|
||||
**Why.** A bench made only of novel private tasks is unfalsifiable to an
|
||||
outside reader — nobody, including future us, can tell whether a strange score
|
||||
means the model is bad or the harness is broken. The reference tier is the
|
||||
credibility anchor and the smoke test: if our IFEval lands far from published
|
||||
values, the harness is wrong.
|
||||
|
||||
**Why IFEval first.** Its scorer is deterministic, so it needs no judge model
|
||||
— no per-run cost, and no judge drift between runs to confound comparisons.
|
||||
It is fast enough for a tight debug loop. And instruction adherence is the
|
||||
property that actually decides whether a small local model can hold a system
|
||||
prompt in production, which is the live question about `brain`.
|
||||
|
||||
---
|
||||
|
||||
## D6 — Self-contained perf load generator, not `vllm bench serve`
|
||||
|
||||
**Decision.** `kbench/perf.py` is an async httpx load generator against any
|
||||
OpenAI-compatible endpoint.
|
||||
|
||||
**Why.** Submissions are the point of the public site, so we cannot assume
|
||||
vLLM — the harness must measure SGLang, llama.cpp, ollama, or whatever a
|
||||
submitted model ends up served under. It also needs nothing installed on the
|
||||
target box, and it lets us fix the metric definitions so numbers stay
|
||||
comparable across engines instead of inheriting each engine's conventions.
|
||||
|
||||
**Two traps it exists to avoid.** Both produce great-looking meaningless
|
||||
numbers:
|
||||
|
||||
1. **Prefix caching.** vLLM caches shared prompt prefixes. Identical
|
||||
concurrent prompts make prefill nearly free after the first request and
|
||||
overstate throughput badly. Every request gets a unique leading nonce.
|
||||
2. **Variable output length.** If the model decides when to stop, concurrency
|
||||
levels finish at different token counts and tok/s stops being comparable.
|
||||
We send `ignore_eos` so every request emits exactly `output_tokens`.
|
||||
|
||||
---
|
||||
|
||||
## D7 — Private gitea now; the harness may open-source later, the data never
|
||||
|
||||
**Decision.** `karti/lumbridge-bench` on gitea, private. Results public.
|
||||
|
||||
_(Updated 2026-08-01: the repo was `karti/bench.karti.ai`, now archived. "Site public" meant the
|
||||
standalone bench.karti.ai leaderboard, which is retired — results surface through
|
||||
`lumbridgecorp.com/bench` instead. The public/private split below is unchanged.)_
|
||||
|
||||
**Why.** The leaderboard is only valuable if people can see it; the eval data
|
||||
is only valuable if they cannot (D3). If we later open-source, the harness
|
||||
splits out and `data/private/` stays behind — which is why the seam between
|
||||
`kbench/` (mechanism) and `data/` (content) is kept clean from the start. A
|
||||
clean seam costs nothing today and keeps the decision reversible.
|
||||
|
||||
---
|
||||
|
||||
## D8 — Submissions execute untrusted code; sandbox before launch
|
||||
|
||||
**Status.** Planned, not yet implemented. Blocking for the public submit form.
|
||||
|
||||
Anonymous "drop a HuggingFace link and we'll benchmark it" means running
|
||||
arbitrary code on our own hardware. HF repos can ship custom modeling files
|
||||
that execute on load, and `.bin` checkpoints are pickles — remote code
|
||||
execution by design. Non-negotiable gates:
|
||||
|
||||
- `safetensors` only; `trust_remote_code=false`; hard-fail otherwise
|
||||
- eval runs in a container with **no network egress** and no tailscale route
|
||||
- pre-flight via the HF API before queueing: reject if params exceed the
|
||||
target box, if config is unreadable, if the repo is gated
|
||||
- one box, one queue: show estimated wait, cap submissions per IP per day
|
||||
|
||||
---
|
||||
|
||||
## D9 — Anonymous submissions get aggregate scores only
|
||||
|
||||
**Decision.** Signed-in users (115karti invite) get per-sample breakdowns and
|
||||
private-by-default results. Anonymous submissions get an aggregate score,
|
||||
published.
|
||||
|
||||
**Why.** This reads like feature gating but it is primarily a contamination
|
||||
defence: detailed per-sample feedback is exactly how someone would
|
||||
reverse-engineer the private set (D3) by probing it one sample at a time.
|
||||
|
||||
---
|
||||
|
||||
## D10 — Public input is a manual suggestion, never an execution queue
|
||||
|
||||
**Decision.** The public action is **Suggest a model**. It records a Hugging Face reference and optional context for owner review. It never downloads, imports, schedules, or executes a model. Karti manually inspects the model card, license, files, compatibility, footprint, and serving path, then explicitly decides whether to run it.
|
||||
|
||||
**Why.** The product does not need a public code-execution service. Manual approval is the intended workflow and keeps ownership of scarce hardware and private evaluation data clear. The Lumbridge control plane stores the inert review record with status `suggested`; only an owner can advance its lifecycle.
|
||||
|
||||
**Supersedes D8 for product behavior.** The sandbox described in D8 is no longer a launch dependency because no public suggestion is automatically executed. If automated third-party execution is ever proposed again, it requires a new explicit decision and the full D8 isolation boundary first.
|
||||
+106
@@ -0,0 +1,106 @@
|
||||
# Deploy
|
||||
|
||||
**Migration status: COMPLETE (2026-08-01).** `karti/lumbridge-bench` is the canonical and only
|
||||
repository — `karti/bench.karti.ai` is archived in gitea, fully contained in this repo's `main`.
|
||||
|
||||
**The standalone site is retired.** Bench is a Lumbridge product, not its own destination, so
|
||||
`bench.karti.ai` now serves a permanent redirect to `https://lumbridgecorp.com/bench`. The
|
||||
`bench-karti.service` backend is stopped and disabled on web-host.
|
||||
|
||||
What that leaves on the box:
|
||||
|
||||
- Caddy vhost for `bench.karti.ai` is now `redir https://lumbridgecorp.com/bench permanent`
|
||||
(a pre-change Caddyfile backup is at `/etc/caddy/Caddyfile.bak-bench-<timestamp>`)
|
||||
- DNS `bench.karti.ai A 170.9.14.61` and the Let's Encrypt cert stay — the redirect needs both
|
||||
- checkout at `/home/ubuntu/workspace/bench.karti.ai` remains, now on `origin` =
|
||||
`karti/lumbridge-bench`; the `bench-karti` unit is stopped and disabled, not removed
|
||||
|
||||
**To bring a Bench UI back up**, deploy it as part of Lumbridge rather than reviving the subdomain.
|
||||
Use `deploy/lumbridge-bench.service` and a fresh checkout; do not re-enable `bench-karti`.
|
||||
|
||||
The eval runner (`kbench`) is unaffected by any of this — it is a CLI, not the site.
|
||||
|
||||
## Gotchas hit during the first deploy
|
||||
|
||||
- **Caddy binds `10.0.0.2`, not loopback.** Testing a vhost from the box with
|
||||
`curl --resolve host:443:127.0.0.1` returns `000` and looks like an outage.
|
||||
Use `--resolve host:443:10.0.0.2`.
|
||||
- **Order matters: DNS before the vhost reload.** Caddy will attempt ACME the
|
||||
moment the vhost loads; if the A record does not exist yet the challenge
|
||||
fails with NXDOMAIN and the retry sits in backoff for minutes. Adding DNS
|
||||
first, or reloading Caddy again after adding it, gets the cert immediately.
|
||||
- **The `oci` CLI is not on PATH on build-host** — it lives at `~/bin/oci`. And
|
||||
`oci dns zone list` needs `--compartment-id`, but `oci dns zone get
|
||||
--zone-name-or-id karti.ai` does not, which is the easier way in.
|
||||
|
||||
## Pre-existing, unrelated
|
||||
|
||||
Caddy's log shows recurring ACME failures for `www.office.karti.ai` and
|
||||
`www.og.karti.ai` — NXDOMAIN, no `www` A records exist. Predates this deploy
|
||||
and does not affect the apex domains, but it means those two sites retry ACME
|
||||
forever. Either add the `www` records or drop them from their vhosts.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| repo | `karti/lumbridge-bench` on gitea, **private** |
|
||||
| host | web-host, port **8909** |
|
||||
| service | `bench-karti` compatibility service (systemd) |
|
||||
| checkout | `/home/ubuntu/workspace/bench.karti.ai` compatibility path |
|
||||
| domain | `bench.karti.ai` → web-host, DNS via oci on build-host (profile `cloud2-sanjose`) |
|
||||
|
||||
## One-time setup on web-host
|
||||
|
||||
```bash
|
||||
cd /home/ubuntu/workspace
|
||||
git clone https://gitea.example.internal:8444/karti/lumbridge-bench.git bench.karti.ai
|
||||
cd bench.karti.ai/site
|
||||
cp .env.example .env.local # loopback Lumbridge control plane
|
||||
npm install && npm run build
|
||||
|
||||
sudo cp ../deploy/bench-karti.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now bench-karti
|
||||
curl -s localhost:8909/api/health
|
||||
```
|
||||
|
||||
Then add the caddy/nginx vhost for `bench.karti.ai` → `localhost:8909` and the
|
||||
DNS A record, matching the other `*.karti.ai` sites.
|
||||
|
||||
## CI secret
|
||||
|
||||
The workflow needs one repo secret, `CLOUD2_SSH_KEY`: the private half of a
|
||||
deploy key whose public half is in `ubuntu@web-host`'s `authorized_keys`.
|
||||
Generate a dedicated key rather than reusing another site's.
|
||||
|
||||
```bash
|
||||
ssh-keygen -t ed25519 -f bench-karti-ci-deploy -C "bench-karti-ci-deploy" -N ""
|
||||
# public half -> web-host authorized_keys; private half -> gitea repo secret
|
||||
```
|
||||
|
||||
## Suggestion inbox
|
||||
|
||||
The public form forwards an inert record to the Lumbridge control plane at
|
||||
`LUMBRIDGE_CONTROL_PLANE_URL`; on web-host this is the private loopback service.
|
||||
It stores only the Hugging Face URL, a review reason, and optional notes. Contact
|
||||
data is not collected. The old Supabase migrations remain as history but are not
|
||||
used by the live suggestion path. Public Bench cards continue to come from
|
||||
reviewed `results/*.json`.
|
||||
|
||||
## Deploying results
|
||||
|
||||
Score cards are committed JSON, so publishing a new measurement is a commit:
|
||||
|
||||
```bash
|
||||
kbench run brain --tasks ifeval --limit 100
|
||||
git add results/ && git commit -m "measure: qwen3.6-35b-a3b @ your-node"
|
||||
git push
|
||||
```
|
||||
|
||||
CI rebuilds and the board picks it up. There is no database sync step because
|
||||
there is no database behind the board.
|
||||
|
||||
## What is NOT wired yet
|
||||
|
||||
- **Automatic submission execution.** Intentionally not part of the product. The public form only records a suggestion for owner review; see `DECISIONS.md#d10`.
|
||||
- **Auth.** Schema and invite code exist; the signup/login UI does not.
|
||||
- **Suggestion workflow UI.** The owner console now lists the review inbox. Accept/reject controls and all run scheduling remain manual.
|
||||
Reference in New Issue
Block a user