bench/docs/DECISIONS.md

# Decisions

Why this repo is shaped the way it is. The reasoning is not recoverable from
the code, which is the only reason this file exists. Append, don't rewrite --
a superseded decision is more useful with its replacement attached than
deleted.

---

## D1 — Depend on `inspect_ai`, do not fork it

**Decision.** `bench.karti.ai` pip-installs `inspect-ai` and `inspect-evals`.
It is not a fork.

**Why.** The obvious move is to fork [inspect_ai][1] and grow from there. The
numbers say otherwise: 407 MB, ~2 commits/day, pushed daily. Forking means
either abandoning that stream of fixes or rebasing against it forever, and we
would be paying that for nothing — we are not changing the execution engine,
the scorer protocol, the sandbox, or the viewer. We are adding tasks, and
tasks are the documented extension point.

The decisive evidence is upstream's own layout: `inspect_evals` is a
**separate package that depends on `inspect_ai`**, holding 200+ evals, none of
them in a fork of core. The maintainers already built the seam. This repo is
simply our private `inspect_evals`.

**When to revisit.** If we ever need to change core execution behaviour. Then
fork from a known-good tag, for a stated reason. Not speculatively.

**Rejected alternative — build our own harness.** Reimplements sandboxed tool
execution, retry/rate-limit handling, parallel rollouts, log format, viewer,
and judge protocol, to arrive somewhere functionally identical. All of this
bench's value is in the task data; none is in the runner.

[1]: https://github.com/UKGovernmentBEIS/inspect_ai

---

## D2 — A target is `(model × host × quant × serving config × checkpoint)`

**Decision.** The registry is keyed on the full tuple, not on model name.
See `kbench/registry/__init__.py`.

**Why.** On unified-memory boxes the serving config moves throughput more than
a model swap does. The live your-node server runs `--enforce-eager` (no CUDA
graphs) and `--gpu-memory-utilization 0.55` (45% of unified memory unused,
capping KV cache and therefore the concurrency ceiling). A result recorded
against "Qwen3.6-35B-A3B" without those flags is not reproducible and not
comparable to anything.

`checkpoint` is in the key from day one because once we fine-tune, we will
compare `karti-7b@step2000` against `karti-7b@step4000` far more often than we
compare Qwen against Llama. Adding it later would invalidate every result
recorded before it.

**Consequence.** Every run snapshots the full target into its result file.
Referencing a registry id would go stale the first time `models.yaml` is
edited.

---

## D3 — Three-tier data split, and the eval set stays private

**Decision.** `data/public/` (a handful of shown examples), `data/private/`
(the real holdout), `data/canary/` (GUID strings embedded in samples).

**Why private — contamination, not secrets.** The instinctive reason to keep
eval data private is that samples derived from real tool calls might contain
sensitive material. That is true but secondary, and it has its own fix (see
below). The primary reason is that **a public test set gets scraped into the
next training corpus and stops measuring capability**. That is how MMLU,
GSM8K, and HumanEval died. Every benchmark with signal left in 2026 keeps a
private holdout. If our samples ever go public, the bench's useful life is
about one training cycle.

**On secrets specifically.** A private repo is not a secret-management
strategy. Anything derived from real workloads is scrubbed **at authoring
time** — no keys, no tokens, no tailscale addresses, no customer data —
because that content would also leak through the model we send it to (often
someone else's API) and through any generated share card. Repo privacy is
defence in depth, not the control.

**Canary tier.** Samples carry embedded GUIDs. Later we can grep a candidate
model for those strings and *prove* contamination rather than suspect it.
Costs nothing now; impossible to add retroactively.

---

## D4 — Per-sample outcomes in git; full outputs stay local

**Decision.** `results/*.json` stores each sample's score, pass/fail, and a
400-char excerpt. Full model outputs stay in the raw Inspect `.eval` log,
which is gitignored.

**Why.** An aggregate score tells you a checkpoint got worse. Per-sample
diffs tell you *which capability broke* — the difference between a leaderboard
and a debugging tool, and the whole reason this is usable for fine-tuning.
Storing complete transcripts, though, would bloat the repo (thousands of
samples × multi-KB reasoning traces × hundreds of runs) and would persist
sensitive echoes of signal-tier prompts into git history forever. Git carries
the shape of the failure; the local log carries the transcript.

---

## D5 — Two task tiers: reference and signal

**Decision.** `reference` tasks come from `inspect_evals` unmodified.
`signal` tasks are ours and private. Both appear on the leaderboard, visually
distinguished, with reference scores captioned *public, likely contaminated,
calibration only*.

**Why.** A bench made only of novel private tasks is unfalsifiable to an
outside reader — nobody, including future us, can tell whether a strange score
means the model is bad or the harness is broken. The reference tier is the
credibility anchor and the smoke test: if our IFEval lands far from published
values, the harness is wrong.

**Why IFEval first.** Its scorer is deterministic, so it needs no judge model
— no per-run cost, and no judge drift between runs to confound comparisons.
It is fast enough for a tight debug loop. And instruction adherence is the
property that actually decides whether a small local model can hold a system
prompt in production, which is the live question about `brain`.

---

## D6 — Self-contained perf load generator, not `vllm bench serve`

**Decision.** `kbench/perf.py` is an async httpx load generator against any
OpenAI-compatible endpoint.

**Why.** Submissions are the point of the public site, so we cannot assume
vLLM — the harness must measure SGLang, llama.cpp, ollama, or whatever a
submitted model ends up served under. It also needs nothing installed on the
target box, and it lets us fix the metric definitions so numbers stay
comparable across engines instead of inheriting each engine's conventions.

**Two traps it exists to avoid.** Both produce great-looking meaningless
numbers:

1. **Prefix caching.** vLLM caches shared prompt prefixes. Identical
   concurrent prompts make prefill nearly free after the first request and
   overstate throughput badly. Every request gets a unique leading nonce.
2. **Variable output length.** If the model decides when to stop, concurrency
   levels finish at different token counts and tok/s stops being comparable.
   We send `ignore_eos` so every request emits exactly `output_tokens`.

---

## D7 — Private gitea now; the harness may open-source later, the data never

**Decision.** `karti/lumbridge-bench` on gitea, private. Results public.

_(Updated 2026-08-01: the repo was `karti/bench.karti.ai`, now archived. "Site public" meant the
standalone bench.karti.ai leaderboard, which is retired — results surface through
`lumbridgecorp.com/bench` instead. The public/private split below is unchanged.)_

**Why.** The leaderboard is only valuable if people can see it; the eval data
is only valuable if they cannot (D3). If we later open-source, the harness
splits out and `data/private/` stays behind — which is why the seam between
`kbench/` (mechanism) and `data/` (content) is kept clean from the start. A
clean seam costs nothing today and keeps the decision reversible.

---

## D8 — Submissions execute untrusted code; sandbox before launch

**Status.** Planned, not yet implemented. Blocking for the public submit form.

Anonymous "drop a HuggingFace link and we'll benchmark it" means running
arbitrary code on our own hardware. HF repos can ship custom modeling files
that execute on load, and `.bin` checkpoints are pickles — remote code
execution by design. Non-negotiable gates:

- `safetensors` only; `trust_remote_code=false`; hard-fail otherwise
- eval runs in a container with **no network egress** and no tailscale route
- pre-flight via the HF API before queueing: reject if params exceed the
  target box, if config is unreadable, if the repo is gated
- one box, one queue: show estimated wait, cap submissions per IP per day

---

## D9 — Anonymous submissions get aggregate scores only

**Decision.** Signed-in users (115karti invite) get per-sample breakdowns and
private-by-default results. Anonymous submissions get an aggregate score,
published.

**Why.** This reads like feature gating but it is primarily a contamination
defence: detailed per-sample feedback is exactly how someone would
reverse-engineer the private set (D3) by probing it one sample at a time.

---

## D10 — Public input is a manual suggestion, never an execution queue

**Decision.** The public action is **Suggest a model**. It records a Hugging Face reference and optional context for owner review. It never downloads, imports, schedules, or executes a model. Karti manually inspects the model card, license, files, compatibility, footprint, and serving path, then explicitly decides whether to run it.

**Why.** The product does not need a public code-execution service. Manual approval is the intended workflow and keeps ownership of scarce hardware and private evaluation data clear. The Lumbridge control plane stores the inert review record with status `suggested`; only an owner can advance its lifecycle.

**Supersedes D8 for product behavior.** The sandbox described in D8 is no longer a launch dependency because no public suggestion is automatically executed. If automated third-party execution is ever proposed again, it requires a new explicit decision and the full D8 isolation boundary first.