Karti Tripathi
9.6 KiB
Decisions
Why this repo is shaped the way it is. The reasoning is not recoverable from the code, which is the only reason this file exists. Append, don't rewrite -- a superseded decision is more useful with its replacement attached than deleted.
D1 — Depend on inspect_ai, do not fork it
Decision. bench.karti.ai pip-installs inspect-ai and inspect-evals.
It is not a fork.
Why. The obvious move is to fork inspect_ai and grow from there. The numbers say otherwise: 407 MB, ~2 commits/day, pushed daily. Forking means either abandoning that stream of fixes or rebasing against it forever, and we would be paying that for nothing — we are not changing the execution engine, the scorer protocol, the sandbox, or the viewer. We are adding tasks, and tasks are the documented extension point.
The decisive evidence is upstream's own layout: inspect_evals is a
separate package that depends on inspect_ai, holding 200+ evals, none of
them in a fork of core. The maintainers already built the seam. This repo is
simply our private inspect_evals.
When to revisit. If we ever need to change core execution behaviour. Then fork from a known-good tag, for a stated reason. Not speculatively.
Rejected alternative — build our own harness. Reimplements sandboxed tool execution, retry/rate-limit handling, parallel rollouts, log format, viewer, and judge protocol, to arrive somewhere functionally identical. All of this bench's value is in the task data; none is in the runner.
D2 — A target is (model × host × quant × serving config × checkpoint)
Decision. The registry is keyed on the full tuple, not on model name.
See kbench/registry/__init__.py.
Why. On unified-memory boxes the serving config moves throughput more than
a model swap does. The live your-node server runs --enforce-eager (no CUDA
graphs) and --gpu-memory-utilization 0.55 (45% of unified memory unused,
capping KV cache and therefore the concurrency ceiling). A result recorded
against "Qwen3.6-35B-A3B" without those flags is not reproducible and not
comparable to anything.
checkpoint is in the key from day one because once we fine-tune, we will
compare karti-7b@step2000 against karti-7b@step4000 far more often than we
compare Qwen against Llama. Adding it later would invalidate every result
recorded before it.
Consequence. Every run snapshots the full target into its result file.
Referencing a registry id would go stale the first time models.yaml is
edited.
D3 — Three-tier data split, and the eval set stays private
Decision. data/public/ (a handful of shown examples), data/private/
(the real holdout), data/canary/ (GUID strings embedded in samples).
Why private — contamination, not secrets. The instinctive reason to keep eval data private is that samples derived from real tool calls might contain sensitive material. That is true but secondary, and it has its own fix (see below). The primary reason is that a public test set gets scraped into the next training corpus and stops measuring capability. That is how MMLU, GSM8K, and HumanEval died. Every benchmark with signal left in 2026 keeps a private holdout. If our samples ever go public, the bench's useful life is about one training cycle.
On secrets specifically. A private repo is not a secret-management strategy. Anything derived from real workloads is scrubbed at authoring time — no keys, no tokens, no tailscale addresses, no customer data — because that content would also leak through the model we send it to (often someone else's API) and through any generated share card. Repo privacy is defence in depth, not the control.
Canary tier. Samples carry embedded GUIDs. Later we can grep a candidate model for those strings and prove contamination rather than suspect it. Costs nothing now; impossible to add retroactively.
D4 — Per-sample outcomes in git; full outputs stay local
Decision. results/*.json stores each sample's score, pass/fail, and a
400-char excerpt. Full model outputs stay in the raw Inspect .eval log,
which is gitignored.
Why. An aggregate score tells you a checkpoint got worse. Per-sample diffs tell you which capability broke — the difference between a leaderboard and a debugging tool, and the whole reason this is usable for fine-tuning. Storing complete transcripts, though, would bloat the repo (thousands of samples × multi-KB reasoning traces × hundreds of runs) and would persist sensitive echoes of signal-tier prompts into git history forever. Git carries the shape of the failure; the local log carries the transcript.
D5 — Two task tiers: reference and signal
Decision. reference tasks come from inspect_evals unmodified.
signal tasks are ours and private. Both appear on the leaderboard, visually
distinguished, with reference scores captioned public, likely contaminated,
calibration only.
Why. A bench made only of novel private tasks is unfalsifiable to an outside reader — nobody, including future us, can tell whether a strange score means the model is bad or the harness is broken. The reference tier is the credibility anchor and the smoke test: if our IFEval lands far from published values, the harness is wrong.
Why IFEval first. Its scorer is deterministic, so it needs no judge model
— no per-run cost, and no judge drift between runs to confound comparisons.
It is fast enough for a tight debug loop. And instruction adherence is the
property that actually decides whether a small local model can hold a system
prompt in production, which is the live question about brain.
D6 — Self-contained perf load generator, not vllm bench serve
Decision. kbench/perf.py is an async httpx load generator against any
OpenAI-compatible endpoint.
Why. Submissions are the point of the public site, so we cannot assume vLLM — the harness must measure SGLang, llama.cpp, ollama, or whatever a submitted model ends up served under. It also needs nothing installed on the target box, and it lets us fix the metric definitions so numbers stay comparable across engines instead of inheriting each engine's conventions.
Two traps it exists to avoid. Both produce great-looking meaningless numbers:
- Prefix caching. vLLM caches shared prompt prefixes. Identical concurrent prompts make prefill nearly free after the first request and overstate throughput badly. Every request gets a unique leading nonce.
- Variable output length. If the model decides when to stop, concurrency
levels finish at different token counts and tok/s stops being comparable.
We send
ignore_eosso every request emits exactlyoutput_tokens.
D7 — Private gitea now; the harness may open-source later, the data never
Decision. karti/lumbridge-bench on gitea, private. Results public.
(Updated 2026-08-01: the repo was karti/bench.karti.ai, now archived. "Site public" meant the
standalone bench.karti.ai leaderboard, which is retired — results surface through
lumbridgecorp.com/bench instead. The public/private split below is unchanged.)
Why. The leaderboard is only valuable if people can see it; the eval data
is only valuable if they cannot (D3). If we later open-source, the harness
splits out and data/private/ stays behind — which is why the seam between
kbench/ (mechanism) and data/ (content) is kept clean from the start. A
clean seam costs nothing today and keeps the decision reversible.
D8 — Submissions execute untrusted code; sandbox before launch
Status. Planned, not yet implemented. Blocking for the public submit form.
Anonymous "drop a HuggingFace link and we'll benchmark it" means running
arbitrary code on our own hardware. HF repos can ship custom modeling files
that execute on load, and .bin checkpoints are pickles — remote code
execution by design. Non-negotiable gates:
safetensorsonly;trust_remote_code=false; hard-fail otherwise- eval runs in a container with no network egress and no tailscale route
- pre-flight via the HF API before queueing: reject if params exceed the target box, if config is unreadable, if the repo is gated
- one box, one queue: show estimated wait, cap submissions per IP per day
D9 — Anonymous submissions get aggregate scores only
Decision. Signed-in users (115karti invite) get per-sample breakdowns and private-by-default results. Anonymous submissions get an aggregate score, published.
Why. This reads like feature gating but it is primarily a contamination defence: detailed per-sample feedback is exactly how someone would reverse-engineer the private set (D3) by probing it one sample at a time.
D10 — Public input is a manual suggestion, never an execution queue
Decision. The public action is Suggest a model. It records a Hugging Face reference and optional context for owner review. It never downloads, imports, schedules, or executes a model. Karti manually inspects the model card, license, files, compatibility, footprint, and serving path, then explicitly decides whether to run it.
Why. The product does not need a public code-execution service. Manual approval is the intended workflow and keeps ownership of scarce hardware and private evaluation data clear. The Lumbridge control plane stores the inert review record with status suggested; only an owner can advance its lifecycle.
Supersedes D8 for product behavior. The sandbox described in D8 is no longer a launch dependency because no public suggestion is automatically executed. If automated third-party execution is ever proposed again, it requires a new explicit decision and the full D8 isolation boundary first.