Lumbridge Bench

data/README.md

@@ -0,0 +1,104 @@
+# Eval data
+
+Three tiers. The split is load-bearing — see `docs/DECISIONS.md#d3`.
+
+| dir | contents | visibility |
+|---|---|---|
+| `public/` | a few example samples | shown on the site, safe to leak |
+| `private/` | the real holdout | **never leaves this repo** |
+| `canary/` | contamination probes (carrier + detectors) | never leaves this repo |
+
+## Sample format
+
+One JSON object per line (JSONL), one file per family:
+`private/agent_ops.jsonl`, `private/repo_edit.jsonl`, ...
+
+```json
+{
+  "id": "agent_ops/0001",
+  "family": "agent_ops",
+  "split": "test",
+  "canary": "KBENCH-CANARY-3f9a1c2e",
+  "input": "The prompt sent to the model.",
+  "target": "The expected answer, or null for rubric-graded samples.",
+  "scorer": "includes",
+  "rubric": null,
+  "metadata": {
+    "source": "openclaw",
+    "authored": "2026-07-27",
+    "tags": ["tool-use", "multi-turn"]
+  }
+}
+```
+
+### Fields
+
+- **`id`** — stable and never reused. Per-sample regression tracking across
+  checkpoints keys on this; renumbering breaks every historical comparison.
+- **`split`** — `train` | `dev` | `test`. Assigned at authoring time and
+  **never changed**. A `test` sample that leaks into fine-tuning makes every
+  subsequent number a lie you have no way to detect.
+- **`canary`** — provenance only, stamped by `kbench add`. It is **not** the
+  contamination control: it never reaches a model, so nothing can reproduce it.
+  The real control is the `canary/` tier — see below.
+- **`scorer`** — how this sample is graded:
+
+  | scorer | grading | judge |
+  |---|---|---|
+  | `exact` | output matches target exactly | no |
+  | `includes` | output contains target | no |
+  | `regex` | output matches target as a regex | no |
+  | `rubric` | model-graded against `rubric` text | **yes** |
+
+  Prefer deterministic scorers. Every rubric sample adds judge cost per run
+  and judge drift between runs, which confounds the checkpoint comparisons
+  this bench exists to make. Reach for `rubric` only when the property is
+  genuinely not checkable mechanically.
+
+## Contamination probes (`canary/`)
+
+The leak vector is named in the authoring rules below: prompts reach third-party
+judge and baseline models no matter how private this repo is. If a provider
+trains on inference traffic, our eval set is in their next model and every
+signal score becomes a measurement of memorisation.
+
+The probe tier catches that, and it only works as a pair:
+
+- **One carrier.** Its prompt contains the canary GUID in full. It always
+  passes, and passing is not the point — its job is to put the GUID into the
+  traffic so a training pipeline can pick it up.
+- **Two or more detectors.** Their prompts withhold the GUID and ask for it
+  cold. A model that has never seen our traffic cannot answer. One that has,
+  can.
+
+**Scoring is inverted here.** `canary_score` of 0 is the healthy result.
+Anything above 0 means signal scores for that target are void and the eval set
+needs regenerating. Never read it as a capability number.
+
+A detector for a GUID that no carrier ever transmitted is unanswerable by
+construction: it can never fire, so it proves nothing while looking exactly
+like a working control.
+
+## Authoring rules
+
+**Scrub before you commit.** No API keys, tokens, tailscale hostnames or IPs,
+customer data, or personal information. This content gets sent to third-party
+judge and baseline models regardless of how private this repo is. Repo
+privacy is defence in depth, not the control.
+
+**Derive from real work.** The value of this bench is that its samples come
+from things we actually asked a model to do. An invented task measures an
+invented capability.
+
+**Write the target before you see a model's answer.** Grading against an
+answer you just read is how a bench quietly turns into a rubber stamp.
+
+## Adding samples
+
+```bash
+kbench add agent_ops --input-file prompt.txt --target "expected" --scorer includes
+kbench add agent_ops --interactive
+```
+
+This assigns the next id, stamps a canary, defaults the split, and validates
+the record — do not hand-edit the JSONL.

data/public/example.jsonl

@@ -0,0 +1,4 @@
+{"id": "example/0001", "family": "example", "split": "test", "canary": "KBENCH-CANARY-7d41af0c", "input": "Reply with only the numeric HTTP status code that means \"Too Many Requests\". No other text.", "target": "429", "scorer": "exact", "rubric": null, "metadata": {"source": "example", "authored": "2026-07-27", "tags": ["format-adherence"], "note": "Demonstrates the `exact` scorer. Also a decent instruction-following probe: verbose models append explanation and fail."}}
+{"id": "example/0002", "family": "example", "split": "test", "canary": "KBENCH-CANARY-2b8e5513", "input": "A systemd unit fails to start with 'Address already in use'. Name one command-line tool that identifies which process is holding the port.", "target": "(lsof|ss|netstat|fuser)", "scorer": "regex", "rubric": null, "metadata": {"source": "example", "authored": "2026-07-27", "tags": ["ops"], "note": "Demonstrates `regex` -- the right choice when several answers are equally correct."}}
+{"id": "example/0003", "family": "example", "split": "test", "canary": "KBENCH-CANARY-9c07de24", "input": "In one sentence: what does the vLLM flag --enforce-eager disable?", "target": "CUDA graph", "scorer": "includes", "rubric": null, "metadata": {"source": "example", "authored": "2026-07-27", "tags": ["inference"], "note": "Demonstrates `includes` -- checks one required concept without constraining phrasing."}}
+{"id": "example/0004", "family": "example", "split": "test", "canary": "KBENCH-CANARY-5a1f6b8e", "input": "A benchmark harness sends the same 1024-token prompt from 32 concurrent clients to a vLLM server and reports throughput. Explain the measurement flaw and how to fix it.", "target": "Prefix caching makes prefill nearly free for all but the first request, so reported throughput is inflated. Fix: make each request's prompt unique, e.g. a random nonce in the leading tokens.", "scorer": "rubric", "rubric": "Grade CORRECT only if the answer identifies that a shared prompt prefix is cached (prefix caching / KV reuse) and that this inflates measured throughput, AND proposes making prompts unique per request. Mentioning only 'add randomness' without connecting it to prefix caching is INCORRECT. Ignore wording and length.", "metadata": {"source": "example", "authored": "2026-07-27", "tags": ["inference", "benchmarking"], "note": "Demonstrates `rubric` -- the property is genuinely not checkable mechanically. Requires a judge model."}}