Karti Tripathi
5 lines
2.4 KiB
JSON
5 lines
2.4 KiB
JSON
{"id": "example/0001", "family": "example", "split": "test", "canary": "KBENCH-CANARY-7d41af0c", "input": "Reply with only the numeric HTTP status code that means \"Too Many Requests\". No other text.", "target": "429", "scorer": "exact", "rubric": null, "metadata": {"source": "example", "authored": "2026-07-27", "tags": ["format-adherence"], "note": "Demonstrates the `exact` scorer. Also a decent instruction-following probe: verbose models append explanation and fail."}}
|
|
{"id": "example/0002", "family": "example", "split": "test", "canary": "KBENCH-CANARY-2b8e5513", "input": "A systemd unit fails to start with 'Address already in use'. Name one command-line tool that identifies which process is holding the port.", "target": "(lsof|ss|netstat|fuser)", "scorer": "regex", "rubric": null, "metadata": {"source": "example", "authored": "2026-07-27", "tags": ["ops"], "note": "Demonstrates `regex` -- the right choice when several answers are equally correct."}}
|
|
{"id": "example/0003", "family": "example", "split": "test", "canary": "KBENCH-CANARY-9c07de24", "input": "In one sentence: what does the vLLM flag --enforce-eager disable?", "target": "CUDA graph", "scorer": "includes", "rubric": null, "metadata": {"source": "example", "authored": "2026-07-27", "tags": ["inference"], "note": "Demonstrates `includes` -- checks one required concept without constraining phrasing."}}
|
|
{"id": "example/0004", "family": "example", "split": "test", "canary": "KBENCH-CANARY-5a1f6b8e", "input": "A benchmark harness sends the same 1024-token prompt from 32 concurrent clients to a vLLM server and reports throughput. Explain the measurement flaw and how to fix it.", "target": "Prefix caching makes prefill nearly free for all but the first request, so reported throughput is inflated. Fix: make each request's prompt unique, e.g. a random nonce in the leading tokens.", "scorer": "rubric", "rubric": "Grade CORRECT only if the answer identifies that a shared prompt prefix is cached (prefix caching / KV reuse) and that this inflates measured throughput, AND proposes making prompts unique per request. Mentioning only 'add randomness' without connecting it to prefix caching is INCORRECT. Ignore wording and length.", "metadata": {"source": "example", "authored": "2026-07-27", "tags": ["inference", "benchmarking"], "note": "Demonstrates `rubric` -- the property is genuinely not checkable mechanically. Requires a judge model."}}
|