family: example

split: test

input: Reply with only the numeric HTTP status code that means "Too Many Requests". No other text.

scorer: exact

source: example

note: Demonstrates the `exact` scorer. Also a decent instruction-following probe: verbose models append explanation and fail.


family: example

split: test

input: A systemd unit fails to start with 'Address already in use'. Name one command-line tool that identifies which process is holding the port.

target: (lsof|ss|netstat|fuser)

scorer: regex

note: Demonstrates `regex` -- the right choice when several answers are equally correct.


family: example

split: test

input: In one sentence: what does the vLLM flag --enforce-eager disable?

target: CUDA graph

scorer: includes

note: Demonstrates `includes` -- checks one required concept without constraining phrasing.


family: example

split: test

input: A benchmark harness sends the same 1024-token prompt from 32 concurrent clients to a vLLM server and reports throughput. Explain the measurement flaw and how to fix it.

target: Prefix caching makes prefill nearly free for all but the first request, so reported throughput is inflated. Fix: make each request's prompt unique, e.g. a random nonce in the leading tokens.

scorer: rubric

rubric: Grade CORRECT only if the answer identifies that a shared prompt prefix is cached (prefix caching / KV reuse) and that this inflates measured throughput, AND proposes making prompts unique per request. Mentioning only 'add randomness' without connecting it to prefix caching is INCORRECT. Ignore wording and length.

note: Demonstrates `rubric` -- the property is genuinely not checkable mechanically. Requires a judge model.