Capture harness, fixture verification, CI, and the public README
The site does no live inference. Rollouts are captured once against spark-1 and replayed at their recorded wall-clock — a public demo with no auth cannot hold an API key, and a recorded run can be scrubbed, permalinked, blind-compared and verified in ways a live one cannot. What stops it being a video is that the browser re-derives every number from the recorded moves. verify_fixtures.py is the Python half of that: it replays every committed fixture through the engine and reproduces its own rewards. All 16 land at delta 0.0. A fixture that cannot be regenerated is a claim with no receipt. First real measurement, thinking off, 8 seeds: solved 0/8. The model repeats guesses it has already played, invents words (trape, slith, postt, boomy), and contradicts its own feedback — consistency 0.09 to 0.17. That is the published failure taxonomy showing up in our own data on the first run, and it is why `consistency` is a reward component rather than a footnote. A capture failure is recorded as a turn with a null reply, never dropped. A capture that silently discarded failed turns would be reporting a better model than the one that ran. CI gates both halves and four things that fail silently in production: the word lists must rebuild byte-identically, the prerendered routes must carry their own baked og tags (crawlers do not run JS, so without them every shared link previews as the homepage), no blob: URL may reach the bundle (the site's CSP has no worker-src, so it falls back to default-src 'self' and a blob worker is blocked with no error), and the conformance digest must match across languages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
This commit is contained in:
karti-ai
@@ -0,0 +1,196 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* The receipt checker.
|
||||
*
|
||||
* Every demo quotes the code that computes its reward, verbatim, next to the
|
||||
* number that code produced. That quote is the page's whole claim to being
|
||||
* evidence rather than marketing, so the thing it quotes has to exist and the
|
||||
* region it quotes has to resolve. A receipt panel that silently renders empty,
|
||||
* or renders the wrong forty lines because someone inserted a function above
|
||||
* the marker, is worse than no receipt at all.
|
||||
*
|
||||
* ── MARKER SYNTAX ──────────────────────────────────────────────────────────
|
||||
*
|
||||
* `RewardSpec.source` is `{ path, code, marker? }`. Without a `marker` the
|
||||
* whole file is the receipt. With one, the receipt is the region delimited by
|
||||
* a matched pair of comment lines in the source file itself:
|
||||
*
|
||||
* # region: pig-demo/<marker>
|
||||
* ... the quoted lines ...
|
||||
* # endregion: pig-demo/<marker>
|
||||
*
|
||||
* Rules, all enforced here:
|
||||
*
|
||||
* · The marker lines are EXCLUSIVE — neither appears in the quoted region.
|
||||
* · EXACTLY ONE pair per file per marker. Zero is a broken receipt; two or
|
||||
* more is ambiguous, and an ambiguous receipt silently quotes whichever
|
||||
* region the reader's implementation happened to find first. Both are
|
||||
* fatal.
|
||||
* · `region` must come before `endregion`, and the region must be non-empty
|
||||
* once the marker lines are removed.
|
||||
* · The comment token is `#` (Python) or `//` (TypeScript), then optional
|
||||
* whitespace. Everything else on the line is ignored, so
|
||||
* `# region: pig-demo/consistency (see PR #41)` is legal.
|
||||
* · The `pig-demo/` prefix is required. It is what makes these greppable and
|
||||
* stops an editor's own `#region` folding markers from being read as
|
||||
* receipts.
|
||||
*
|
||||
* Marker names are `[a-z0-9][a-z0-9-]*`: they end up in a grep, a CI message
|
||||
* and a code comment, and mixed case in all three is a bug factory.
|
||||
*/
|
||||
|
||||
import path from 'node:path';
|
||||
|
||||
import {
|
||||
Report,
|
||||
abs,
|
||||
allDemoDirs,
|
||||
demoFiles,
|
||||
evalLiteral,
|
||||
exists,
|
||||
isUnresolved,
|
||||
literalsAfter,
|
||||
read,
|
||||
rel,
|
||||
} from './_lib.mjs';
|
||||
|
||||
const report = new Report('check-receipts');
|
||||
|
||||
const RULE_PATH = 'reward.source.path';
|
||||
const RULE_MARKER = 'reward.source.marker';
|
||||
|
||||
const MARKER_NAME = /^[a-z0-9][a-z0-9-]*$/;
|
||||
const markerLine = (kind, marker) =>
|
||||
new RegExp(`^[ \\t]*(?:#|//)[ \\t]*${kind}:[ \\t]*pig-demo/${marker.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}(?![\\w-])`, 'm');
|
||||
|
||||
/** Every `source: { ... }` literal in a demo, wherever the demo chose to put it. */
|
||||
function receipts(slug) {
|
||||
const out = [];
|
||||
for (const file of demoFiles(slug)) {
|
||||
const src = read(file);
|
||||
for (const literal of literalsAfter(src, /\bsource\s*:\s*/, '{')) {
|
||||
// `code` is legitimately an identifier here — the Python arrives through
|
||||
// a Vite `?raw` import, which cannot resolve in plain Node — so the
|
||||
// literal is read leniently and only `path` and `marker` are trusted.
|
||||
const evaluated = evalLiteral(literal.text, 'RewardSpec.source', { lenient: true });
|
||||
if (!evaluated.ok) {
|
||||
out.push({ file, error: evaluated.error });
|
||||
continue;
|
||||
}
|
||||
const value = evaluated.value;
|
||||
if (!value || typeof value !== 'object' || !('path' in value)) continue; // some other `source:` key
|
||||
const line = src.slice(0, literal.start).split('\n').length;
|
||||
out.push({ file, line, value });
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
const slugs = allDemoDirs().filter((s) => !s.startsWith('_'));
|
||||
if (slugs.length === 0) {
|
||||
console.log('check-receipts: no demos to check.');
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
for (const slug of slugs) {
|
||||
const found = receipts(slug);
|
||||
|
||||
if (found.length === 0) {
|
||||
report.fail(
|
||||
rel(path.join(abs('src', 'demos'), slug)),
|
||||
RULE_PATH,
|
||||
'no `reward.source` found. Every demo quotes the code that computes its reward; without a source ' +
|
||||
'the receipt panel has nothing to show and the page is asserting its numbers rather than proving them.',
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
for (const receipt of found) {
|
||||
const where = `${rel(receipt.file)}${receipt.line ? `:${receipt.line}` : ''}`;
|
||||
if (receipt.error) {
|
||||
report.fail(where, RULE_PATH, receipt.error);
|
||||
continue;
|
||||
}
|
||||
const { path: sourcePath, marker } = receipt.value;
|
||||
|
||||
if (typeof sourcePath !== 'string' || sourcePath.trim() === '') {
|
||||
report.fail(
|
||||
where,
|
||||
RULE_PATH,
|
||||
`\`path\` is ${isUnresolved(sourcePath) ? 'an imported constant' : JSON.stringify(sourcePath)}. ` +
|
||||
'It must be a literal repo-relative path, e.g. "envs/wordle_five/wordle_five/rubric.py", ' +
|
||||
'so a reader can open the same file this page quotes.',
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
// Repo-relative by contract. A leading slash would mean the site root,
|
||||
// which is not where the Python lives, and an absolute disk path would
|
||||
// only work on the machine that wrote it.
|
||||
const clean = sourcePath.replace(/^\.\//, '');
|
||||
if (path.isAbsolute(clean)) {
|
||||
report.fail(where, RULE_PATH, `\`path\` is absolute ("${sourcePath}"). Use a repo-relative path.`);
|
||||
continue;
|
||||
}
|
||||
const onDisk = abs(clean);
|
||||
if (!report.check(exists(onDisk), where, RULE_PATH, `\`path\` is "${sourcePath}", and ${rel(onDisk)} does not exist.`)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (marker === undefined || marker === null) continue; // whole file is the receipt
|
||||
|
||||
if (typeof marker !== 'string' || !MARKER_NAME.test(marker)) {
|
||||
report.fail(
|
||||
where,
|
||||
RULE_MARKER,
|
||||
`\`marker\` is ${JSON.stringify(marker)}. Markers are lower-kebab-case (${MARKER_NAME}) because they ` +
|
||||
'appear in a grep, a CI message and a code comment, and mixed case in all three is a bug factory.',
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
const body = read(onDisk);
|
||||
const lines = body.split('\n');
|
||||
const opens = [];
|
||||
const closes = [];
|
||||
lines.forEach((line, i) => {
|
||||
if (markerLine('region', marker).test(line)) opens.push(i);
|
||||
if (markerLine('endregion', marker).test(line)) closes.push(i);
|
||||
});
|
||||
|
||||
const syntax = `Expected exactly one "# region: pig-demo/${marker}" and one "# endregion: pig-demo/${marker}".`;
|
||||
if (opens.length !== 1 || closes.length !== 1) {
|
||||
report.fail(
|
||||
rel(onDisk),
|
||||
RULE_MARKER,
|
||||
`marker "${marker}" (referenced from ${where}) resolves to ${opens.length} region marker(s) and ` +
|
||||
`${closes.length} endregion marker(s). ${syntax} ` +
|
||||
(opens.length > 1 || closes.length > 1
|
||||
? 'More than one pair is ambiguous: the receipt would quote whichever region the reader found first.'
|
||||
: 'A missing marker renders the receipt panel empty, which reads as the code not existing.'),
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
const [open] = opens;
|
||||
const [close] = closes;
|
||||
if (close <= open) {
|
||||
report.fail(
|
||||
rel(onDisk),
|
||||
RULE_MARKER,
|
||||
`marker "${marker}": the endregion is on line ${close + 1}, at or before the region on line ${open + 1}.`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
const region = lines.slice(open + 1, close);
|
||||
report.check(
|
||||
region.some((line) => line.trim() !== ''),
|
||||
rel(onDisk),
|
||||
RULE_MARKER,
|
||||
`marker "${marker}" delimits an empty region (lines ${open + 2}-${close}). The markers are exclusive, ` +
|
||||
'so a pair on adjacent lines quotes nothing.',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
report.finish();
|
||||
Reference in New Issue
Block a user