Files

T

karti-ai 2c2dcad9fd Document the contract and the traps

CONTRACT.md states the twelve rules and, for each, the failure it prevents.
AGENTS.md lists the three directories a demo agent may touch and the seven traps
that have already cost time here — pnpm 11's settings move, the non-portable
RNG, the two-pass rule, the two reward-denominator subtleties, the silent CSP
worker block, and Caddy's silent bind.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB

2026-08-28 15:48:20 -07:00

4.4 KiB

Raw Blame History

The demo contract

Every demo is one directory under src/demos/<slug>/ plus one Python package under envs/. The shell renders anything that satisfies the interfaces in src/lib/demo-kit/types.ts. Nothing else in the repository needs editing — the header, the gallery, the router and the sitemap are all generated from the registry, and the registry finds demos by existence.

That is the whole design goal: demo number nine must not be able to break demo number one.

The rules, and why each exists

The directory name is the slug. Two sources of truth for a URL is one too many.
meta.ts is eager, serialisable and React-free. It is imported for every demo on every page load, because the header needs all of them to render. icon is a lucide icon name, not a component — importing the component would pull lucide into the entry chunk on behalf of a demo nobody opened.
demo.tsx is lazy. Everything expensive lives behind it.
A demo imports from @/lib/demo-kit and nowhere deeper. Reaching into @/lib/demo-kit/player couples a demo to an implementation detail; the barrel is the contract's surface.
A demo never imports from @/components/demo/. The shell renders demos; demos do not reach into the shell.
The shell never mentions a slug. if (slug === 'wordle') in src/components/demo/ is a contract bug — either fix the contract or expose a SlotRegion. check-demos fails on it.
Reward weights sum to 1.0, and at least one component is a counterweight. A reward with only objectives teaches the crude version of what you asked for.
A counterweight must be in genuine tension with the objective. If every good policy also scores 1.0 on it, it is a gate — declare it as one. The probe ladder is where you prove the difference: two good policies, neither dominating.
null is "not scored", never 0.0. A zero is a claim that the policy did badly. A null is an admission that we do not know. Rendering the second as the first is the quiet way a demo starts lying.
Every DemoStep sets announce. prefers-reduced-motion clamps every animation to nothing, so colour alone carries the result — and colour alone is not a result. The announcement is the feedback for anyone not looking at the screen.
A RunRef with kind: 'intervened' must name its intervention. This exists so a prompt change can never be presented as a training result by omission.
A spec demo ships a real specification — task, legal actions, deterministic grader, counterweight, and the eval command that would run it. A coming-soon card is not a specification, and check-demos refuses one.

Adding a demo

pnpm demo:new claims          # copies src/demos/_template and envs/_template

Then, in order:

Write the environment first, in envs/<pkg>/. The engine, the reward, and a probe ladder. If the reward cannot be probed, it cannot be trusted, and nothing downstream is worth building.
Port the scorer to TypeScript and add it to the conformance gate. The browser must be able to re-derive what the environment recorded, or the verify badge is decoration.
Capture rollouts with envs/capture.py. Include at least one run the agent loses; a demo where the agent always wins teaches nothing about the reward.
Write meta.ts, narrative.ts, surface.tsx, adapter.ts and the demo module. The narrative's limits must name which demo answers each gap.
pnpm check && pnpm build.

The marker syntax for code receipts

RewardSpec.source.marker selects a region of a source file to highlight:

# region: pig-demo/reward
...
# endregion: pig-demo/reward

Exclusive of the marker lines, exactly one pair per file. Zero or two or more is a fatal error in check-receipts.mjs, because a silently-wrong region would quote the wrong code under a claim that it is the code that ran.

What to expect to change

The contract was written against a turn-based game. The first vertical demo is deliberately a one-shot classifier, so it stresses the weakest axis — no engine, no interactive driver, possibly a single run. Budget one demo-kit refactor there and treat it as expected rather than as a failure. If more than a slot is needed, widen the contract once, deliberately, updating CONTRACT.md and _template in the same commit.