Document the contract and the traps
CONTRACT.md states the twelve rules and, for each, the failure it prevents. AGENTS.md lists the three directories a demo agent may touch and the seven traps that have already cost time here — pnpm 11's settings move, the non-portable RNG, the two-pass rule, the two reward-denominator subtleties, the silent CSP worker block, and Caddy's silent bind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
This commit is contained in:
+104
@@ -0,0 +1,104 @@
|
||||
# The demo contract
|
||||
|
||||
Every demo is one directory under `src/demos/<slug>/` plus one Python package
|
||||
under `envs/`. The shell renders anything that satisfies the interfaces in
|
||||
`src/lib/demo-kit/types.ts`. Nothing else in the repository needs editing —
|
||||
the header, the gallery, the router and the sitemap are all generated from the
|
||||
registry, and the registry finds demos by existence.
|
||||
|
||||
That is the whole design goal: **demo number nine must not be able to break
|
||||
demo number one.**
|
||||
|
||||
## The rules, and why each exists
|
||||
|
||||
1. **The directory name is the slug.** Two sources of truth for a URL is one
|
||||
too many.
|
||||
|
||||
2. **`meta.ts` is eager, serialisable and React-free.** It is imported for
|
||||
every demo on every page load, because the header needs all of them to
|
||||
render. `icon` is a lucide icon *name*, not a component — importing the
|
||||
component would pull lucide into the entry chunk on behalf of a demo nobody
|
||||
opened.
|
||||
|
||||
3. **`demo.tsx` is lazy.** Everything expensive lives behind it.
|
||||
|
||||
4. **A demo imports from `@/lib/demo-kit` and nowhere deeper.** Reaching into
|
||||
`@/lib/demo-kit/player` couples a demo to an implementation detail; the
|
||||
barrel is the contract's surface.
|
||||
|
||||
5. **A demo never imports from `@/components/demo/`.** The shell renders demos;
|
||||
demos do not reach into the shell.
|
||||
|
||||
6. **The shell never mentions a slug.** `if (slug === 'wordle')` in
|
||||
`src/components/demo/` is a contract bug — either fix the contract or expose
|
||||
a `SlotRegion`. `check-demos` fails on it.
|
||||
|
||||
7. **Reward weights sum to 1.0, and at least one component is a
|
||||
`counterweight`.** A reward with only objectives teaches the crude version of
|
||||
what you asked for.
|
||||
|
||||
8. **A counterweight must be in genuine tension with the objective.** If every
|
||||
good policy also scores 1.0 on it, it is a `gate` — declare it as one. The
|
||||
probe ladder is where you prove the difference: two good policies, neither
|
||||
dominating.
|
||||
|
||||
9. **`null` is "not scored", never `0.0`.** A zero is a claim that the policy
|
||||
did badly. A null is an admission that we do not know. Rendering the second
|
||||
as the first is the quiet way a demo starts lying.
|
||||
|
||||
10. **Every `DemoStep` sets `announce`.** `prefers-reduced-motion` clamps every
|
||||
animation to nothing, so colour alone carries the result — and colour alone
|
||||
is not a result. The announcement *is* the feedback for anyone not looking
|
||||
at the screen.
|
||||
|
||||
11. **A `RunRef` with `kind: 'intervened'` must name its `intervention`.** This
|
||||
exists so a prompt change can never be presented as a training result by
|
||||
omission.
|
||||
|
||||
12. **A `spec` demo ships a real specification** — task, legal actions,
|
||||
deterministic grader, counterweight, and the eval command that would run
|
||||
it. A coming-soon card is not a specification, and `check-demos` refuses one.
|
||||
|
||||
## Adding a demo
|
||||
|
||||
```bash
|
||||
pnpm demo:new claims # copies src/demos/_template and envs/_template
|
||||
```
|
||||
|
||||
Then, in order:
|
||||
|
||||
1. **Write the environment first**, in `envs/<pkg>/`. The engine, the reward,
|
||||
and a probe ladder. If the reward cannot be probed, it cannot be trusted, and
|
||||
nothing downstream is worth building.
|
||||
2. **Port the scorer to TypeScript** and add it to the conformance gate. The
|
||||
browser must be able to re-derive what the environment recorded, or the
|
||||
verify badge is decoration.
|
||||
3. **Capture rollouts** with `envs/capture.py`. Include at least one run the
|
||||
agent loses; a demo where the agent always wins teaches nothing about the
|
||||
reward.
|
||||
4. **Write `meta.ts`, `narrative.ts`, `surface.tsx`, `adapter.ts`** and the
|
||||
demo module. The narrative's `limits` must name which demo answers each gap.
|
||||
5. `pnpm check && pnpm build`.
|
||||
|
||||
## The marker syntax for code receipts
|
||||
|
||||
`RewardSpec.source.marker` selects a region of a source file to highlight:
|
||||
|
||||
```python
|
||||
# region: pig-demo/reward
|
||||
...
|
||||
# endregion: pig-demo/reward
|
||||
```
|
||||
|
||||
Exclusive of the marker lines, **exactly one pair per file**. Zero or two or
|
||||
more is a fatal error in `check-receipts.mjs`, because a silently-wrong region
|
||||
would quote the wrong code under a claim that it is the code that ran.
|
||||
|
||||
## What to expect to change
|
||||
|
||||
The contract was written against a turn-based game. The first vertical demo is
|
||||
deliberately a one-shot classifier, so it stresses the weakest axis — no engine,
|
||||
no interactive driver, possibly a single run. **Budget one demo-kit refactor
|
||||
there and treat it as expected rather than as a failure.** If more than a slot
|
||||
is needed, widen the contract once, deliberately, updating `CONTRACT.md` and
|
||||
`_template` in the same commit.
|
||||
Reference in New Issue
Block a user