karti-ai
2c2dcad9fd
CONTRACT.md states the twelve rules and, for each, the failure it prevents. AGENTS.md lists the three directories a demo agent may touch and the seven traps that have already cost time here — pnpm 11's settings move, the non-portable RNG, the two-pass rule, the two reward-denominator subtleties, the silent CSP worker block, and Caddy's silent bind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
105 lines
4.4 KiB
Markdown
105 lines
4.4 KiB
Markdown
# The demo contract
|
|
|
|
Every demo is one directory under `src/demos/<slug>/` plus one Python package
|
|
under `envs/`. The shell renders anything that satisfies the interfaces in
|
|
`src/lib/demo-kit/types.ts`. Nothing else in the repository needs editing —
|
|
the header, the gallery, the router and the sitemap are all generated from the
|
|
registry, and the registry finds demos by existence.
|
|
|
|
That is the whole design goal: **demo number nine must not be able to break
|
|
demo number one.**
|
|
|
|
## The rules, and why each exists
|
|
|
|
1. **The directory name is the slug.** Two sources of truth for a URL is one
|
|
too many.
|
|
|
|
2. **`meta.ts` is eager, serialisable and React-free.** It is imported for
|
|
every demo on every page load, because the header needs all of them to
|
|
render. `icon` is a lucide icon *name*, not a component — importing the
|
|
component would pull lucide into the entry chunk on behalf of a demo nobody
|
|
opened.
|
|
|
|
3. **`demo.tsx` is lazy.** Everything expensive lives behind it.
|
|
|
|
4. **A demo imports from `@/lib/demo-kit` and nowhere deeper.** Reaching into
|
|
`@/lib/demo-kit/player` couples a demo to an implementation detail; the
|
|
barrel is the contract's surface.
|
|
|
|
5. **A demo never imports from `@/components/demo/`.** The shell renders demos;
|
|
demos do not reach into the shell.
|
|
|
|
6. **The shell never mentions a slug.** `if (slug === 'wordle')` in
|
|
`src/components/demo/` is a contract bug — either fix the contract or expose
|
|
a `SlotRegion`. `check-demos` fails on it.
|
|
|
|
7. **Reward weights sum to 1.0, and at least one component is a
|
|
`counterweight`.** A reward with only objectives teaches the crude version of
|
|
what you asked for.
|
|
|
|
8. **A counterweight must be in genuine tension with the objective.** If every
|
|
good policy also scores 1.0 on it, it is a `gate` — declare it as one. The
|
|
probe ladder is where you prove the difference: two good policies, neither
|
|
dominating.
|
|
|
|
9. **`null` is "not scored", never `0.0`.** A zero is a claim that the policy
|
|
did badly. A null is an admission that we do not know. Rendering the second
|
|
as the first is the quiet way a demo starts lying.
|
|
|
|
10. **Every `DemoStep` sets `announce`.** `prefers-reduced-motion` clamps every
|
|
animation to nothing, so colour alone carries the result — and colour alone
|
|
is not a result. The announcement *is* the feedback for anyone not looking
|
|
at the screen.
|
|
|
|
11. **A `RunRef` with `kind: 'intervened'` must name its `intervention`.** This
|
|
exists so a prompt change can never be presented as a training result by
|
|
omission.
|
|
|
|
12. **A `spec` demo ships a real specification** — task, legal actions,
|
|
deterministic grader, counterweight, and the eval command that would run
|
|
it. A coming-soon card is not a specification, and `check-demos` refuses one.
|
|
|
|
## Adding a demo
|
|
|
|
```bash
|
|
pnpm demo:new claims # copies src/demos/_template and envs/_template
|
|
```
|
|
|
|
Then, in order:
|
|
|
|
1. **Write the environment first**, in `envs/<pkg>/`. The engine, the reward,
|
|
and a probe ladder. If the reward cannot be probed, it cannot be trusted, and
|
|
nothing downstream is worth building.
|
|
2. **Port the scorer to TypeScript** and add it to the conformance gate. The
|
|
browser must be able to re-derive what the environment recorded, or the
|
|
verify badge is decoration.
|
|
3. **Capture rollouts** with `envs/capture.py`. Include at least one run the
|
|
agent loses; a demo where the agent always wins teaches nothing about the
|
|
reward.
|
|
4. **Write `meta.ts`, `narrative.ts`, `surface.tsx`, `adapter.ts`** and the
|
|
demo module. The narrative's `limits` must name which demo answers each gap.
|
|
5. `pnpm check && pnpm build`.
|
|
|
|
## The marker syntax for code receipts
|
|
|
|
`RewardSpec.source.marker` selects a region of a source file to highlight:
|
|
|
|
```python
|
|
# region: pig-demo/reward
|
|
...
|
|
# endregion: pig-demo/reward
|
|
```
|
|
|
|
Exclusive of the marker lines, **exactly one pair per file**. Zero or two or
|
|
more is a fatal error in `check-receipts.mjs`, because a silently-wrong region
|
|
would quote the wrong code under a claim that it is the code that ran.
|
|
|
|
## What to expect to change
|
|
|
|
The contract was written against a turn-based game. The first vertical demo is
|
|
deliberately a one-shot classifier, so it stresses the weakest axis — no engine,
|
|
no interactive driver, possibly a single run. **Budget one demo-kit refactor
|
|
there and treat it as expected rather than as a failure.** If more than a slot
|
|
is needed, widen the contract once, deliberately, updating `CONTRACT.md` and
|
|
`_template` in the same commit.
|