karti
516685526c
The ledger answers which contracted capacity is sold, to whom, at what margin. It says nothing about the motion — the repeatable practice that turns a customer conversation into a scoped deployment, and turns that deployment into something the next one reuses. Motion is deliberately not a parallel entity tree. DEMAND_STAGES already is the motion, so Motion binds reusable artefacts to the stages of a demand deal that already exists: an engagement hangs off one deal, cascade deleted, one per deal by unique constraint. Nine closed kinds, each declaring which stages it serves, and a starter library of twelve templates covering all eight open stages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
224 lines
28 KiB
JSON
224 lines
28 KiB
JSON
{
|
||
"kind": "playbook",
|
||
"slug": "production-readiness",
|
||
"title": "Production Readiness Checklist",
|
||
"summary": "The checkable gate between a POC that met its threshold and a model carrying production load, covering the inference path, the traffic ramp, fallbacks, the live eval and its own baseline, retraining triggers, paging and the handover that decides whether the customer really owns their intelligence.",
|
||
"stage": "deployment",
|
||
"body": "*Playbook stage: deployment. Follows **Post-Training POC Structure**; precedes **Case Study Frame**.*\n\n# Production readiness checklist\n\nThe POC cleared its threshold — a claim about a sealed held-out set, judged offline, nobody's customer waiting. Production is a different claim: that the model holds up under real arrival patterns, real failure, real latency budgets and real people who route around it the first week it embarrasses them. Demos die in this gap quietly — nobody cancels, traffic never moves past the pilot slice, and renewal arrives with nothing to point at. All of it assumes a production owner exists on the customer side; if nobody does, that is a sponsorship problem, not a deployment one.\n\n## The inference path\n\nStart from the shape chosen in **Reference Architectures**. What is served is not a hosted endpoint — it is adapted open weights the customer owns, inside the boundary they chose, with the verifier they own available at serve time. Those are checks an API vendor never has to pass, and they stop go-lives.\n\nDraw the path in one diagram, naming every hop, which artefact version is loaded from where, and whether the verifier scores inline or out of band. Then confirm the serving location against the `egress-boundary` and `inference-location` decisions in the **Data and Security Brief**, with the control owner named there. A model served just outside the boundary legal agreed is the commonest reason a finished deployment never opens.\n\nGet the latency budget as a number with a name attached: p95 and p99 in milliseconds, and who owns a breach. Say this early: a model that wins on quality and loses on p99 does not ship. The median is a marketing number; users experience the tail. A model whose p50 is {{300}} ms and whose p99 is {{6}} seconds is, to the calling service, an outage on a schedule.\n\nLoad-test at the real shape: peak concurrency ({{4}}x the mean is a placeholder for the measured peak), held until queueing effects appear, with batch size, streaming, 99th-percentile context length, and cold starts if the stack scales to zero. Cold starts and long-tail inputs cause most p99 breaches and are invisible in a POC.\n\n## The traffic ramp\n\nGo-live is not a switch. Record the ramp here as percentages against dates: the **Strategic Deployment Playbook** exits this stage on the traffic percentage recorded on this checklist being served, so that field must exist for the exit test to resolve.\n\nShadow mode first: {{14}} days serving every eligible request to both model and incumbent path, scoring both, shipping only the incumbent's answer. That is where you learn the live distribution without owning the consequences. Then a ramp — {{5}} per cent, {{25}}, {{50}}, {{100}} — each step held a stated number of days, with a stated reading before the next and an abort condition beside it: what returns traffic to the previous split, and who calls it. It is the same no-deploy switch as the down fallback, because an abort that needs a release is not an abort.\n\n## The fallback\n\nWrong, slow and down are three failures needing three answers; detection and response for each are in the fields. Conflating them is how one incident becomes a contract conversation; drilling all three before traffic moves, with the customer's on-call in the room, makes the answers real.\n\nWrong is the under-specified one. An abstention route needs more than one clause. Name what produces the confidence signal: the verifier's own score, where it can run on a single live item without a reference. Where it cannot — anything needing an adjudicated comparison, a downstream outcome, or a later tool result — say what abstains instead: a cheap proxy the POC measured against the verifier, or blanket supervision for that case type. Then calibrate the bar on held-out data at a stated precision and recall point rather than by eye. Slow and down turn on one discipline: the cut is automatic, or a named engineer's no-deploy switch, never invented mid-incident.\n\n## The live eval\n\nThe held-out set proved the model in the POC. It cannot tell you the model is still good in March: it is sealed, static, and predates the deployment. The live eval is a separate instrument on a different distribution, and needs a scale of its own. It runs continuously — {{2}} per cent, or {{500}} items a day, whichever is larger, stratified so the minority outcome is not swamped — judged by the verifier the POC defended, on the same rubric.\n\nDo not compare its reading to the POC threshold: that came from a stratified sealed set, the live sample is unstratified production traffic, and the two are not on the same ruler. Take a baseline instead: the mean daily reading over the first {{7}} days of served traffic, with its confidence interval. The control band is ±{{1.96}} standard errors of the daily sample at that sample size, and its numeric bounds go on the checklist once computed. Every later judgement — trigger, escalation, handover — moves against that baseline and band.\n\nWhere model-judged scoring is primary, keep a human-rated stream at {{50}} items a week, adjudicated by the POC's adjudicators. That rate is an accrual, not a new audit: it accumulates to a re-run of the POC's dual-labelled audit, same size, same floor. The floor is set by the **Strategic Deployment Playbook**; measurement belongs to **Post-Training POC Structure**.\n\nMonitoring tells you the system is up; the live eval tells you the output is still correct, which no infrastructure stack knows to ask. Build both. The spoken version of this argument is narrative 7, \"Prototype to production\", in **Technical Narratives**.\n\n## Drift and the retraining trigger\n\nName the trigger before go-live, in writing. An engagement with no named trigger retrains when someone complains, which is too late.\n\n| Trigger | Written as | Who pulls it |\n| --- | --- | --- |\n| Quality | Live eval below the baseline band for {{3}} consecutive weekly readings | Production owner, on the lead's recommendation |\n| Distribution | A monitor compares live input features to the training distribution against a stated bound and fires to a named recipient: new case types, a channel absent at training, volume mix moving more than {{20}} per cent | Customer flags, Applied Research confirms |\n| Product | Any change to the upstream workflow, the tool set the model calls, or the policy it applies | Customer, and they must tell us before it ships |\n\nThe distribution row is gated because it needs an instrument built, not a number agreed. The product row catches people: product changes are not drift in the statistical sense and no monitor sees them, but they invalidate a model faster than anything else.\n\n## Human review in the loop\n\nEvery case type sits in one of three states: supervised, sampled, or automated. Assign each one, using the case-type list from the environment specification produced in **Reference Architectures**.\n\nItems move on evidence, not confidence. Promotion from supervised to sampled needs its own quantity — {{500}} reviewed items with reviewer corrections below {{2}} per cent is a reasonable assumption. That is not the verifier floor, which bounds agreement between verifier and adjudicated labels and says nothing about a per-item pass rate in production. Promotion to automated needs that bar sustained over a stated period with the fallback exercised once; demotion is easier, one trigger breach moving a case type back a state. The customer will want to automate faster than the evidence allows. Point at the rule you both signed, not at your judgement.\n\n## Ownership and paging\n\nWrite the split down. The customer's on-call owns the inference path, the routing switch, the fallback and everything the model sits inside. They are paged at 3am, and their runbook answers the four questions in `own-02`, the last being when to wake us.\n\nPrime Intellect owns the model and the training and evaluation apparatus, on a {{business-hours}} response targeting {{4}} working hours, with a named escalation contact. Treat that as a planning assumption: the real terms are whatever the signed agreement says, and commercial terms belong to **Pricing and Packaging Inputs**. Under it the forward deployed lead is not on the rota and should say so out loud, or the unwritten expectation surfaces mid-incident.\n\nThe playbook's exit for this stage is three things — the recorded traffic percentage actually being served, {{7}} days of live-eval data, and a named rota owner. All three are checks below; two gate traffic, and the {{7}} days arrive as ev-02. A shared mailbox is not an owner, so verify the rota in the paging tool, not verbally.\n\n## The handover\n\nThis is where \"own your intelligence\" is either true or was marketing, and it is the best predictor of expansion. The test is whether the customer's engineer, without us on the call, can do four things.\n\n1. Retrain from current data and reproduce a run within a stated tolerance.\n2. Run the eval end to end and read the result.\n3. Rebuild the environment from source.\n4. Roll back to the previous model version in production.\n\nOnly rollback gates traffic, because only it is needed the moment the first request lands; the other three must be passed by day {{30}}, and handover waits on them. Rollback also has an artefact test, not only an act: the previous version stays pinned and retained for a stated period, identified by hash. Verify by watching in silence: where we drive the keyboard, the handover fails the first time it matters. Record date and engineer in PIG.\n\n## Cost at steady state\n\nPut inference cost per unit of work beside the pre-deployment cost of the same unit — incumbent, frontier model, or human — including serving overhead and the surviving review queue, not only tokens. Then say the honest thing: a better and more expensive model still has to clear a bar, and that bar belongs to the customer's finance owner. Get the name and the number on the checklist before go-live: a bar discovered at renewal is a bar you failed.\n\n## The go/no-go gate\n\nGated items are marked `gate: true` in the fields, each written so a third party could check it: a document, a dashboard, a drill record, a name in a rota. No traffic moves — not the first {{5}} per cent — until every one is yes, not \"yes with a caveat\", because a caveat is a no with better manners. Ungated items must still exist, but they are worked in the first thirty days and do not hold traffic.\n\n## The first thirty days\n\nThe watch list is in the fields; two points are worth arguing. The daily {{7}} days are watched with the customer, in their channel, so the first surprise is shared rather than reported later; they also produce the baseline, so nothing is judged against a band until day {{8}}. And the handover point is a date, not a feeling: at day {{30}}, if the eval has held within band for {{3}} consecutive readings, the fallback has been exercised once for real, and the customer has passed all four parts of the handover test, the forward deployed lead moves to a monthly review.\n\nSay that date out loud at go-live. A lead who never stops watching has not deployed anything; they have become part of the system.",
|
||
"fields": {
|
||
"checks": [
|
||
{
|
||
"id": "inf-01",
|
||
"area": "Inference path",
|
||
"check": "The end-to-end inference path is drawn as one diagram, starting from the serving shape and component table in Reference Architectures, naming every hop, which model artefact version is loaded and from where, and whether the verifier runs inline at serve time or out of band.",
|
||
"evidence": "The diagram, attached to the checklist and linked from the demand record in PIG, with the Reference Architectures shape named on it.",
|
||
"owner": "joint",
|
||
"gate": false
|
||
},
|
||
{
|
||
"id": "inf-02",
|
||
"area": "Inference path",
|
||
"check": "The location where the adapted weights are actually served matches the egress boundary and inference-location decisions recorded in the Data and Security Brief.",
|
||
"evidence": "Written confirmation from the control owner or auditor named in the Data and Security Brief, citing the serving region, tenancy and network path.",
|
||
"owner": "joint",
|
||
"gate": true
|
||
},
|
||
{
|
||
"id": "inf-03",
|
||
"area": "Inference path",
|
||
"check": "A latency budget is stated as p95 and p99 in milliseconds, with the name of the person who owns the consequence of a breach.",
|
||
"evidence": "The budget line on the checklist, with a name and a date, acknowledged in writing by that person.",
|
||
"owner": "customer",
|
||
"gate": true
|
||
},
|
||
{
|
||
"id": "inf-04",
|
||
"area": "Inference path",
|
||
"check": "A load test has been run at the customer’s measured peak concurrency, sustained long enough for queueing effects to appear, and p99 stays inside budget including cold starts if the stack scales to zero.",
|
||
"evidence": "Load test output showing concurrency, duration, the p50/p95/p99 distribution, and either cold-start timings or the configuration showing a warm floor.",
|
||
"owner": "joint",
|
||
"gate": true
|
||
},
|
||
{
|
||
"id": "rmp-01",
|
||
"area": "Traffic ramp",
|
||
"check": "The shadow-mode period and the agreed ramp are recorded as traffic percentages against dates, with the reading required before each step.",
|
||
"evidence": "The ramp table on the checklist — shadow start and end, then each percentage with its date and its entry condition — countersigned by the production owner.",
|
||
"owner": "joint",
|
||
"gate": true
|
||
},
|
||
{
|
||
"id": "rmp-02",
|
||
"area": "Traffic ramp",
|
||
"check": "An abort condition is written that returns traffic to the previous split without a deploy, naming who may call it and what reading triggers it.",
|
||
"evidence": "The abort line on the checklist plus the drill record showing traffic returned to the previous split and the elapsed time.",
|
||
"owner": "customer",
|
||
"gate": true
|
||
},
|
||
{
|
||
"id": "fb-01",
|
||
"area": "Fallback",
|
||
"check": "An abstention route exists, the signal it uses is named, the case types where the verifier cannot score inline have a stated alternative, and the confidence bar was calibrated on held-out data at a stated precision and recall point rather than chosen.",
|
||
"evidence": "The threshold in configuration, the calibration curve with the chosen operating point, the list of case types using the alternative signal, and a test case showing an item routed to the human queue.",
|
||
"owner": "joint",
|
||
"gate": true
|
||
},
|
||
{
|
||
"id": "fb-02",
|
||
"area": "Fallback",
|
||
"check": "A hard timeout is set at the latency budget and cuts automatically to the pre-deployment path, as the calling service’s default rather than a manual decision.",
|
||
"evidence": "Timeout value in the calling service’s configuration and the drill record showing the cut.",
|
||
"owner": "customer",
|
||
"gate": true
|
||
},
|
||
{
|
||
"id": "fb-03",
|
||
"area": "Fallback",
|
||
"check": "All three fallback drills — wrong, slow, down — have been run against the production configuration, with the down drill performed by a named on-call engineer flipping the routing switch without a code deploy.",
|
||
"evidence": "Three dated drill records signed by the customer’s on-call, the down record naming the engineer and the time taken to flip.",
|
||
"owner": "joint",
|
||
"gate": true
|
||
},
|
||
{
|
||
"id": "ev-01",
|
||
"area": "Live eval",
|
||
"check": "A live eval samples production traffic continuously at {{2}} per cent or {{500}} items a day, whichever is larger, stratified so the minority outcome is not swamped, scored by the same verifier version defended in the POC against the same rubric.",
|
||
"evidence": "The sampling configuration, the verifier version identifier matching the POC scorecard, and {{7}} days of readings taken during shadow mode, visible in the customer’s own dashboard rather than in a document only we open.",
|
||
"owner": "prime",
|
||
"gate": true
|
||
},
|
||
{
|
||
"id": "ev-02",
|
||
"area": "Live eval",
|
||
"check": "A baseline live-eval reading is established over the first {{7}} days of served traffic, stated with its confidence interval, and the resulting control band is written down as numeric upper and lower bounds.",
|
||
"evidence": "The {{7}} daily readings, the computed mean and interval, and the two band bounds recorded on the checklist. Not available before traffic moves, so this is a day-{{8}} item.",
|
||
"owner": "prime",
|
||
"gate": false
|
||
},
|
||
{
|
||
"id": "ev-03",
|
||
"area": "Live eval",
|
||
"check": "A human-rated stream of {{50}} items a week accrues towards a re-run of the POC’s dual-labelled audit, at the same sample size and against the same verifier floor, using the POC’s trained adjudicators.",
|
||
"evidence": "First week of rated items, the accrual schedule showing when audit size is reached, and the agreement figure measured by the method owned by Post-Training POC Structure.",
|
||
"owner": "joint",
|
||
"gate": false
|
||
},
|
||
{
|
||
"id": "ev-04",
|
||
"area": "Live eval",
|
||
"check": "Alerts fire on p99 latency, error rate and cost per unit of work against stated numeric thresholds, routed to the on-call rota named in own-01, and separately from the live eval.",
|
||
"evidence": "The alert definitions with their threshold values and routing destination, plus one test page delivered to the rota.",
|
||
"owner": "customer",
|
||
"gate": false
|
||
},
|
||
{
|
||
"id": "dr-01",
|
||
"area": "Retraining trigger",
|
||
"check": "A quality trigger is written as a decline against the live-eval baseline band in ev-02 over a stated number of consecutive readings, not against the POC threshold, with the person allowed to pull it named; and the customer’s product owner is on a notification path for any change to the upstream workflow, the tool set the model calls, or the policy it applies.",
|
||
"evidence": "The trigger line on the checklist citing the band bounds from ev-02, its window and the name; plus the named product owner and the channel by which they notify us before a change ships.",
|
||
"owner": "joint",
|
||
"gate": false
|
||
},
|
||
{
|
||
"id": "dr-02",
|
||
"area": "Retraining trigger",
|
||
"check": "A distribution monitor compares live input features against the training distribution on a stated bound, and fires to a named recipient.",
|
||
"evidence": "The monitor configuration and its bound, plus one week of readings.",
|
||
"owner": "joint",
|
||
"gate": true
|
||
},
|
||
{
|
||
"id": "hr-01",
|
||
"area": "Human review",
|
||
"check": "Every case type in the environment specification produced by Reference Architectures is assigned exactly one state — supervised, sampled or automated — and the evidence to promote or demote is written as a volume, a reviewer-correction rate and a period.",
|
||
"evidence": "The state table on the checklist, reconciled line by line against the environment specification, with the promotion rule countersigned by the production owner.",
|
||
"owner": "joint",
|
||
"gate": false
|
||
},
|
||
{
|
||
"id": "dat-01",
|
||
"area": "Data handling",
|
||
"check": "Sampling, human rating and abstention logging of live traffic fall inside the permitted use scope and output retention window recorded in the Data and Security Brief.",
|
||
"evidence": "The use-scope and output-retention lines from the Data and Security Brief set beside the live eval sampling, rating and abstention-log configurations, confirmed by the control owner named there.",
|
||
"owner": "joint",
|
||
"gate": true
|
||
},
|
||
{
|
||
"id": "own-01",
|
||
"area": "Ownership",
|
||
"check": "The alert resolves to a named person in the customer’s own on-call rota, not a shared mailbox or a team alias.",
|
||
"evidence": "The rota entry in the customer’s paging tool, plus one test page acknowledged by a human.",
|
||
"owner": "customer",
|
||
"gate": true
|
||
},
|
||
{
|
||
"id": "own-02",
|
||
"area": "Ownership",
|
||
"check": "The on-call runbook covers confirming the model is the cause, flipping to fallback without a deploy, verifying the fallback is serving, and when to escalate to Prime Intellect.",
|
||
"evidence": "The runbook document, exercised in a drill by an on-call engineer who did not write it.",
|
||
"owner": "joint",
|
||
"gate": true
|
||
},
|
||
{
|
||
"id": "ho-01",
|
||
"area": "Handover",
|
||
"check": "A customer engineer has, unaided and with us watching in silence, retrained and reproduced a prior run within a stated tolerance, run the eval end to end and read the result, and rebuilt the environment from source.",
|
||
"evidence": "Run identifiers, eval run link and environment build log, all initiated from a customer account, with the engineer’s name, date and tolerance achieved. Due by day {{30}}.",
|
||
"owner": "customer",
|
||
"gate": false
|
||
},
|
||
{
|
||
"id": "ho-02",
|
||
"area": "Handover",
|
||
"check": "A rollback to the previous model version has been performed by the customer in production or an identical staging environment, and that previous version is pinned, retained for a stated period, and identified by artefact hash.",
|
||
"evidence": "Rollback record with timestamps and the operator’s name, plus the pinned version’s hash, storage location and retention period, consistent with the output-retention and termination decisions in the Data and Security Brief.",
|
||
"owner": "joint",
|
||
"gate": true
|
||
},
|
||
{
|
||
"id": "co-01",
|
||
"area": "Cost",
|
||
"check": "Inference cost per unit of work is calculated including serving overhead and any surviving review queue, placed beside the pre-deployment baseline for the same unit, and beside a written cost-per-unit bar from a named finance owner.",
|
||
"evidence": "The cost comparison on the checklist with its assumptions stated, and the bar and the finance owner’s name acknowledged by that person before go-live.",
|
||
"owner": "joint",
|
||
"gate": true
|
||
}
|
||
],
|
||
"fallbacks": [
|
||
{
|
||
"failure": "The model is wrong — a plausible output that is incorrect, off-policy, or outside the case types it was trained on.",
|
||
"detection": "The verifier scoring inline where it can run against a single live item without a reference; for case types where it cannot, the cheap proxy named in fb-01. Also downstream rejection by the consuming system, a reviewer catching it in the sampled queue, or a spike in the live eval’s minority outcome.",
|
||
"response": "Abstain rather than answer. Below the calibrated confidence bar the item routes to the human queue, and the case type is demoted a supervision state automatically if abstentions or reviewer corrections breach their stated bound. Log every abstention as training signal for the next run, inside the retention window in dat-01."
|
||
},
|
||
{
|
||
"failure": "The model is slow — the latency budget is breached at p99 while the service is nominally healthy.",
|
||
"detection": "Latency monitoring on the calling service, alerting on p99 against the budget rather than on the median. Queue depth rising ahead of the latency alert is the earlier signal.",
|
||
"response": "Hard timeout at the budget with an automatic cut to the pre-deployment path — incumbent system, prompted frontier model, or a queue that degrades throughput rather than quality. This is the calling service’s default behaviour, never a human decision taken during an incident. Page the customer’s on-call if the cut persists beyond {{15}} minutes."
|
||
},
|
||
{
|
||
"failure": "The model is down — the endpoint is unavailable, erroring, or serving a corrupted or wrong-version artefact.",
|
||
"detection": "Health checks against the serving endpoint plus error-rate alerting. Version assertion on every response catches the wrong-artefact case, which health checks alone will not.",
|
||
"response": "The named on-call engineer flips the routing switch without a deploy, confirms the fallback is serving, and follows the runbook to decide whether to wake Prime Intellect. The workflow’s documented behaviour while flipped is communicated to affected internal users, not discovered by them. If this happens during the ramp, it is also the abort path in rmp-02."
|
||
}
|
||
],
|
||
"firstThirtyDays": [
|
||
{
|
||
"when": "Days 1 to {{7}}, daily, reviewed in the customer’s own channel",
|
||
"watch": "The daily live eval reading, accumulating towards the baseline in ev-02; p99 latency against budget; count of fallback activations by type; human review queue depth and the age of the oldest item; the traffic percentage actually served against the ramp recorded in rmp-01.",
|
||
"escalateIf": "The daily live eval reading falls more than {{2}} standard errors below the running mean of the days before it; p99 breaches budget more than once; the fallback activates without anyone noticing before the dashboard showed it; the served percentage does not reach the first ramp step on its date; or the queue grows faster than reviewers can clear it. Judge nothing against the POC threshold in this window."
|
||
},
|
||
{
|
||
"when": "Days {{8}} to {{30}}, weekly, reviewed with the production owner",
|
||
"watch": "Live eval trend against the control band computed in ev-02; cost per unit of work against the finance owner’s bar in co-01; the served traffic percentage against the ramp in rmp-01; the distribution monitor’s readings against its bound; case types whose evidence now supports a state change; open runbook gaps found during any incident.",
|
||
"escalateIf": "Two consecutive readings sit below the band; cost per unit exceeds the finance owner’s stated bar; the served percentage has not reached the next recorded ramp step for two weeks; the distribution monitor has fired without a response; or the customer proposes automating a case type that has not met its promotion evidence."
|
||
},
|
||
{
|
||
"when": "Day {{30}}, the named handover point",
|
||
"watch": "That the live eval has held within its band for {{3}} consecutive readings, the fallback has been exercised at least once for real, and the customer has passed all four parts of the handover test in ho-01 and ho-02. If all three hold, the forward deployed lead stops daily watching and moves to a monthly review, and the expansion conversation from Case Study Frame opens.",
|
||
"escalateIf": "Any of the three is not true at day {{30}}. Do not extend the daily watch silently — name the failing item to the sponsor, set a new date, and record it against the demand record in PIG."
|
||
}
|
||
]
|
||
},
|
||
"notes": "This template exists to prevent the deployment that never happens: a POC that met its threshold, a pleased sponsor, and traffic that quietly never moves past the pilot slice because nobody owned the p99, the ramp, the fallback or the page. Gated checks must be yes before the first percentage of traffic moves; ungated ones must exist but are worked inside the first thirty days, so a lead who treats every line as a blocker will stall a ready deployment. The four-part handover test is the one that predicts expansion — run it by watching in silence, never by driving the keyboard. Numbers in double braces are stated planning assumptions, not observed figures or commitments; support and commercial terms are whatever the signed agreement says, and where anything here appears to conflict with verifier measurement or POC shape, Post-Training POC Structure governs."
|
||
}
|