pig/packages/db/src/seed/motion/case-study-frame.json

{
  "kind": "case_study",
  "slug": "case-study-frame",
  "title": "Case Study Frame",
  "summary": "The structure every completed deployment gets written into, with the evidence rules that keep it honest, a fully worked fictional example, and the during-engagement harvest checklist that makes it writable at all.",
  "stage": "expansion",
  "body": "*Playbook stage: expansion — but the harvest starts at qualification.* This is the expansion-stage artefact of the **Strategic Deployment Playbook**, and the finished document is consumed back at qualification and proposal. The harvest checklist below runs across every stage, so open this file in week one, not at the end.\n\n## What this document is\n\nA case study is not a testimonial and not a press release. It is a short evidence document that lets a stranger in a similar seat decide, in four minutes, whether the same motion would work for them.\n\nThe discipline that makes it worth writing is simple and unforgiving: a case study without a baseline is marketing, not evidence. If you cannot say what the number was before, no reader can tell whether the number after is good. Half-written case studies are usually the ones where nobody measured the \"before\", and by the time anyone noticed, the old system had been switched off.\n\nWrite to the same standard you would apply to a customer's own eval report. Every number traces to a measurement someone can point at: a run, a dataset, a date, a person who ran it. Where a number cannot be sourced, it does not appear.\n\n## The ten sections\n\n| Section | The question it answers | What a weak version looks like |\n| --- | --- | --- |\n| The customer, in one line | Who is this, and why is their situation legible to the reader? | Adjectives instead of a sector, a scale and a constraint |\n| The workflow and what it cost | What specific loop was slow, expensive or unreliable, and by how much? | \"Manual processes were inefficient\" |\n| Why it was not a prompting problem | What did they already try, and why did it plateau? | Silence, which reads as \"they never tried the cheap thing\" |\n| What we built | Eval, verifier, environment, recipe, in that order | A product feature list |\n| How we know the verifier holds | Judge agreement with humans, reward hacking found, what the verifier misses | An assertion that the task is \"mostly verifiable\" |\n| The numbers | Baseline, result, held-out set, who measured | A single percentage with no denominator |\n| What surprised us | The finding that changed the plan mid-engagement | A generic anecdote with no date attached to it |\n| What the customer owns now | Assets, weights, environments, and their ongoing loop | Vague partnership language |\n| What came next commercially | The expansion that followed, and on what trigger | Omitted, which wastes the artefact |\n| Approvals | Who at the customer signed off on these words | Missing, which makes the whole thing unpublishable |\n\nThe ordering is deliberate. Eval before environment, always. If the eval is written after the environment, it will quietly be shaped to flatter the environment, and the reader who matters most — a sceptical head of ML at the next account — will spot it. The verifier section sits between the build and the numbers for the same reason: a number produced by an unexamined judge is not a measurement, and that reader will ask about the judge before they ask about the score.\n\n## Evidence discipline\n\nFour rules, non-negotiable.\n\n**One: every number traces to a run.** Keep the run identifier, the dataset version, the date, and the name of the person who executed it. Label customer-run and our-run measurements separately and never blur them. At least one headline row must be a customer-run measurement; ours may support it but may not substitute for it.\n\n**Two: the held-out set must be described, not just named.** Five fields, every time: size, sampling period, selection method, freeze date, and written confirmation that it was excluded from training and from verifier development. A held-out set assembled after the fact is not held out. State the sampling method even when it is boring: \"{{n}} tickets sampled uniformly from {{month}}, frozen on {{date}}, excluded from training and from verifier development per {{name}}'s written confirmation\".\n\n**Three: comparisons must survive a change of definition.** If the label space, the taxonomy or the population changed during the engagement — and on a good engagement it usually does — the baseline is re-run on the new definition and both figures are reported. Merging the two classes an old system confused will raise its successor's agreement score on arithmetic alone. Say which definition each number was measured under, or the strongest number in the document becomes the easiest one to dismiss.\n\n**Four: the customer approves the words.** Not the gist, the words. Send the exact paragraphs, with the numbers in them, to the named approver and get written confirmation. Anonymised case studies still need this — \"a top-five logistics platform\" is identifiable to anyone in the sector, and the legal exposure is identical.\n\nEngagements that missed their exit criteria get written up on this same template, internal-only, and the harvest is mandatory regardless of outcome. A library containing only wins is a selection bias, and the sceptical reader invoked above is precisely the person who infers one. The failure write-ups are what make the published ones credible to our own people.\n\nWhere a number is genuinely unavailable at writing time, leave a marked placeholder in braces rather than an estimate. A published placeholder is embarrassing; a published guess is a credibility event.\n\n## Worked example, illustrative and fictional\n\nThe following is a fictional example. Verrata Freight does not exist, and no part of it describes a real Prime Intellect customer, deployment or result. Every figure in it is invented to show the shape filled in.\n\n**The customer, in one line.** Verrata Freight is a fictional mid-market freight brokerage, roughly 400 staff, running about 9,000 shipment exception cases a week across three regions.\n\n**The workflow and what it cost.** When a shipment goes off-plan — missed pickup, temperature excursion, customs hold — an operations coordinator must read the carrier's notes and the shipment record, decide on one of eleven remedies, notify the customer, and file the claim. Median handling was 14 minutes on TMS timestamps. Two of the eleven remedies carried financial exposure and were misapplied often enough that the finance team ran a weekly manual audit costing about 30 hours.\n\n**Why it was not a prompting problem.** They had already built a prompted assistant on a frontier model and iterated on it for five months, reaching an informal 71% agreement with their senior coordinators on the original eleven-way taxonomy. It stalled there. The failures were not phrasing failures: the model could not sequence a multi-step lookup across the carrier portal and their TMS, and it had no way to learn from the coordinators' corrections, which were being logged and thrown away.\n\n**What we built.** First an eval: 620 dev-split exceptions used for iteration, each labelled with the remedy the senior coordinator actually applied and the financial outcome ninety days later; then a disjoint 300-case held-out set drawn from the following quarter, sampled uniformly by week, frozen on {{date}} before any training began. Then a verifier: the remedy code is checkable exactly, and the claim filing is checkable against schema plus the carrier's acceptance response, so roughly 80% of the trajectory is programmatically verifiable and only the customer-facing message needed a rubric judge. Then an environment: a sandboxed replica of the TMS and carrier portal with recorded responses, so the agent could take real multi-step actions without touching production. Then the recipe: supervised fine-tuning of a {{base model class}} on 4,100 coordinator trajectories drawn only from cases predating both eval sets, then RL against a composite reward — exact remedy-code match at 0.6, claim schema and carrier acceptance at 0.25, rubric judge on the customer message at 0.15, with a hard zero on any trajectory that filed a claim the sandbox rejected. Because the message score carried the least weight, we ran a regression check on it explicitly: message quality on the held-out set was flat within the judge's noise band while remedy accuracy rose. The whole run was {{n}} GPU-hours over eleven days, of which the environment build was seven.\n\n**How we know the verifier holds.** The rubric judge was validated against two of Verrata's senior coordinators on 200 held-out messages neither had seen: judge-to-human agreement 91%, human-to-human 93%, so the judge is roughly as reliable as the raters it replaces and no more. Two reward-hacking behaviours showed up in RL. The policy learned to pad messages with hedging clauses the rubric read as thoroughness, patched with a length penalty and a rubric revision; and under uncertainty it drifted towards the single most common remedy, which the aggregate reward was indifferent to and per-class recall caught. One thing the verifier does not cover at all is escalation tone — whether a message that should sound apologetic does — so 50 cases were read by hand by Verrata's ops lead and signed off separately.\n\n**The numbers.** During the engagement the remedy taxonomy changed from eleven codes to ten (see below), so agreement figures are reported under both.\n\n| Measure | Baseline | Result | Measured on | Measured by |\n| --- | --- | --- | --- | --- |\n| Remedy agreement, ten-way taxonomy | 74%, their prompted system re-run unchanged in {{month}} | 89% | The frozen 300, held out from training and verifier development | Verrata's ops analytics team |\n| Remedy agreement, original eleven-way taxonomy | 71%, their own informal figure over five months of iteration | not measured; the eleven-way taxonomy was retired before training | — | Verrata |\n| High-exposure remedy misapplication | 6.2% | 1.4% | The same frozen 300; the two collapsed codes were not the two exposure-bearing remedies, so this denominator is unchanged across both taxonomies | Verrata, audited by finance |\n| Median handling time, assisted | 14 min | 6 min | One region, all exception types, weeks 1–3 after cutover, 2,300 cases; both figures from TMS timestamps, the baseline from the same region's preceding eight weeks | Verrata's TMS |\n\nThe handling-time row has an obvious confound: the taxonomy was simplified in the same window, which would shorten handling on its own. A second region ran unassisted on the new taxonomy through the same three weeks and held at 13 minutes, which is the best available control and is worth exactly as much as the reader thinks it is.\n\n**What surprised us.** The largest single gain came before any training. Building the eval exposed that two of the eleven remedy codes were used inconsistently by the coordinators themselves; agreement between two senior coordinators on the same case was only 84%. On {{date}} we collapsed those two codes into one with a follow-up field, re-scored the prompted system against the collapsed taxonomy on the same frozen 300, and it came back at 74%. Three of the eighteen points we eventually reported were a definition fix, not a model, and saying so costs nothing next to being caught not saying it.\n\n**What the customer owns now.** The environment, the verifier and its rubric, both eval sets, the trained weights, and a retraining loop that ingests coordinator corrections weekly instead of discarding them. They run the eval themselves on a schedule and it gates their own deploys.\n\n**What came next commercially.** The exception environment became the template for two adjacent workflows, carrier onboarding checks and rate-quote validation, contracted as a second phase. The trigger was their own eval showing the same plateau pattern in a neighbouring queue.\n\n**Approvals.** {{Name}}, VP Operations, approved v4 of this text on {{date}}; legal cleared the named use on {{date}}. Run identifiers r-2291 and r-2317, dataset v3, held on the demand record in PIG.\n\n## The harvest checklist\n\nCase studies die in reconstruction. Six weeks after go-live nobody can find the pre-POC baseline, the person who ran it has changed teams, and the surprising finding has been smoothed into \"it went well\". Capture as you go.\n\nThe rule of thumb: at every stage, ask \"if we win, what will I wish I had written down today?\" and write it down today. Three items carry most of the mortality. The baseline has a shelf life measured in days, because the moment a POC starts the old system stops being run cleanly enough to measure. The dated definition-change log is the only thing that later lets you re-run a baseline under the definition you actually reported. And the run of record — identifier, dataset version, operator — is what separates a number you can publish from a number you merely remember. The register below is the full set.\n\nKeep the harvest on the demand record in PIG rather than in someone's notebook, or in the shared deal folder until demand attachments exist. Wherever it lives, it has to survive an account handover.\n\n## Before you publish\n\nConfirm four things. The baseline is stated, sourced, and measured under the same definition as the result. The held-out set carries all five of its fields: size, sampling period, selection method, freeze date, and written confirmation of exclusion from training and from verifier development. The named approver at the customer has confirmed the exact wording in writing. And legal has cleared any named use of the customer's brand against whatever publicity or confidentiality terms the executed agreement actually carries — read it, because some agreements prohibit naming outright, and that is a fact you need in week one rather than at publication. If any of the four is missing, the document is an internal reference note rather than a case study, and it should be labelled as such.",
  "fields": {
    "sections": [
      {
        "name": "The customer, in one line",
        "prompt": "Sector, scale, and the operating constraint that makes their situation legible. One sentence. If the reader cannot place themselves next to this company, everything after it is wasted.",
        "evidenceRule": "Scale figures (headcount, volume, regions) come from the customer, not from a public profile or an inference. Anonymised versions still need the customer's sign-off on the descriptor, because sector plus scale usually identifies them anyway."
      },
      {
        "name": "The workflow and what it cost them before",
        "prompt": "Name the specific loop: who does it, how many times a week, how long each instance takes, and what it costs when it goes wrong. Force a number here or the whole document floats.",
        "evidenceRule": "Every cost figure names its source system or the team who produced it: ticket timestamps, finance audit hours, headcount times rate. No modelled or extrapolated costs unless labelled as an estimate with its assumption stated inline."
      },
      {
        "name": "Why it was not a prompting problem",
        "prompt": "What did they already try, for how long, and where did it plateau? Name the model class and the ceiling number. If they never tried prompting, say so and say why not.",
        "evidenceRule": "The plateau number must come from the customer's own prior system. If their prior evaluation was informal, describe it as informal rather than dressing it up as a benchmark, and re-run that system unchanged on the frozen set before comparing it to anything of ours."
      },
      {
        "name": "What we built",
        "prompt": "In order: the eval, the verifier, the environment, the recipe. State what fraction of the trajectory is programmatically verifiable versus judged, what the environment actually simulates, and for the recipe: base model class, the reward and its weights, the regression check on anything the reward under-weights, and one figure for compute or wall-clock.",
        "evidenceRule": "Describe the eval's construction date relative to environment work. Eval-after-environment is a legitimate thing to admit but never a thing to hide, because the reader will ask. State the relationship between every dataset named — dev split, held-out set, training trajectories — including their sizes and whether they overlap."
      },
      {
        "name": "How we know the verifier holds",
        "prompt": "The verifier is the hard part and the part a technical reader will interrogate first. Report what was validated and what was found, not that it was 'robust'.",
        "evidenceRule": "Three things or the section does not ship: judge-to-human agreement on a labelled sample with n stated, alongside human-to-human agreement on the same sample; the reward-hacking behaviours found during RL and what was patched; and at least one dimension the verifier does not cover, checked by hand, with the number of cases and who checked them."
      },
      {
        "name": "The numbers",
        "prompt": "A table: measure, baseline, result, held-out set, measured by whom. Include at least one measure the customer cared about before we arrived, not only the one we improved most. Where a definition changed mid-engagement, report under both definitions or state plainly which one each row uses.",
        "evidenceRule": "Each row traces to a run identifier, dataset version and date held on file. The held-out set is described by size, sampling period, selection method, freeze date and written confirmation of exclusion. Baseline and result are measured on the same label space and the same population, or the difference is stated in the row. At least one headline row is customer-run and labelled as such. Any live production figure names its subset and rules out, or names, the confounds running in the same window."
      },
      {
        "name": "What surprised us",
        "prompt": "The finding that changed the plan mid-engagement, usually a definitional or data problem found while building the eval. One paragraph, honest, with the consequence it had on scope.",
        "evidenceRule": "Must be a dated entry drawn from the harvest log. If the log holds no definition-change entry, publish without the section and note that the definitions held: a manufactured surprise is a fabricated number by another name."
      },
      {
        "name": "What the customer owns now",
        "prompt": "Enumerate the assets: environment, verifier, eval set, weights, and the loop that keeps them current. Say who runs the eval on an ongoing basis and what decision it gates.",
        "evidenceRule": "Ownership claims must match the executed agreement and its IP schedule, not the intent expressed in the proposal. Read the executed document before writing this section."
      },
      {
        "name": "What came next commercially",
        "prompt": "The expansion that followed, the trigger that caused it, and the workflow it targets. If nothing has followed yet, state the named next candidate and its blocker.",
        "evidenceRule": "Only describe expansion that is contracted, or that has a written scope and a named owner on the demand record in PIG. Pipeline hope is not an outcome and must not be written as one."
      },
      {
        "name": "Approvals",
        "prompt": "Name the approver, their role, the date they approved, and the exact version of the text they saw. Held with the document rather than published in it.",
        "evidenceRule": "Written confirmation from the named approver on the final wording, plus internal legal clearance against the publicity and confidentiality terms the executed agreement actually carries. Verbal approval does not count and does not survive a personnel change."
      }
    ],
    "harvest": [
      {
        "when": "Contract signature",
        "capture": "The publicity and confidentiality terms verbatim, and the anonymisation ceiling: may we say \"freight brokerage, 400 staff\", or only \"logistics\", or nothing at all?",
        "why": "This decides whether the engagement can ever produce a public asset or only a reference-call script. Discovering in week twenty that naming is prohibited means the entire harvest was collected for a document that cannot be published."
      },
      {
        "when": "Qualification, first technical discovery",
        "capture": "The customer's own words for the problem, the volumes, and the number they are judged on internally. Verbatim quotes with the speaker's name and role.",
        "why": "The framing sentence of the finished case study is almost always something the customer said in week one. Reconstructed later it becomes our language, and it stops sounding like them."
      },
      {
        "when": "Before the POC touches anything",
        "capture": "The baseline: current performance on the measure that will be reported, the method used to obtain it, who ran it, the raw output file, and the definition it was measured under.",
        "why": "This is the single highest-mortality item. Once a POC starts, the incumbent process stops being run cleanly, and a baseline cannot be recovered retrospectively without an argument about fairness."
      },
      {
        "when": "Scoping, when the eval is defined",
        "capture": "The construction record for every dataset and how they relate: dev split, held-out set and training pool, with sizes, sampling windows, selection rules, freeze dates, whether they overlap, and written confirmation the held-out set was excluded from training and from verifier development.",
        "why": "The first question a sceptical reader asks is how the held-out set was built and whether the training data touched it. Without a contemporaneous record you are answering from memory, which is exactly the answer that erodes trust."
      },
      {
        "when": "Whenever a definition changes",
        "capture": "A dated log entry: what was ambiguous, what the inter-annotator agreement was, what we changed, who at the customer agreed, and the baseline re-scored under the new definition on the same frozen set.",
        "why": "This is the raw material for the surprises section and the only thing that keeps a headline comparison valid across a taxonomy change. Re-scoring the baseline costs an afternoon during the engagement and is impossible six months later."
      },
      {
        "when": "When the verifier is first trusted with reward",
        "capture": "The judge validation sample with its human labels and n, the human-to-human agreement on the same sample, and thereafter a running list of reward-hacking behaviours observed and patched.",
        "why": "Judge quality is asserted in most write-ups and evidenced in almost none. The hacking list is also the most credible thing in the document, because nobody fabricates a list of ways their own system was gamed."
      },
      {
        "when": "Every training or eval run of record",
        "capture": "Run identifier, dataset version, configuration, date and operator, pinned to the demand record in PIG rather than left in a notebook or a chat thread.",
        "why": "Numbers you cannot re-derive cannot be published. Six months later the cluster of runs is indistinguishable and nobody can say which one produced the headline figure."
      },
      {
        "when": "Go-live and the first weeks in production",
        "capture": "Production-side measures with their source system named (timestamps, audit results, incident counts), the selection rule for whatever subset they cover, anything else that changed in the same window, and exports or screenshots taken on the day.",
        "why": "Live numbers drift, dashboards get rebuilt, and a production gain that coincided with a process change is worth little unless the coincidence was recorded at the time. A control cohort is cheap to arrange before cutover and impossible after."
      },
      {
        "when": "At handover to the customer's own team",
        "capture": "The asset inventory as it actually stands, and the name of the person who now runs the eval on a schedule.",
        "why": "The ownership section is the part customers most often ask to correct, because what the proposal intended drifts from what was really transferred."
      },
      {
        "when": "Within two weeks of the result landing",
        "capture": "The approval thread: the draft text sent, the approver's written reply, and legal's clearance note against the terms captured at signature.",
        "why": "Approvals get much harder as the champion moves on. Asking while the result is fresh and the champion is proud of it is the difference between a publishable document and a shelved one."
      }
    ]
  },
  "notes": "This template exists to prevent the three ways case studies die. The first is the missing baseline: nobody measured the before, so the after is unarguable and unpublishable, and the fix has to happen in week one rather than at the end. The second is the moving definition: the taxonomy or population changes mid-engagement, nobody re-scores the baseline against it, and the headline comparison silently becomes two different tasks. The third is retrospective reconstruction, where run identifiers, held-out set provenance and the mid-engagement surprises have all evaporated into chat threads, leaving only adjectives. Harvest during the engagement, keep the harvest on the demand record in PIG so it survives an account handover, and treat the customer's written approval of the exact wording as a hard gate rather than a courtesy. On stage: this is filed under expansion because it is produced after a deployment lands, but it is consumed in qualification and proposal, so anyone browsing the library by stage should be pointed here from those two stages as well."
}