{
  "key": "quillon-eval-harness",
  "dealName": "DEMO — Evaluation harness pilot",
  "accountName": "DEMO — Quillon AI",
  "stage": "qualification",
  "status": "open",
  "acv": 135475,
  "summary": "DEMO — Quillon AI's contract-review product ships on vibes: 31,000 documents a month, rich acceptance telemetry, and a 42-case spreadsheet that passed 42 of 42 straight through a regression that cost them a top-ten account's confidence. Scored 54.5%, scope down, so we are selling the measurement and nothing else; the blocker is that there is no budget code and the metric owner has cancelled twice.",
  "openedDaysAgo": 34,
  "artefacts": [
    {
      "templateSlug": "technical-discovery",
      "stage": "qualification",
      "title": "DEMO — First technical discovery: Quill Review clause suggestions",
      "status": "final",
      "authoredDaysAgo": 22,
      "body": "*Discovery call, 90 minutes, on their side: Tom Feld (CTO, co-founder), Asha Venkataraman (VP Engineering), Radek Wilczek (staff engineer, owns the eval spreadsheet). Rosalind Danquah (Head of Product) accepted and did not attend.*\n\n## The task, in one sentence\n\nGiven an inbound supplier contract and the buying organisation's clause playbook, Quill Review produces a clause-level redline plus a severity-ranked risk summary, which a procurement reviewer accepts, edits or rejects clause by clause inside the Quill web app.\n\n## Volume and shape\n\nAbout 31,000 documents a month across 60 enterprise accounts, from a query Radek ran on the call rather than from memory. Nine flagged clauses per document on average, so roughly 280,000 clause suggestions a month. The shape repeats: the same twelve clause families (limitation of liability, indemnity, data processing, assignment, termination for convenience, audit rights and so on) account for 78% of all suggestions. This is a high-volume, repeating, same-shape task, which is the first thing I was listening for.\n\n## What is actually going wrong\n\nThey cannot tell whether a change makes Quill better. On 9 March they replaced the retrieval layer that grounds suggestions in the customer's playbook. Acceptance on data-processing clauses fell from 58% to 39% and nobody noticed for five weeks, until a top-ten account escalated after their own counsel found three inverted DPA suggestions in one week. Tom's phrase, verbatim: \"we shipped it because the demo felt better.\"\n\nThey have evals. Forty-two hand-written cases in a spreadsheet, judged pass or fail by Tom, last edited in February, currently passing 42 of 42. It did not move on 9 March and it has never moved. It is decorative, and Radek said so in the room before I had to.\n\n## Traces, and the rights to use them\n\nStrong. Every suggestion has been logged since March 2024 with the reviewer's disposition (accepted as written, edited, rejected) and, where edited, the reviewer's final text. That is a per-clause outcome signal at roughly 280,000 rows a month, already joined to the document and the playbook version.\n\nTwo problems with it. Four tenants have bulk auto-accept switched on and represent 22% of volume, so their dispositions are meaningless. And reviewers under deadline accept things they would not defend, which Asha believes but cannot quantify. So the telemetry is a proxy, not an adjudicated label.\n\nRights: their MSA permits use of customer documents for service improvement in 44 of 60 accounts; 16 have opted out. Nobody has asked counsel about training, but nobody needs to yet, because nothing here is a training engagement. The 16 opt-outs do bind the measurement work, and I have carried them forward as a hard constraint on the sampling frame.\n\n## Verification\n\nThere is a computable proxy today and a credible path to adjudicated labels: two of their solutions engineers are ex-procurement and can label clause suggestions against the playbook. No human-labelled held-out set exists. Zero labels, today.\n\n## What I wrote down\n\nRepeatable: yes. Traced: yes, richly. Checkable: proxy only, unmeasured against human judgement. That combination is a scoping conversation with a verifier-measurement phase in front of it, and the verifier-measurement phase is the entire deal. No training in scope. Nothing should be trained until somebody can say what better means and prove the measurement agrees with a human.",
      "kind": "discovery"
    },
    {
      "templateSlug": "trainability-qualification",
      "stage": "qualification",
      "title": "DEMO — Trainability scorecard read-out: measure before you train",
      "status": "final",
      "authoredDaysAgo": 20,
      "body": "*Scored from the discovery notes of two days ago, with Asha Venkataraman on the phone for the commercial half. This is the trainability scorecard, run here to establish the opposite of what it usually establishes: that Quillon are not yet trainable. Weighted result 218 of 400, or 54.5%. Band: scope down.*\n\n## Where the points went\n\nTechnical subtotal 132 of 228. Commercial subtotal 86 of 172. That split is the whole story: a real, well-traced problem inside a company that has not decided to pay for it out of any particular pocket.\n\n**Budget line and its source, loss of 27.** This is the bad one and I am naming it as bad rather than softening it. There is no budget code. Tom gestured at a roughly $400k infrastructure envelope he controls and said \"this would come out of that\", which is an intention, not a line. Nothing is displaced, and no approval path exists because nothing of this shape has been bought here before. Scored 1. If this is still a 1 at proposal, we do not have a deal, we have a workshop.\n\n**Verifier availability and quality, loss of 26.** Scored 2. The acceptance telemetry is a genuine computable proxy, which keeps this clear of the no-verifier disqualifier, but it has never been measured against human judgement, 22% of volume comes from auto-accept tenants and is worthless, and there are zero adjudicated labels in existence. The unblocking artefact is the labelled set sized in the spreadsheet note: 240 adjudicated items, 20 per clause family, with rater-pair agreement reported as kappa alongside judge-versus-human agreement. Those numbers come from Quillon's own family count and their raters' committed hours, not from our template.\n\n**Measured baseline, loss of 24.** Scored 1. Nothing is measured. The 42-case spreadsheet passes 42 of 42 and did not move through a regression that cost them a top-ten account's confidence, so it is not a baseline, it is a decoration.\n\n**Headroom over prompting and workflow design, loss of 20.** Scored 2, and worth being precise about, because it decides whether this engagement leads anywhere. Their prompt authors are actively rewriting the retrieval and grounding layer, and the 42-case sheet stayed green straight through it. Prompting headroom here is untested, not exhausted: nobody can tell whether better prompting closes the gap until the harness exists. A 2 is what \"we cannot distinguish\" is worth.\n\n## What is not wrong\n\nTask definability 3, traces 3, environment constructibility 3: they can replay any historical document through any pipeline version deterministically, having built that harness for support debugging. Expansion surface 3, with obligations extraction and renewal triage named as adjacent workflows with named owners.\n\n## The commercial items\n\nMetric owner scored 2. Rosalind Danquah owns acceptance rate and has cancelled twice, on 9 and 16 July, without rescheduling. Asha will not state a threshold on Rosalind's behalf and is right not to. This grazes the \"nobody states the number\" disqualifier. It does not fire yet, because Asha has committed to getting Rosalind's threshold in writing before we propose; if that sentence does not exist by 31 August, it fires and this score becomes irrelevant.\n\nExecutive sponsor scored 3. Tom is a co-founder, was in every call, and asked for the work himself; he has just never spent money on anything shaped like it.\n\nSecurity and legal path scored 2. The MSA permits service-improvement use in 44 of 60 accounts and 16 have opted out, which fixes the sampling frame at 40 accounts before we start. Counsel has never been asked anything, and nobody at Quillon owns that question.\n\nForcing function scored 2. They intend to migrate Quill's core model by 15 November and cannot responsibly do it blind. Real and dated, but self-imposed, and nobody has costed a slip.\n\n## The action the band prescribes\n\nScope down: do not sell the workflow. The sub-task that stands on its own is the measurement, meaning the held-out set, the verifier and the agreement number. Training stays out of the conversation until the verifier exists with an agreement figure attached.",
      "kind": "qualification"
    },
    {
      "templateSlug": null,
      "stage": "qualification",
      "title": "DEMO — Why the 42-case spreadsheet always says 100%",
      "status": "final",
      "authoredDaysAgo": 15,
      "body": "*Written after Radek Wilczek sent me the sheet. Shared back to him and to Asha; Radek's reply was \"yes, this is the argument I have been losing internally for a year.\"*\n\n## The artefact\n\n`quill_eval_cases_v3.xlsx`. Forty-two rows. Columns: case name, input document, expected behaviour in prose, pass or fail, notes. Last modified 14 February. Current result 42 of 42 pass. It is run by hand before major releases, by whoever remembers.\n\n## One correction to my own discovery note\n\nI wrote down nine flagged clauses per document from the call. Radek's query says 9.4 once suppressed low-severity flags are counted, which the log does and his verbal answer did not. That is about 291,000 suggestions a month, and it matters less for the headline than for the per-family floor below: the suppressed flags sit in the two rarest families, exactly where the sample is thinnest.\n\n## Why it always passes\n\n**It was written from fixed bugs.** Every case reconstructs something that went wrong and was fixed. A suite built backwards from resolved defects cannot fail unless a fix regresses, and cannot see a quality change in the behaviour nobody wrote a case for.\n\n**Pass is a human verdict rendered by the person who wants to ship.** The expected-behaviour column says things like \"should flag the uncapped indemnity and explain why\". Tom reads the output and decides. That is not a verifier. It is a mood, recorded in a cell.\n\n**Forty-two cases against twelve clause families is under four per family.** The March regression moved accepted-as-written from 58% to 39% across roughly 14,000 data-processing suggestions a month, a family below the twelve-family average because only the 38% of contracts touching personal data carry a DPA clause at all. The sheet holds three data-processing cases. All three passed on 10 March, the day after the change, because all three sit at the easy end of the distribution.\n\n**There is no held-out set and no split.** Every case has been visible to every prompt author for over a year, and prompts have been tuned until the sheet is green, which is the definition of fitting to the eval.\n\n**It has no notion of severity or of disagreement.** A wrongly inverted DPA suggestion and a redundant assignment-clause comment are the same single row of \"fail\".\n\n## What replaces it, and how big it has to be\n\nSizing first, because everything downstream cites it. Twelve families at 20 adjudicated items each is 240. Twenty is not a round number chosen for comfort: below about fifteen the per-family agreement interval is wider than the effect we are looking for, and above twenty their raters run out of hours. Both raters have committed one day a week for three weeks, 24 hours each, and at the twelve items an hour they managed on a warm-up batch of thirty, double-labelled, that is roughly 290 items of capacity. So 240 target, 200 hard floor. The two rarest families sit near 2% of volume, so 20 apiece means oversampling them about 8:1 against their natural rate.\n\nThen, in order:\n\n1. A held-out set drawn from the live log after the last prompt edit on 2 July, stratified by clause family and tenant, frozen on day one and withheld from prompt authors thereafter. Their prompts are tuned continuously, so no historical window is untainted; freezing forward is the only honest split available here.\n2. Adjudicated labels from the two ex-procurement solutions engineers, with a written adjudication rule and both independent verdicts kept rather than merged away. Rater-pair agreement reported as Cohen's kappa: below 0.6 the labelling rule is the defect, not the judge, and we stop and rewrite it.\n3. A judge measured against those labels, reported as kappa with an interval, per family. Acceptance telemetry stays as a cheap continuous proxy, but only once we know what it correlates with, and with the four auto-accept tenants excluded.\n\nUntil step three exists, the 15 November model migration is a coin toss they will not be able to score.",
      "kind": "narrative"
    },
    {
      "templateSlug": null,
      "stage": "qualification",
      "title": "DEMO — Scope-down recommendation: a six-week eval harness, no training",
      "status": "draft",
      "authoredDaysAgo": 6,
      "body": "*Draft. Not sent. Written to have something concrete on the table before Tom's next call, and deliberately narrower than what he asked for.*\n\n## What Tom asked for, and what he is getting instead\n\nTom opened the last call asking whether we could \"train a model on our accept data\". Not yet, and the scorecard says so at 54.5% with a measured-baseline score of 1: you cannot train against a signal nobody has validated. What we propose instead is the measurement, delivered as a working harness their engineers own and run.\n\n## Scope\n\nSix weeks. One forward deployed lead, plus their two ex-procurement solutions engineers as raters for one day a week each.\n\n**Weeks 1 to 2, the held-out pool.** Sample 1,200 clause suggestions from the live log since the last prompt edit on 2 July, stratified across the twelve clause families that make up 78% of volume and across tenants, excluding the four auto-accept tenants and the 16 accounts that opted out of service-improvement use, leaving 40 of 60 accounts in the frame. Frozen on day one and withheld from prompt authors thereafter; their prompts are tuned continuously, so a forward freeze is the only real split available.\n\n**Weeks 2 to 4, adjudicated labels.** 240 items drawn from the frozen 1,200, 20 per clause family, the two rarest oversampled about 8:1 against their natural rate; the rest of the pool is held for a second round once the judge is calibrated, so we never re-label the set the judge was fitted against. Both raters label independently against the customer playbook, with a written rule for what correct means at clause level and a recorded adjudication for every disagreement. Rater-pair agreement is reported as Cohen's kappa; below 0.6 we stop and rewrite the labelling rule. Hard floor 200 items; the sizing derivation is in the spreadsheet note.\n\n**Weeks 4 to 5, the judge.** A model-based verifier scored against the adjudicated labels, reported as kappa with an interval, per clause family. If it does not clear the bar on a family, we say so and that family stays human-scored rather than being quietly averaged in.\n\n**Weeks 5 to 6, the harness.** Runs in their CI against any pipeline version and reports per-family deltas. Its acceptance test replays the March regression from the historical log, a separate slice from the held-out set used only for that replay. Plus a written note on what the acceptance telemetry actually correlates with, the cheap signal they live on afterwards.\n\n## Where Tom pushed back\n\nHe wants four weeks, not six, and before the November migration work starts rather than alongside it. He is also unconvinced two raters at a day a week is worth the disruption, and suggested Radek label the set alone over a weekend. I have no good answer on four weeks beyond \"then the labelling is what gets cut\", and single-rater labelling removes the only agreement number that makes the rest defensible. Settle on the next call; I expect to lose a week somewhere.\n\n## Acceptance\n\nThe engagement is accepted when the harness, run by Radek on the 9 March change with no help from us, shows the data-processing regression as a significant negative delta. If it cannot see a regression we already know happened, we have not delivered.\n\n## The two things blocking this becoming a proposal\n\n**No budget line.** Not a soft no, an absent yes. Tom must produce a code and name what this displaces. The indicative figure carried on the demand record is $135,475.\n\n**Rosalind Danquah has not stated a threshold.** She owns acceptance rate and has cancelled twice. I want one sentence from her in writing: what accept-as-written rate on the top twelve families would make her comfortable shipping the November migration. The scorecard gives that until 31 August, eighteen days from today, before the \"nobody states the number\" disqualifier fires.\n\nNot in scope, and I want this sentence in the proposal verbatim: no training, no fine-tuning, no environment, no weights. Those are a second conversation that this engagement earns the right to have.",
      "kind": "proposal"
    }
  ],
  "scores": [
    {
      "daysAgo": 20,
      "note": "DEMO — First scoring, straight off the discovery call. Technical 132 of 228 and commercial 86 of 172 puts this at 54.5%, scope down: the traces and the task are real, but nothing is measured, no verifier has ever been checked against a human, prompting headroom is untested either way, and there is no budget code at all. Still open: Rosalind Danquah has not stated a threshold and has cancelled twice, on 9 and 16 July.",
      "dimensions": [
        {
          "id": "task_definability",
          "weight": 10,
          "score": 3
        },
        {
          "id": "verifier_quality",
          "weight": 13,
          "score": 2
        },
        {
          "id": "trace_data_rights",
          "weight": 8,
          "score": 3
        },
        {
          "id": "baseline_measured",
          "weight": 8,
          "score": 1
        },
        {
          "id": "headroom",
          "weight": 10,
          "score": 2
        },
        {
          "id": "env_constructibility",
          "weight": 8,
          "score": 3
        },
        {
          "id": "metric_owner",
          "weight": 9,
          "score": 2
        },
        {
          "id": "budget_source",
          "weight": 9,
          "score": 1
        },
        {
          "id": "exec_sponsor",
          "weight": 4,
          "score": 3
        },
        {
          "id": "security_legal_path",
          "weight": 8,
          "score": 2
        },
        {
          "id": "forcing_function",
          "weight": 8,
          "score": 2
        },
        {
          "id": "expansion_surface",
          "weight": 5,
          "score": 3
        }
      ]
    }
  ],
  "promote": null
}
