arena/environments/grand_exchange_live/measure_ladder.py

"""Measure the policy ladder in the SHIPPED engine, before any constant is locked.

The spec this environment came from locked `horizon`, `FREEZE_RANGE`, `TARGET_SHARE` and
`TURNS` against arithmetic from a throwaway engine that no longer exists, and its ladder did
not reconcile with its own reward — impatient computes to 0.612 where 0.623 was reported.
Nothing here is taken from that document. Every row below is produced by `live.play`, the
engine the taskset will actually run, over the baskets a run will actually serve.

Two questions this file exists to answer, in this order:

  1. What does each policy score? Nine of them, from doing nothing to the reference itself,
     each differing from its neighbour by one judgement.
  2. **Does the turn budget bind?** Every reward in Arena saturates — `profit_ratio` and
     `efficiency` clip at 1.0 and `clean` is a boolean band — so an exhaustive policy that
     clears the band scores 1.000 and house rule 4 is structurally unreachable. That is
     risk #2 in the programme's register, it is answered as a NUMBER, and if the number is
     zero that is a finding about the reward shape rather than a failure of this lane.

Read block-wise, never as one mean: realised P&L over sixty ticks is the noisiest thing in
the repository, and `probe.py` already documents this generator swinging 0.48 against 0.18
between blocks of twenty-four. A ladder read off one block is a ladder read off the seed.

    uv run --no-project python measure_ladder.py
    uv run --no-project python measure_ladder.py --sweep-target
    uv run --no-project python measure_ladder.py --sweep-requote
    uv run --no-project python measure_ladder.py --sweep-reference
    uv run --no-project python measure_ladder.py --sweep-freeze
    uv run --no-project python measure_ladder.py --sweep-shape

--- what it measured, 2026-08-21, amd-server, 120 baskets from SEED_BASE ------------------

TURNS=8, TICKS_PER_TURN=7, TAIL_TICKS=4, HORIZON=60, FREEZE_RANGE=(1,4), TARGET_SHARE=0.90.

    policy                                total  profit  effcy  discip  clean       gp  looks
    inaction                              0.000   0.000  0.000   0.000  0.000        0   0.00
    staller                               0.000   0.000  0.000   0.000  0.000        0   0.00
    crude (market orders, one look)       0.126   0.207  0.186   0.095  0.017   -2,684   1.00
    spammer                               0.158   0.262  0.219   0.114  0.025   13,938   7.89
    restate (re-quote the same book)      0.219   0.375  0.362   0.168  0.000   25,244   7.90
    plausible (mean anchor, no filter)    0.406   0.563  0.524   0.391  0.142   37,084   1.00
    churn (re-place at full size)         0.447   0.684  0.550   0.430  0.042   49,926   8.00
    impatient (one-shot ref, one look)    0.538   0.739  0.722   0.560  0.150   50,172   1.00
    hindsight (whole-window anchor)       0.598   0.792  0.763   0.630  0.208   54,599   1.00
    exhaustive (re-quote every look)      0.631   0.849  0.749   0.669  0.192   63,802   7.97
    oracle (the live reference)           1.000   1.000  1.000   1.000  1.000   76,744   3.99

Every row reconciles with 0.45*profit + 0.30*discipline + 0.25*clean to three decimals,
which the spec's ladder did not.

**The budget binds. Oracle - exhaustive = 0.369, stdev 0.054 over five blocks of 24, worst
block 0.316.** That is fifteen times the 0.02 margin B2 proposed and it clears the 0.25
quantum of the binary `gate` term outright, so the comparison is not a cliff. It is also
not an artifact of the denominator: the reference earns 76,744 gp against exhaustive's
63,802 and out-returns it on peak capital, 0.302 against 0.219.

`hindsight` is the row that says the stepped form measures something the one-shot form
cannot. It quotes off the mean of the WHOLE window, held-out ticks included, and still
scores 0.598 — perfect price foresight in a single plan is worth less than playing the
window with none.

Three things this measurement overturned:

  the first engine's rule-4 gap was fake. With a reference that never redeployed sale
  proceeds, exhaustive earned 22% MORE gp than the reference and scored 0.848: the gap was
  the denominator being under-tuned, exactly the imitation-score defect `book.py` documents
  paying for once already. Fixed by giving the reference the idle-cash trigger.

  the freeze alone is not what binds. At freeze 0 the gap is still 0.180 — the queue reset
  and the capital churn carry it. The freeze sets the slope: 0.238 at 1 tick, 0.505 at 4,
  0.902 at 8. FREEZE_RANGE=(1,4) is kept because 8 turns a look into a catastrophe rather
  than a cost, and because flat-per-look is the settled decision.

  more turns bind harder, monotonically: 4x14+4 gives 0.155, 8x7+4 gives 0.369, 12x4+12
  gives 0.517. Eight is chosen for the token cost of a rollout, not because the gap needs
  it. If the gap ever needs widening, this is the lever, and it is measured.
"""

from __future__ import annotations

import random
import statistics
from dataclasses import replace
import sys
import time
import types
from pathlib import Path

HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE))
try:
    # The installed package, when there is one — `taskset.py` reads `__all__` off whatever
    # `sys.modules` holds, and a namespace shim has none, so an unconditional shim here
    # would break every taskset lookup in any process that also imports this file.
    import grand_exchange_live  # noqa: F401
except ImportError:
    # Nothing installed: register the package as a namespace pointing at the source and let
    # the leaf modules import under it, so `__init__` — and with it `verifiers` — never runs.
    # This is the path `probe.py` takes, and it is the reason this file can be graded with
    # the training stack absent.
    _shim = types.ModuleType("grand_exchange_live")
    _shim.__path__ = [str(HERE / "grand_exchange_live")]
    sys.modules["grand_exchange_live"] = _shim

from grand_exchange_live.book import (  # noqa: E402
    BUY_BAND,
    FILL_SHARE,
    MAX_ITEM_SHARE,
    MIN_CROSSINGS,
    SELL_BAND,
    TAX,
    Order,
    crossings,
    reference_orders,
)
from grand_exchange_live.live import (  # noqa: E402
    HORIZON,
    IDLE_SHARE,
    LIVE_VIABILITY_TRIES,
    REQUOTE_BAND,
    TAIL_TICKS,
    TICKS_PER_TURN,
    TURNS,
    Leg,
    Reference,
    desired_orders,
    live_reference,
    play,
    viable_live_market,
)
from grand_exchange_live.market import SEED_BASE, build_market  # noqa: E402
import grand_exchange_live.reward as reward  # noqa: E402

HEADER = "requote / idle"
BLOCK = 24
BLOCKS = 5
TARGET_SHARE = reward.TARGET_SHARE
"""The ceiling band, imported rather than restated. It was defined here while it was being
measured — `--sweep-target` is what chose 0.90 — and it moved into the shipped package the
day the taskset landed: a reward the probe computes and a reward the run computes are two
rewards, and the day they drift the probe is gating something the model is not paid for.
`reward.TARGET_SHARE` carries the measurement that chose it."""

score = reward.score
"""Likewise. One definition, three callers: the run, this ladder, and `probe.py` through it."""


# --- the baskets a run actually serves --------------------------------------------------
def baskets(count: int, base: int = SEED_BASE):
    """Skip-aware, exactly as `taskset.LiveExchangeTaskset.load` is: `viable_live_market` may pass over a
    seed the LIVE reference loses money in, and the next task starts after the seed it
    landed on. Grading a contiguous `range` instead would grade a different set of baskets
    from the one a model is served."""
    out, seed = [], base
    for _ in range(count):
        market = viable_live_market(seed, 5, 56, HORIZON)
        seed = market.seed + 1
        out.append(market)
    return out


# --- the ladder, as policies rather than as adjectives ----------------------------------
def inaction(view):
    """Never submits a sheet. House rule 3's floor."""
    return []


def staller(view):
    """Takes every look and does nothing with any of them. Scores what inaction scores —
    the point of the row is that BURNING the budget is not itself worth anything, which is
    the degenerate strategy a time-averaged reward would pay for."""
    return []


def spammer(seed: int):
    """Random legs on every look. Acts constantly, reads nothing."""
    rng = random.Random(seed)

    def policy(view):
        items = view.market.items
        return [
            Leg(i.name, rng.randint(1, i.buy_limit),
                round(i.prices[view.seen - 1] * rng.uniform(0.85, 1.15)),
                round(i.prices[view.seen - 1] * rng.uniform(0.85, 1.15)))
            for i in items if rng.random() < 0.6
        ]

    return policy


def one_look(plan_of):
    """Submit a plan on turn 1, never look again. The one-shot form, played live."""
    def make(market):
        state = {"done": False}
        orders = plan_of(market)

        def policy(view):
            if state["done"]:
                return []
            state["done"] = True
            return [Leg(o.item, o.quantity, o.buy, o.sell) for o in orders]

        return policy

    return make


def band(market, items, anchor, sizer, filtered=False, ranked=False):
    """`probe.py`'s strategy family, unchanged, so the live ladder's crude and plausible
    rungs are the same strategies the one-shot ladder grades."""
    plans, out, purse = [], [], float(market.capital)
    for item in items:
        prices = item.visible_prices(market.visible)
        volumes = item.visible_volumes(market.visible)
        if filtered and crossings(prices) < MIN_CROSSINGS:
            continue
        a = anchor(prices)
        buy = max(1, round(a * (1.0 - BUY_BAND)))
        sell = max(buy + 1, round(a * (1.0 + SELL_BAND)))
        lows = [p for p in prices if p <= buy]
        highs = [p for p in prices if p >= sell]
        qty = sizer(item, market, buy, prices, volumes)
        if qty <= 0:
            continue
        s = 0.0
        if ranked:
            if not lows or not highs:
                continue
            entry, exit_ = statistics.fmean(lows), statistics.fmean(highs)
            s = (exit_ * (1.0 - TAX) - entry) / entry
            if s <= 0:
                continue
        plans.append((s, item.name, qty, buy, sell))
    plans.sort(key=lambda p: -p[0])
    for _, name, qty, buy, sell in plans:
        qty = min(qty, int(purse // buy))
        if qty <= 0:
            continue
        purse -= qty * buy
        out.append(Order(name, qty, buy, sell))
    return out


def by_volume(item, market, buy, prices, volumes):
    lows = [p for p in prices if p <= buy]
    reach = int(FILL_SHARE * statistics.median(volumes) * (len(lows) / len(prices))
                * market.held_out)
    return min(item.buy_limit, reach, int(MAX_ITEM_SHARE * market.capital // buy))


def market_orders(market):
    """Buy ten percent over the last price and sell ten under: every offer fills and every
    round trip pays the tax. The crude maximiser."""
    return [
        Order(i.name, i.buy_limit, round(i.prices[market.visible - 1] * 1.10),
              round(i.prices[market.visible - 1] * 0.90))
        for i in market.items
    ]


def churn(market):
    """Re-place the ORIGINAL plan at full size on every look.

    Not the same policy as re-stating the book: the units already bought release their
    reservation and sale proceeds have returned to the purse, so asking for the full size
    again spends them. It earns roughly what a single look earns off materially more peak
    capital, which is precisely the shape `efficiency` exists to catch.
    """
    orders = reference_orders(market)
    return lambda view: [Leg(o.item, o.quantity, o.buy, o.sell) for o in orders]


def restate(market):
    """Place the one-shot reference on turn 1, then re-submit the book exactly as it stands
    on every look after it.

    The pure cost of looking, isolated: same names, same prices, same units still sought,
    nothing learned, one freeze and one queue reset per turn. Its gap below `impatient` — an
    identical plan that simply stops looking — is the price of a look with no information in
    it, in reward units.
    """
    orders = reference_orders(market)

    def policy(view):
        if view.turn == 1:
            return [Leg(o.item, o.quantity, o.buy, o.sell) for o in orders]
        return [Leg(o.item, o.quantity, o.buy, o.sell) for o in view.open_orders.values()]

    return policy


def exhaustive(market):
    """Spend the entire budget: re-quote on EVERY look, off everything observed so far.

    This is the rule-4 policy. It makes every judgement the reference makes and one more
    look than the reference thinks is worth paying for, so if it scores at the oracle the
    budget is free and the environment measures nothing the one-shot form did not.
    """
    inner = Reference(requote_band=0.0)
    return inner


def perfect_hindsight(market):
    """Not shipped, not a rung: a sanity ceiling. Quotes off the mean of the WHOLE window,
    held-out ticks included. If the oracle were close to this, the visible half would not be
    the thing being read."""
    prices = {i.name: list(i.prices) for i in market.items}
    volumes = {i.name: list(i.volumes) for i in market.items}
    orders = desired_orders(market.items, prices, volumes, float(market.capital),
                            market.held_out, {i.name: i.buy_limit for i in market.items})
    return one_look(lambda m: orders)(market)


LADDER = [
    ("inaction", lambda m: inaction),
    ("staller", lambda m: staller),
    ("spammer", lambda m: spammer(m.seed)),
    ("crude (market orders, one look)", one_look(market_orders)),
    ("plausible (mean anchor, no filter)",
     one_look(lambda m: band(m, m.items, statistics.fmean, by_volume))),
    ("impatient (one-shot reference, one look)", one_look(reference_orders)),
    ("restate (re-quote the same book every look)", restate),
    ("churn (re-place at full size every look)", churn),
    ("exhaustive (re-quote every look)", exhaustive),
    ("oracle (the live reference)", lambda m: Reference()),
    ("hindsight (not a rung: whole-window anchor)", perfect_hindsight),
]


def ladder(markets, target_share: float = TARGET_SHARE) -> dict[str, dict[str, float]]:
    rows: dict[str, dict[str, float]] = {}
    for name, make in LADDER:
        totals: dict[str, float] = {}
        gp = looks = frozen = 0.0
        peak = 0.0
        for market in markets:
            result = play(market, make(market))
            for key, value in score(result, live_reference(market), target_share).items():
                totals[key] = totals.get(key, 0.0) + value
            gp += result.realised
            looks += result.looks
            frozen += result.frozen_ticks
            peak += result.peak_employed
        n = len(markets)
        rows[name] = {k: v / n for k, v in totals.items()}
        rows[name].update(gp=gp / n, looks=looks / n, frozen=frozen / n, peak=peak / n)
    return rows


def table(markets, target_share: float = TARGET_SHARE) -> None:
    rows = ladder(markets, target_share)
    print(f"{'policy':<46}{'total':>8}{'profit':>8}{'effcy':>8}{'discip':>8}{'clean':>7}"
          f"{'gp':>12}{'peak gp':>10}{'looks':>7}{'frozen':>7}")
    for name, row in rows.items():
        print(f"{name:<46}{row['total']:>8.3f}{row['profit_ratio']:>8.3f}"
              f"{row['efficiency']:>8.3f}{row['discipline']:>8.3f}{row['clean']:>7.3f}"
              f"{row['gp']:>12,.0f}{row['peak']:>10,.0f}{row['looks']:>7.2f}"
              f"{row['frozen']:>7.2f}")


def main() -> None:
    flags = set(sys.argv[1:])
    started = time.monotonic()
    seeds = baskets(BLOCK * BLOCKS)
    build = time.monotonic() - started
    blocks = [seeds[i:i + BLOCK] for i in range(0, len(seeds), BLOCK)]
    skipped = sum(1 for i, m in enumerate(seeds)
                  if m.seed != (seeds[i - 1].seed + 1 if i else SEED_BASE))

    print(f"engine     TURNS={TURNS} TICKS_PER_TURN={TICKS_PER_TURN} TAIL={TAIL_TICKS} "
          f"HORIZON={HORIZON} REQUOTE_BAND={REQUOTE_BAND:.4f} TARGET_SHARE={TARGET_SHARE}")
    print(f"baskets    {len(seeds)} over {BLOCKS} blocks of {BLOCK}, "
          f"{skipped} seeds skipped by viability (cap {LIVE_VIABILITY_TRIES}), "
          f"built in {build:.2f}s")
    print(f"freeze     {statistics.fmean([m.freeze for m in seeds]):.2f} ticks mean, "
          f"distribution {sorted({f: [m.freeze for m in seeds].count(f) for f in {m.freeze for m in seeds}}.items())}")
    print()

    print("=== the ladder, all 120 baskets ===")
    table(seeds)
    print()

    print("=== does the budget bind? exhaustive against oracle, block by block ===")
    print(f"{'block':<22}{'exhaustive':>12}{'oracle':>10}{'gap':>10}"
          f"{'exh gp':>12}{'ora gp':>12}{'exh looks':>11}{'ora looks':>11}")
    gaps = []
    for i, block in enumerate(blocks):
        rows = ladder(block)
        e, o = rows["exhaustive (re-quote every look)"], rows["oracle (the live reference)"]
        gaps.append(o["total"] - e["total"])
        print(f"{f'{block[0].seed}-{block[-1].seed}':<22}{e['total']:>12.3f}{o['total']:>10.3f}"
              f"{o['total'] - e['total']:>10.3f}{e['gp']:>12,.0f}{o['gp']:>12,.0f}"
              f"{e['looks']:>11.2f}{o['looks']:>11.2f}")
    print(f"{'mean':<22}{'':>12}{'':>10}{statistics.fmean(gaps):>10.3f}")
    print(f"{'stdev':<22}{'':>12}{'':>10}{statistics.stdev(gaps):>10.3f}")
    print(f"{'worst block':<22}{'':>12}{'':>10}{min(gaps):>10.3f}")
    print()

    print("=== every rung, block by block (total only) ===")
    per_block = [ladder(b) for b in blocks]
    print(f"{'policy':<46}" + "".join(f"{f'b{i}':>9}" for i in range(len(blocks)))
          + f"{'spread':>9}")
    for name, _ in LADDER:
        values = [rows[name]["total"] for rows in per_block]
        print(f"{name:<46}" + "".join(f"{v:>9.3f}" for v in values)
              + f"{max(values) - min(values):>9.3f}")
    print()

    if "--sweep-target" in flags:
        print("=== TARGET_SHARE sweep (the ceiling band) ===")
        names = [n for n, _ in LADDER]
        print(f"{'share':<8}" + "".join(f"{n.split(' ')[0]:>12}" for n in names))
        for share in (0.75, 0.80, 0.85, 0.90, 0.95, 1.00):
            rows = ladder(seeds, share)
            print(f"{share:<8.2f}" + "".join(f"{rows[n]['total']:>12.3f}" for n in names))
        print()

    if "--sweep-requote" in flags:
        print("=== REQUOTE_BAND sweep (what the reference should pay a freeze for) ===")
        print(f"{'band':<8}{'gp':>12}{'roc':>10}{'looks':>8}{'frozen':>8}")
        for band_value in (0.0, 0.005, 0.01, 0.015, 0.025, 0.04, 0.06, 0.10, 1.0):
            gp = roc = looks = frozen = 0.0
            for market in seeds:
                result = play(market, Reference(requote_band=band_value))
                gp += result.realised
                roc += result.roc
                looks += result.looks
                frozen += result.frozen_ticks
            n = len(seeds)
            print(f"{band_value:<8.3f}{gp / n:>12,.0f}{roc / n:>10.4f}"
                  f"{looks / n:>8.2f}{frozen / n:>8.2f}")
        print()

    if "--sweep-reference" in flags:
        # The denominator has to be the profit-maximising point of its OWN family, or some
        # other member earns more gp and scores less, and the reward's argmax is a pair of
        # constants rather than a strategy. This is the same sweep `book.py` documents for
        # the one-shot bands, over the two triggers that make the live reference act.
        print("=== reference family sweep: gp per basket ===")
        idles = (0.0, 0.05, 0.10, 0.15, 0.20, 0.30, 0.50, 1.01)
        print(f"{HEADER:<16}" + "".join(f"{i:>10.2f}" for i in idles))
        best = (0.0, None)
        for band_value in (0.0, 0.005, 0.010, 0.015, 0.025, 0.040, 0.060, 1.0):
            row = []
            for idle in idles:
                gp = sum(play(m, Reference(band_value, idle)).realised for m in seeds)
                row.append(gp / len(seeds))
                if row[-1] > best[0]:
                    best = (row[-1], (band_value, idle))
            print(f"{band_value:<16.3f}" + "".join(f"{v:>10,.0f}" for v in row))
        print(f"argmax {best[1]} at {best[0]:,.0f} gp per basket; shipped "
              f"({REQUOTE_BAND}, {IDLE_SHARE}) is "
              f"{sum(play(m, Reference()).realised for m in seeds) / len(seeds):,.0f}")
        print()

        print("=== the same family, in return on peak capital ===")
        print(f"{HEADER:<16}" + "".join(f"{i:>10.2f}" for i in idles))
        for band_value in (0.0, 0.005, 0.010, 0.015, 0.025, 0.040, 0.060, 1.0):
            row = []
            for idle in idles:
                roc = sum(play(m, Reference(band_value, idle)).roc for m in seeds)
                row.append(roc / len(seeds))
            print(f"{band_value:<16.3f}" + "".join(f"{v:>10.4f}" for v in row))
        print()

        print("=== and what each family member SCORES against the shipped reference ===")
        print(f"{HEADER:<16}" + "".join(f"{i:>10.2f}" for i in idles))
        for band_value in (0.0, 0.005, 0.010, 0.015, 0.025, 0.040, 0.060, 1.0):
            row = []
            for idle in idles:
                total = sum(score(play(m, Reference(band_value, idle)), live_reference(m))["total"]
                            for m in seeds)
                row.append(total / len(seeds))
            print(f"{band_value:<16.3f}" + "".join(f"{v:>10.3f}" for v in row))
        print()

    if "--sweep-freeze" in flags:
        # If no freeze makes looking expensive enough for the optimum to be interior, the
        # freeze is decoration and rule 4 is being carried by the denominator alone.
        print("=== freeze sweep: what a look has to cost before it is worth declining ===")
        print(f"{'freeze':<10}{'exhaust gp':>12}{'oracle gp':>12}{'exh score':>11}"
              f"{'gap':>8}{'ora looks':>11}{'ora frozen':>12}")
        for forced in (0, 1, 2, 3, 4, 6, 8, 12):
            shaped = [replace(m, freeze=forced) for m in seeds]
            e = g = 0.0
            egp = ogp = looks = frozen = 0.0
            for m in shaped:
                ref = play(m, Reference())
                exh = play(m, Reference(requote_band=0.0, idle_share=0.0))
                if ref.realised <= 0:
                    continue
                e += score(exh, ref)["total"]
                g += 1.0 - score(exh, ref)["total"]
                egp += exh.realised
                ogp += ref.realised
                looks += ref.looks
                frozen += ref.frozen_ticks
            n = len(shaped)
            print(f"{forced:<10}{egp / n:>12,.0f}{ogp / n:>12,.0f}{e / n:>11.3f}"
                  f"{g / n:>8.3f}{looks / n:>11.2f}{frozen / n:>12.2f}")
        print()

    if "--sweep-shape" in flags:
        print("=== window shape sweep: does a different budget bind harder? ===")
        print(f"{'turns x step + tail':<22}{'horizon':>9}{'exhaust':>9}{'oracle':>9}"
              f"{'gap':>9}{'ora looks':>11}{'held out':>10}")
        for turns, step, tail in ((4, 14, 4), (6, 9, 6), (8, 7, 4), (8, 5, 20),
                                  (10, 5, 10), (12, 4, 12)):
            horizon = turns * step + tail
            shaped = [build_market(m.seed, 5, 56, horizon) for m in seeds]
            e = o = 0.0
            looks = 0.0
            for market in shaped:
                ref = play(market, Reference(), turns=turns, step=step, tail=tail)
                exh = play(market, Reference(requote_band=0.0), turns=turns, step=step,
                           tail=tail)
                if ref.realised <= 0:
                    continue
                e += score(exh, ref)["total"]
                o += score(ref, ref)["total"]
                looks += ref.looks
            n = len(shaped)
            print(f"{f'{turns} x {step} + {tail}':<22}{horizon:>9}{e / n:>9.3f}{o / n:>9.3f}"
                  f"{(o - e) / n:>9.3f}{looks / n:>11.2f}{step + tail:>10}")
        print()

    print(f"total wall clock {time.monotonic() - started:.1f}s")


if __name__ == "__main__":
    main()