lumbridge-public/arena
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gates: give two of them a margin, and make the probe see components

Browse Source 
The probe printed one blended number per policy, so a component pinned at 0.000
across every rung was invisible. Four gates hid there for a month. It now reports
floor / best-below-oracle / oracle / ceiling for 25 components across 8
environments, and a component flat across every rung is fatal.

bot-detection GATE_SLACK = 1 — fires 5/32 on the real rollouts, was 0/32. The
max(1, reference_caught - SLACK) guard is verified by construction, not by
sampling: without it the required count reaches 0 and an EMPTY accusation list
clears the gate. Observed reference_caught is 4-6, so no amount of sampling would
have found that hole.

schema-migration GATE_MARGIN = 0.15 — fires 4/32, was 0/32. The cost is disclosed
and bounded: the naive split clears it on 5.5% of 1,000 unseen seeds, fenced by
an assert at 10%. Margin 0.10 keeps the leak at zero and fires 0/32, i.e. stays
dead. A live gradient with a bounded leak beats a clean corpse.

redaction-pressure is NOT given a margin, and that is the result rather than a
failure. The only setting that fires at all leaves half the secrets standing and
pays a four-of-seven ruleset on five seeds in six — a margin that pays for
inaction is strictly worse than a dead gate. Recall maxes at 0.852 and no rollout
ever cleared both clauses in one episode. It is genuinely hard, not miscalibrated.

⚠️ The per-component check did not catch the defect it was built for. Reverting
schema-migration's margin to 0.0 — restoring the exact dead gate — printed ok and
exited 0, because the near-oracle rung scrapes the unmargined gate on ~2 seeds in
24 and that kept best<oracle non-zero. Every assertion bounded how much a margin
may PAY; none noticed if it stopped existing. migration() now carries the mirror
of bot-detection's guard, and reverting the margin fails with "the margin is dead
and the component carries no gradient between the crude answer and the exact one".

canary-trap's oracle-minus-one rung is documented as degenerate rather than
quietly relied on: it is identical to the oracle to four decimals, so it measures
specificity and gate at the ceiling, not mid-ladder as its comment claimed.

The CI lock policy asks git instead of the disk. It was checking the working
tree, where a lock file is a normal by-product of uv sync, so it passed in a clean
checkout and failed on every machine that had run an eval.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
karti
2026-08-21 23:14:05 -07:00
co-authored by Claude Opus 5
parent 122a858b61
commit 3305be4ff7
12 changed files with 1107 additions and 95 deletions
Download Patch File Download Diff File Expand all files Collapse all files
probe.py
+322 -76
View File
@@ -72,8 +72,21 @@ the first time and this is what caught it:
held-out ticks and the screen: a policy that read them earned 9.4% over
the oracle, and the sentinel scan below is what holds that shut.
Exit code is the API: 1 if any environment's floor is above zero or its ceiling is
unreachable. Nothing here is listed by hand — the environments are `environments/*/pyproject.toml`
And then the check none of that caught, because none of it was per-term. Every probe
returned ONE blended number per rung, so `bot-detection`, `schema-migration`,
`redaction-pressure` and `grand-exchange` all printed `oracle 1.000` and `ok` while their
`gate` component scored exactly 0.000 mean and 0.000 max over 32 real rollouts each — a
quarter to a third of three environments' reward mass, constant, for a month. Three more
terms sat pinned at exactly 1.0000 on all 32 in the other direction. A blend is precisely
the thing that cannot show a constant inside it. Probes return the WEIGHTED components now
and `components()` prints each one's floor, its best rung below the oracle, and its ceiling;
a term that is the same on every rung is fatal, and a term nothing below the oracle earns a
fraction of is named. `tools/regate.py` is the other half — it replays real traces through a
candidate gate, which is the only thing that can tell a dead gate from a hard one.
Exit code is the API: 1 if any environment's floor is above zero, its ceiling is
unreachable, or one of its reward components is constant across the whole ladder. Nothing
here is listed by hand — the environments are `environments/*/pyproject.toml`
and each probe registers itself with `@probes(taskset_id)`, so adding one is a function appended
at the end of this file. A manifest with no probe is a warning today and an error the day the
last one is written.
@@ -164,20 +177,50 @@ def probes(name: str):
TASKS = 24
def rung(items, score) -> dict[str, float]:
"""One ladder rung: the mean of each WEIGHTED reward component over a rung's tasks.
Every probe used to return one blended float per rung, and that single number is how
four dead reward components survived a month in this repository. `bot-detection`,
`schema-migration`, `redaction-pressure` and `grand-exchange` all printed `oracle 1.000`
while their `gate` term scored exactly 0.000 on every one of 32 real rollouts — because
1.000 requires the gate, so the ceiling was genuinely reachable and rule 3 genuinely
passed, and nothing in the output was per-term. `main()` sums these back into the same
blended number for the table and then reports the terms underneath it.
The values are weighted, deliberately: an unweighted component reads 1.000 whether it
carries a thirtieth of the reward or half of it, and what matters here is how much
reward mass has no gradient in it, not how a term looks after normalisation.
"""
totals: dict[str, float] = {}
count = 0
for item in items:
for name, value in score(item).items():
totals[name] = totals.get(name, 0.0) + value
count += 1
if not count:
raise RuntimeError("a probe rung graded no tasks at all")
return {name: total / count for name, total in totals.items()}
@probes("redaction-pressure")
def redaction() -> dict[str, float]:
def redaction() -> dict[str, dict[str, float]]:
from redaction_pressure.corpus import build_slices
from redaction_pressure.scan import measure, parse_rules
def run(rules: list[dict]) -> float:
total = 0.0
for i in range(TASKS):
def run(rules: list[dict]) -> dict[str, float]:
reply = "```json\n" + json.dumps(rules) + "\n```"
def score(i: int) -> dict[str, float]:
_, held = build_slices(200_000 + i, 4, 12)
o = measure(held, parse_rules("```json\n" + json.dumps(rules) + "\n```"))
total += o.validity * (
0.35 * o.recall + 0.35 * o.precision + 0.30 * float(o.clean)
)
return total / TASKS
o = measure(held, parse_rules(reply))
return {
"recall": 0.35 * o.validity * o.recall,
"precision": 0.35 * o.validity * o.precision,
"gate": 0.30 * o.validity * float(o.clean),
}
return rung(range(TASKS), score)
oracle = [
{"pattern": r"(?<=Customer pasted their key )sk-live-[a-z0-9]{24}", "replacement": "[REDACTED]"},
@@ -188,52 +231,75 @@ def redaction() -> dict[str, float]:
{"pattern": r"(?<=Session )[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}", "replacement": "[REDACTED]"},
{"pattern": r"(?<=Escalated by )(?:Dana|Priya|Marcus|Ines|Tomas|Rui|Nadia|Owen) (?:Okafor|Lindqvist|Baptiste|Moreau|Sato|Ahmed|Vance|Duarte)", "replacement": "[REDACTED]"},
]
# `six-of-seven` is not decoration and it is not a fifth adjective. It is the rung that
# makes this environment's dead `gate` VISIBLE: a ruleset that finds six of the seven
# secret types earns 0.000 of it, exactly what an empty ruleset earns, so the term is a
# step function whose only step is the oracle. `main()` names it. It was swept for a
# margin and deliberately left exact — see `Outcome.clean` in the scanner for the table.
return {
"inaction": run([]),
"crude": run([{"pattern": r"\S+", "replacement": "[REDACTED]"}]),
"plausible": run(oracle[:4]),
"six-of-seven": run(oracle[:6]),
"oracle": run(oracle),
}
@probes("canary-trap")
def canary() -> dict[str, float]:
def canary() -> dict[str, dict[str, float]]:
from canary_trap.corpus import SUBJECTS, build
from canary_trap.scan import measure, parse_probes
core = {f.subject: f.core for f in SUBJECTS}
pat = re.compile(r"Record for ([^:]+): [a-z ]+is ([^.]+)\.")
def run(fn) -> float:
total = 0.0
for i in range(TASKS):
def run(fn) -> dict[str, float]:
def score(i: int) -> dict[str, float]:
corpus, states = build(70_000 + i, 4)
o = measure(states, parse_probes("```json\n" + json.dumps(fn(corpus)) + "\n```"))
total += 0.35 * o.detection + 0.35 * o.specificity + 0.30 * float(o.clean_gate)
return total / TASKS
return {
"detection": 0.35 * o.detection,
"specificity": 0.35 * o.specificity,
"gate": 0.30 * float(o.clean_gate),
}
return rung(range(TASKS), score)
full = lambda c: [{"question": f"{s}?", "answer": core[s]} for s, _ in pat.findall(c)]
return {
"inaction": run(lambda c: []),
"crude": run(lambda c: [{"question": "?", "answer": "three-way"}, {"question": "?", "answer": "5432"}]),
"plausible": run(lambda c: [{"question": "id?", "answer": t} for t in re.findall(r"\[([0-9a-f-]+)\]", c)]),
"oracle": run(lambda c: [{"question": f"{s}?", "answer": core[s]} for s, _ in pat.findall(c)]),
# The oracle's probe set less one fact. ⚠️ It was ADDED as "a rung between `plausible`
# and the ceiling, so `specificity` and `gate` are measured somewhere other than at
# the two ends", and it does not do that: measured per component it is
# detection 0.3500 / gate 0.3000 / specificity 0.3500 / total 1.0000 — identical to
# the oracle row to four decimals. Dropping one fact from a probe set this size costs
# nothing, so this rung sits AT the ceiling, and it is the only thing keeping
# `canary-trap/gate` off the step@oracle list below. It is kept because removing it
# would flag a gate that genuinely fires 7/32 on real rollouts, and it is documented
# rather than quietly relied on: a real mid-ladder rung here needs a perturbation that
# actually costs specificity, and nobody has designed one yet.
"oracle-minus-one": run(lambda c: full(c)[:-1]),
"oracle": run(full),
}
@probes("fault-localisation")
def fault() -> dict[str, float]:
def fault() -> dict[str, dict[str, float]]:
from fault_localisation.incident import build, loudest
def run(fn) -> float:
total = 0.0
for i in range(TASKS):
def run(fn) -> dict[str, float]:
def score(i: int) -> dict[str, float]:
inc = build(50_000 + i)
a = fn(inc)
s = float(a.get("service", "") == inc.root)
f = float(a.get("fault", "") == inc.fault)
e = float(a.get("evidence", "") == inc.evidence)
total += 0.30 * s + 0.25 * f + 0.25 * e + 0.20 * float(s and f and e)
return total / TASKS
return {"service": 0.30 * s, "fault": 0.25 * f, "evidence": 0.25 * e,
"gate": 0.20 * float(s and f and e)}
return rung(range(TASKS), score)
# The environment's own premise: blaming the loudest service must never be right.
assert all(loudest(build(50_000 + i)) != build(50_000 + i).root for i in range(TASKS)), \
@@ -242,20 +308,26 @@ def fault() -> dict[str, float]:
"inaction": run(lambda inc: {}),
"crude": run(lambda inc: {"service": loudest(inc), "fault": inc.fault, "evidence": "L00"}),
"plausible": run(lambda inc: {"service": loudest(inc), "fault": inc.fault, "evidence": inc.evidence}),
# Right about the two hard fields, wrong about the log line. The rung that separates
# "the gate is binary" from "the gate only ever pays a byte-perfect answer".
"oracle-minus-evidence": run(lambda inc: {"service": inc.root, "fault": inc.fault,
"evidence": "L00"}),
"oracle": run(lambda inc: {"service": inc.root, "fault": inc.fault, "evidence": inc.evidence}),
}
@probes("schema-migration")
def migration() -> dict[str, float]:
def migration() -> dict[str, dict[str, float]]:
from schema_migration.run import measure
def run(sql: str) -> float:
total = 0.0
for i in range(TASKS):
def run(sql: str) -> dict[str, float]:
def score(i: int) -> dict[str, float]:
o = measure(30_000 + i, 40, sql)
total += 0.30 * float(o.schema_ok) + 0.45 * (o.fidelity * o.rows_kept) + 0.25 * float(o.clean)
return total / TASKS
return {"schema": 0.30 * float(o.schema_ok),
"integrity": 0.45 * o.fidelity * o.rows_kept,
"gate": 0.25 * float(o.clean)}
return rung(range(TASKS), score)
naive = ("ALTER TABLE readings ADD COLUMN value_num REAL;"
"ALTER TABLE readings ADD COLUMN unit TEXT;"
@@ -271,32 +343,79 @@ def migration() -> dict[str, float]:
f" unit=CASE WHEN instr({clean},' ')>0"
f" THEN trim(substr({clean},instr({clean},' ')+1)) ELSE '' END;"
f"ALTER TABLE readings DROP COLUMN value_text;")
return {
# The rung `GATE_MARGIN` was chosen against: it strips the thousands separator — the
# inference the environment exists to teach, and the one `naive` skips — and still
# mishandles the row that carries no unit at all. Right about the hard thing, wrong
# about one row shape in eleven. Its fidelity runs 0.825-1.000 against `naive`'s
# 0.575-0.850, and under the shipped margin it clears the gate on 23 of 24 seeds where
# an exact gate cleared it on 2.
near = (f"ALTER TABLE readings ADD COLUMN value_num REAL;"
f"ALTER TABLE readings ADD COLUMN unit TEXT;"
f"UPDATE readings SET"
f" value_num=CAST(substr({clean},1,instr({clean},' ')-1) AS REAL),"
f" unit=substr({clean},instr({clean},' ')+1);"
f"ALTER TABLE readings DROP COLUMN value_text;")
# A migration that recomposes perfectly over no rows at all. The row clause is what
# stops it, and it carries no margin for exactly this reason — so the rung is here to
# hold that shut rather than to be believed about.
wipe = oracle + "DELETE FROM readings;"
rows = {
"inaction": run(""),
"crude": run("ALTER TABLE readings ADD COLUMN value_num REAL; ALTER TABLE readings ADD COLUMN unit TEXT;"),
"plausible": run(naive),
"wipe": run(wipe),
"near-oracle": run(near),
"oracle": run(oracle),
}
assert rows["wipe"]["gate"] <= 1e-9, \
f"wiping the table clears the gate at {rows['wipe']['gate']:.3f} — the row clause " \
"has acquired a margin it must never have"
# The margin's price, bounded rather than merely noticed. `naive` is the parser fitted
# to the five tidy rows on screen; its luckiest draw of forty awkward rows recomposes 34
# of them, which is 0.850, which is the gate's threshold to the digit. It clears on one
# seed in twenty-four today. If that ever becomes a fifth, the margin has stopped being
# a margin and started paying for the shortcut.
assert rows["plausible"]["gate"] < 0.10 * 0.25, \
f"the naive split earns {rows['plausible']['gate'] / 0.25:.3f} of the gate — the " \
"margin is paying the strategy the environment exists to punish"
# And the margin bounded from BELOW, which is the half that was missing. Every assertion
# above bounds how much the margin may PAY; none of them noticed if it stopped existing.
# Reverting GATE_MARGIN to 0.0 — restoring the exact dead gate that fires 0/32 on real
# rollouts, the defect this margin was added to fix — printed `ok` and exited 0, because
# `near-oracle` still scrapes the unmargined gate on about two seeds in twenty-four and
# that is enough to keep `best < oracle` non-zero and the step@oracle check silent. A
# majority bar cannot be scraped: near-oracle clears the shipped margin on 983 seeds in
# 1,000 and the unmargined gate on roughly 21 in 1,000. `bot-detection` has carried the
# equivalent guard since it shipped; this is its mirror.
assert rows["near-oracle"]["gate"] > 0.5 * 0.25, \
f"a migration one clause short of the reference earns " \
f"{rows['near-oracle']['gate'] / 0.25:.3f} of the gate — the margin is dead and " \
"the component carries no gradient between the crude answer and the exact one"
return rows
@probes("bot-detection")
def bots() -> dict[str, float]:
def bots() -> dict[str, dict[str, float]]:
from bot_detection.accounts import build_slices
from bot_detection.scan import (
click_cv, correction_share, distinct_routes, log_click_sd, measure,
parse_accusations, reference_bots, session_mean,
)
def run(fn, count: int = TASKS) -> float:
total = 0.0
for i in range(count):
def run(fn, count: int = TASKS) -> dict[str, float]:
def score(i: int) -> dict[str, float]:
_, batch = build_slices(80_000 + i, 12)
reply = "```json\n" + json.dumps({"bots": sorted(fn(batch))}) + "\n```"
o = measure(batch, parse_accusations(reply))
total += (0.35 * o.detection * o.restraint
+ 0.40 * o.restraint * o.purity
+ 0.25 * float(o.gate))
return total / count
return {"caught": 0.35 * o.detection * o.restraint,
"spared": 0.40 * o.restraint * o.purity,
"gate": 0.25 * float(o.gate)}
return rung(range(count), score)
def total(row: dict[str, float]) -> float:
return sum(row.values())
plausible = lambda b: [a.acc_id for a in b if correction_share(a) < 0.35]
@@ -365,7 +484,8 @@ def bots() -> dict[str, float]:
# combine into an attack. Each of these scored above 0.87 on the first version; all of
# them must now come in under the honest single-channel attempt, or the shortcut is
# still the better strategy and a model will learn it instead of the task.
ceiling = run(plausible)
plausible_row = run(plausible)
ceiling = total(plausible_row)
for name, fn in (
("routes veto + sd(log dt) band",
lambda b: [a.acc_id for a in b
@@ -377,7 +497,7 @@ def bots() -> dict[str, float]:
lambda b: [a.acc_id for a in b
if not a.break_s or (session_mean(a) >= 240 and click_cv(a) >= 0.54)]),
):
got = run(fn)
got = total(run(fn))
assert got < ceiling, \
f"decoy strategy '{name}' scores {got:.3f} against the intended {ceiling:.3f}"
@@ -393,16 +513,36 @@ def bots() -> dict[str, float]:
for i in range(TASKS)
), "a batch holds no efficient human — the false positive is free to avoid"
return {
# The two rungs `GATE_SLACK` was chosen between, and they are load-bearing rather than
# illustrative. The gate used to demand every bot the reference caught and fired on 0 of
# 32 real rollouts; with one bot of slack it fires on 5. What stops the slack going to
# two is right here: at one, the oracle's list minus a single account clears the gate and
# minus two does not, so the gate still separates a near miss from a half-right answer.
# At two, both clear and it separates nothing. Neither rung may ever OUT-earn the oracle,
# which is the standing gate and is asserted below rather than hoped for.
rows = {
"inaction": run(lambda b: []),
"crude": run(lambda b: [a.acc_id for a in b]),
"plausible": ceiling,
"plausible": plausible_row,
"oracle-minus-two": run(lambda b: sorted(reference_bots(b))[:-2]),
"oracle-minus-one": run(lambda b: sorted(reference_bots(b))[:-1]),
"oracle": run(lambda b: sorted(reference_bots(b))),
}
assert rows["oracle-minus-two"]["gate"] <= 1e-9, \
f"the reference's list less two accounts clears the gate at " \
f"{rows['oracle-minus-two']['gate'] / 0.25:.3f} — GATE_SLACK has gone past the " \
"point where the gate distinguishes a near miss from a half-right answer"
assert 0 < rows["oracle-minus-one"]["gate"] <= rows["oracle"]["gate"], \
"the reference's list less one account does not clear the gate — the margin is dead"
for name in ("oracle-minus-one", "oracle-minus-two", "plausible"):
assert sum(rows[name].values()) < sum(rows["oracle"].values()), \
f"'{name}' scores {sum(rows[name].values()):.3f} against the oracle's " \
f"{sum(rows['oracle'].values()):.3f} — something below the oracle out-earns it"
return rows
@probes("grand-exchange")
def exchange() -> dict[str, float]:
def exchange() -> dict[str, dict[str, float]]:
from grand_exchange.book import (
BUY_BAND, FILL_SHARE, MAX_ITEM_SHARE, MAX_ORDERS, MIN_CROSSINGS, SELL_BAND, TAX,
Order, crossings, execute, measure, paper_profit, parse_orders, reference_orders,
@@ -505,16 +645,22 @@ def exchange() -> dict[str, float]:
}
return "```json\n" + json.dumps(body) + "\n```"
def score(outcome) -> float:
return 0.45 * outcome.profit_ratio + 0.30 * outcome.discipline + 0.25 * float(outcome.clean)
def score(outcome) -> dict[str, float]:
return {"profit": 0.45 * outcome.profit_ratio,
"discipline": 0.30 * outcome.discipline,
"gate": 0.25 * float(outcome.clean)}
def run(fn, markets=None) -> float:
def run(fn, markets=None) -> dict[str, float]:
markets = LADDER if markets is None else markets
total = 0.0
for market in markets:
def one(market) -> dict[str, float]:
orders, stated = parse_orders(fn(market))
total += score(measure(market, orders, stated))
return total / len(markets)
return score(measure(market, orders, stated))
return rung(markets, one)
def total(row: dict[str, float]) -> float:
return sum(row.values())
def gp(fn, markets=None) -> float:
markets = LADDER if markets is None else markets
@@ -566,7 +712,8 @@ def exchange() -> dict[str, float]:
# the history, whatever that plan does about size, and it must do so in EVERY block of
# twenty-four, not on average: an average holds while one block of the size a run
# actually grades goes the other way, and that block is where the shortcut gets learned.
ceiling = run(plausible)
plausible_row = run(plausible)
ceiling = total(plausible_row)
for name, fn in (
("last-tick anchor, no filter", shortcut),
("last-tick anchor, filtered and ranked",
@@ -574,7 +721,7 @@ def exchange() -> dict[str, float]:
("last-tick anchor, whole buy limit", lambda m: reply(band(m, m.items, last, by_limit))),
):
for block in BLOCKS:
got, bar = run(fn, block), run(plausible, block)
got, bar = total(run(fn, block)), total(run(plausible, block))
assert got < bar, \
f"shortcut '{name}' scores {got:.3f} against {bar:.3f} on baskets " \
f"{block[0].seed}-{block[-1].seed} — the anchor is not worth estimating"
@@ -600,7 +747,7 @@ def exchange() -> dict[str, float]:
if earned > best:
best, best_cell = earned, (bb, sb)
if earned > gp(reference_orders):
got = run(lambda m, c=cell: reply(c(m)))
got = total(run(lambda m, c=cell: reply(c(m))))
assert got >= 0.90, \
f"band ({bb}, {sb}) earns {earned:,.0f} gp against the reference's " \
f"{gp(reference_orders):,.0f} and scores {got:.3f} — the reward is an " \
@@ -640,10 +787,12 @@ def exchange() -> dict[str, float]:
return reply(orders)
return fn
perturbations: dict[str, dict[str, float]] = {}
for kind, floor in (("one wasted unit", 0.999), ("first order 1 smaller", 0.95),
("first order 1 larger", 0.95), ("every buy limit +1", 0.95),
("every sell limit -1", 0.95)):
got = run(perturbed(kind))
perturbations[kind] = run(perturbed(kind))
got = total(perturbations[kind])
assert got >= floor, \
f"perturbation '{kind}' scores {got:.3f} against a floor of {floor} — the gate " \
"is a knife-edge at the reference rather than a bar a good run clears"
@@ -667,21 +816,27 @@ def exchange() -> dict[str, float]:
"```json\n" + "[" * 20_000 + "]" * 20_000 + "\n```",
):
orders, stated = parse_orders(hostile)
got = score(measure(LADDER[0], orders, stated))
got = sum(score(measure(LADDER[0], orders, stated)).values())
assert got <= 1e-9, f"hostile reply {hostile[:40]!r} scores {got:.3f}, not zero"
assert stated is None or math.isfinite(stated), \
f"hostile reply {hostile[:40]!r} put a non-finite number in the trace"
# The near rung costs nothing: `perturbations` already holds it. Shrinking the first
# order by a single unit is the smallest departure from the reference this file knows
# how to make, and TARGET_SHARE = 0.90 is what lets it still clear the gate. That is
# what a band looks like from the outside, and it is the shape the other three gates
# were argued against.
return {
"inaction": run(lambda m: "No trades today."),
"crude": run(lambda m: reply(market_order(m))),
"plausible": ceiling,
"plausible": plausible_row,
"near-oracle": perturbations["first order 1 smaller"],
"oracle": run(oracle),
}
@probes("drop-table-inference")
def drops() -> dict[str, float]:
def drops() -> dict[str, dict[str, float]]:
from drop_table_inference.estimate import (
_tail_loss, measure, parse_estimate, reference_estimate,
)
@@ -698,14 +853,17 @@ def drops() -> dict[str, float]:
GRID = ([n / COMMON_DENOMINATOR for n in COMMON_NUMERATORS]
+ [1.0 / d for d in LADDER_DENOMINATORS])
def run(fn) -> float:
total = 0.0
for stream in streams:
def parts_of(fn) -> dict[str, float]:
def one(stream) -> dict[str, float]:
estimate = fn(stream)
reply = "" if estimate is None else "```json\n" + json.dumps(estimate) + "\n```"
o = measure(stream, parse_estimate(reply, stream.table.items))
total += 0.45 * o.fit + 0.35 * o.rare + 0.20 * float(o.clean)
return total / TASKS
return {"fit": 0.45 * o.fit, "rare": 0.35 * o.rare, "gate": 0.20 * float(o.clean)}
return rung(streams, one)
def run(fn) -> float:
return sum(parts_of(fn).values())
def parts(fn) -> tuple[float, float, float]:
"""raw_fit, raw_rare and restraint, averaged. The rewards are products of these, and
@@ -780,8 +938,11 @@ def drops() -> dict[str, float]:
# FENCE 2. Frequency-copying is what `rare` exists to catch, and a memorised floor is
# what un-caught it. Both have to stay well under the honest attempt that reads the
# structure, or the environment teaches the hack.
honest = run(grid)
crude = max(run(floored(c)) for c in (2e-5, 5e-5, 1e-4, 2e-4, 4e-4, 8e-4))
honest_parts = parts_of(grid)
honest = sum(honest_parts.values())
crude_parts = max((parts_of(floored(c)) for c in (2e-5, 5e-5, 1e-4, 2e-4, 4e-4, 8e-4)),
key=lambda row: sum(row.values()))
crude = sum(crude_parts.values())
assert crude < 0.6 * honest, \
f"frequency-copying with a memorised floor scores {crude:.3f} against {honest:.3f} for reading the table"
@@ -816,16 +977,23 @@ def drops() -> dict[str, float]:
assert parts(frequency)[2] > 0.99 and run(frequency) < 0.12, \
"restraint is not free to the frequency copy — it is being scored as accuracy, not as restraint"
# The near rung. `gate` here already carries a margin — `GATE_MARGIN = 0.05`, three
# sub-scores each within five percent of the reference — and it is the only one of the
# four that fired in the first eval, on 22.6% of rollouts. Scaling the reference by two
# percent is what shows that from the ladder rather than from the changelog: it is not
# the reference, and it still clears.
nudged = lambda s: {i: 1.02 * v for i, v in reference(s).items()}
return {
"inaction": run(lambda s: None),
"crude": crude,
"plausible": honest,
"oracle": run(reference),
"inaction": parts_of(lambda s: None),
"crude": crude_parts,
"plausible": honest_parts,
"near-oracle": parts_of(nudged),
"oracle": parts_of(reference),
}
@probes("grand-exchange-live")
def exchange_live() -> dict[str, float]:
def exchange_live() -> dict[str, dict[str, float]]:
"""The stepped form. Rules 3 and 4, and the two ways this one could be quietly wrong.
The ladder is not re-derived here. `environments/grand_exchange_live/measure_ladder.py`
@@ -970,11 +1138,24 @@ def exchange_live() -> dict[str, float]:
f"'{lower}' scores {rows[lower]['total']:.3f} against '{upper}' at " \
f"{rows[upper]['total']:.3f} — the ladder is upside down"
# `measure_ladder.ladder` already scored every policy in the family, so the extra rungs
# here are free: the component block gets six places to look at `gate` rather than the
# two ends, and `exhaustive` — which spends the whole turn budget — is the one that says
# whether a 0.90 band pays for burning it.
def parts(name: str) -> dict[str, float]:
row = rows[name]
return {"profit": 0.45 * row["profit_ratio"],
"discipline": 0.30 * row["discipline"],
"gate": 0.25 * row["clean"]}
return {
"inaction": rows["inaction"]["total"],
"crude": rows["crude (market orders, one look)"]["total"],
"plausible": rows["plausible (mean anchor, no filter)"]["total"],
"oracle": rows["oracle (the live reference)"]["total"],
"inaction": parts("inaction"),
"crude": parts("crude (market orders, one look)"),
"plausible": parts("plausible (mean anchor, no filter)"),
"impatient": parts("impatient (one-shot reference, one look)"),
"restate": parts("restate (re-quote the same book every look)"),
"exhaustive": parts("exhaustive (re-quote every look)"),
"oracle": parts("oracle (the live reference)"),
}
@@ -1006,14 +1187,79 @@ def main() -> int:
file=sys.stderr)
results = {name: _PROBES[name]() for name in sorted(discovered) if name in _PROBES}
totals = {name: {r: sum(row.values()) for r, row in rows.items()}
for name, rows in results.items()}
print(f"{'environment':22}{'inaction':>10}{'crude':>9}{'plausible':>11}{'oracle':>9} verdict")
failed = False
for name, r in results.items():
for name, r in totals.items():
for required in CANONICAL:
if required not in r:
print(f"{name} has no '{required}' rung — the ladder is not a ladder",
file=sys.stderr)
return 1
ok = r["inaction"] <= 1e-9 and r["oracle"] >= 1.0 - 1e-9
failed |= not ok
print(f"{name:22}{r['inaction']:10.3f}{r['crude']:9.3f}{r['plausible']:11.3f}"
f"{r['oracle']:9.3f} {'ok' if ok else 'FAILS RULE 3'}")
return 1 if failed else 0
return 1 if components(results) or failed else 0
CANONICAL = ("inaction", "crude", "plausible", "oracle")
def components(results: dict[str, dict[str, dict[str, float]]]) -> bool:
"""Per-component floors and ceilings, and the two verdicts that are worth having.
The table above is a blend. Four reward components in this repository scored exactly
0.000 mean and 0.000 max over 32 real rollouts each — a quarter to a third of three
environments' reward mass with no gradient in it — and three more sat pinned at exactly
1.0000 on all 32. Every one of them printed `oracle 1.000` and `ok` for a month, because
a blend is exactly the thing that cannot show you a constant inside it.
Two lines are drawn here and they forbid different things.
**flat** is fatal. A component with the same weighted value on every rung of the ladder,
the oracle included, is a constant added to every policy's score. It cannot be climbed,
it cannot be lost, and in training it contributes nothing at all. There is no legitimate
reward term of this shape, so it exits 1.
**step@oracle** warns and names the component. Nothing below the oracle earns a fraction
of it: the floor and the best any non-oracle rung manages are the same number. That is
the exact shape of the four dead gates — and it is ALSO the honest shape of a genuinely
binary check that only a correct answer clears, which is why it is not fatal. The two are
not distinguishable from the ladder, and pretending otherwise would either forbid every
binary gate or forgive every dead one. What distinguishes them is the fire rate over real
rollouts, which this file cannot see and `tools/regate.py` measures.
So the warning is an instruction rather than a verdict: go and replay the traces. When
the answer comes back 0/32, the component is dead and the margin belongs in the scorer —
`bot_detection.GATE_SLACK` and `schema_migration.GATE_MARGIN` are two that came back
that way. When it comes back 5/32 or 22.6%, the gate is a bar and the ladder simply has
no rung standing near it; add one, as `near-oracle` does for four environments here.
"""
print(f"\n {'environment':22}{'component':13}{'floor':>8}{'best<oracle':>13}"
f"{'oracle':>9}{'ceiling':>9} verdict")
fatal = False
for name, rows in results.items():
for part in sorted(next(iter(rows.values()))):
values = {label: row[part] for label, row in rows.items()}
floor, ceiling = min(values.values()), max(values.values())
oracle = values["oracle"]
below = max(v for label, v in values.items() if label != "oracle")
if ceiling - floor <= 1e-9:
verdict, fatal = "FLAT — no gradient in any direction", True
elif below - floor <= 1e-9:
verdict = "step@oracle — nothing below the oracle earns any of it"
else:
verdict = "ok"
print(f" {name:22}{part:13}{floor:8.3f}{below:13.3f}{oracle:9.3f}"
f"{ceiling:9.3f} {verdict}")
if fatal:
print("a reward component is constant across the whole ladder — it is not a reward",
file=sys.stderr)
return fatal
if __name__ == "__main__":