Three OSRS environments, and the first real model scores
grand-exchange, bot-detection and drop-table-inference. All three produce unbounded or irreducible-error scores, unlike the first four, so all three normalise against a REFERENCE STRATEGY rather than an absolute -- which is what makes 1.000 reachable rather than aspirational. drop-table-inference is the clearest case. Scoring KL against the true drop table would put the ceiling out of reach, because sampling error is irreducible: ground truth scores 0.900 against the Bayesian posterior's 1.000, and that ordering is correct. The best estimate available from 1,200 kills is not the true table, and an environment that demands it is measuring luck. grand-exchange needed the market to carry structure a model can actually infer, or profit is noise and no oracle exists. Measured over 120 baskets: the reference earns 38,821 gp and is profitable in 120/120, random orders lose 561, and trading only the random-walk items -- which look like the widest-swinging lines on the board -- loses 16,860. bot-detection generates naive bots, cloaked bots that jitter on purpose, and efficient humans who look bot-like on every naive statistic. Timing features are drawn BEFORE the generator decides who is a bot, so every latency rule sits at chance. Reference discriminator F1 1.000, random 0.430. Each was built by one agent then attacked by two independent reviewers on exploitability and soundness. They earned their keep: drop-table-inference's first reward let a memorised constant score 0.901 without opening the kill log, and its reference was 19% worse than a flat number. probe.py now carries three named fences asserting those attacks stay dead. First evaluation also lands. Nemotron 3.5 Lightning, thinking off, 32 rollouts per environment: 0.570 down to 0.059, every gate at or near zero, nothing solved and nothing unsolvable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,3 @@
|
||||
from bot_detection.taskset import BotTaskset
|
||||
|
||||
__all__ = ["BotTaskset"]
|
||||
@@ -0,0 +1,367 @@
|
||||
"""Generating one batch of accounts: a cover drawn before anybody knows who is a bot.
|
||||
|
||||
The naive discriminator for this task is "low timing variance means bot". An environment
|
||||
that only generates metronomic scripts teaches that rule and nothing else, so the cover
|
||||
statistics — how fast an account clicks, how long it plays, how far it wanders — are drawn
|
||||
BEFORE the population is assigned, and the assignment is a uniform coin the style never
|
||||
sees. That ordering is the whole design. The first version of this file drew the cover per
|
||||
population instead, and gave a grinder `sigma ~ U(0.30, 0.45)` where a cloaked bot got
|
||||
`U(0.65, 0.90)`; the two ranges do not touch, so `sd(log click_dt_ms)` cut cleanly at 0.52
|
||||
and a two-line rule that never opened `misclick_next`, `break_s` or `login_gap_min` scored
|
||||
0.953 against the oracle's 1.000. Session length and route count were worse than that: they
|
||||
were EXACT labels — every account with four or more routes was a casual human, all 7200 of
|
||||
them — so half the humans in every batch were free and the false positive the environment
|
||||
is built around cost nothing to avoid.
|
||||
|
||||
Drawing the style first makes P(bot | any statistic of the style) exactly the batch's bot
|
||||
rate, for every statistic, with no distributional argument required and nothing to drift.
|
||||
There are three populations:
|
||||
|
||||
human a person. Long-tailed click latency, breaks that cluster into micro-pauses and
|
||||
long absences, corrects most misclicks, logs in when they are free.
|
||||
scripted a naive bot. Near-constant delays, no breaks at all, one route, never corrects
|
||||
anything. Caught by everything, and meant to be — it is the floor of the task,
|
||||
not the task. A click-variance rule finds these and nothing else, which is
|
||||
worth about a quarter of the ceiling.
|
||||
cloaked a sophisticated bot wearing a style drawn from the same pool a human's came
|
||||
from. It leaks on exactly ONE of three behavioural channels, chosen per
|
||||
account:
|
||||
|
||||
uncorrected it never corrects a misclick. A person does.
|
||||
memoryless its breaks are exponential, so they land in the 3-8 minute dead
|
||||
band that a human's bimodal breaks never occupy.
|
||||
metronomic it logs in on a fixed period, so every gap between session
|
||||
starts is the same to within a couple of minutes.
|
||||
|
||||
Exactly one leak per cloaked account is the difficulty: a discriminator built on any single
|
||||
channel catches a third of them, and the labelled slice shows one of each so the three are
|
||||
learnable rather than guessable.
|
||||
|
||||
Every separation is guaranteed at generation time rather than hoped for — the resample
|
||||
loops below are why — and so is every NON-separation, which is what the style-first
|
||||
ordering buys. A ceiling that is unreachable on one seed in two hundred is still an
|
||||
unreachable ceiling, and house rule 3 is measured, not assumed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import random
|
||||
from dataclasses import dataclass
|
||||
|
||||
CLICKS = 32
|
||||
BREAKS = 12
|
||||
SESSIONS = 4
|
||||
MISCLICKS = 8
|
||||
TRAVERSALS = 20
|
||||
|
||||
# Seconds. Human breaks are bimodal — a micro-pause or a real absence — so nothing of
|
||||
# theirs lands in here. An exponential break generator fills it, which is the whole tell.
|
||||
DEAD_BAND = (180, 480)
|
||||
|
||||
BOT_POPULATIONS = ("scripted", "cloaked")
|
||||
TELLS = ("uncorrected", "memoryless", "metronomic")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Style:
|
||||
"""The cover an account wears, drawn before it is decided whether it is a person.
|
||||
|
||||
`efficient` is the corner every naive flagger points at: tight fast clicking, one or
|
||||
two memorised routes, sessions measured in hours. Half the styles in a batch are drawn
|
||||
there and the population coin does not know which, so an efficient account is exactly
|
||||
as likely to be a bot as a rambling one. The two corners OVERLAP on all three
|
||||
statistics as well, so there is not even a clean latent to recover.
|
||||
"""
|
||||
|
||||
efficient: bool
|
||||
click_median: int
|
||||
click_sigma: float
|
||||
session_min: list[int]
|
||||
routes: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Account:
|
||||
"""One account's activity log. Every field is raw per-event data rather than a summary,
|
||||
because the summary IS the answer — handing over a variance or a correction rate would
|
||||
leave nothing to find."""
|
||||
|
||||
acc_id: str
|
||||
population: str
|
||||
tell: str
|
||||
"""Which channel a cloaked account leaks on; empty for everyone else."""
|
||||
style: str
|
||||
"""`efficient`, `casual`, or `constant` for a naive script. Never rendered — it is here
|
||||
so the probe can assert it carries no information about `is_bot`."""
|
||||
click_dt_ms: list[int]
|
||||
break_s: list[int]
|
||||
session_min: list[int]
|
||||
login_gap_min: list[int]
|
||||
"""Minutes between consecutive session starts."""
|
||||
misclick_next: list[str]
|
||||
"""What the account did after each misclick: `fix` or `ignore`."""
|
||||
route_ids: list[str]
|
||||
|
||||
@property
|
||||
def is_bot(self) -> bool:
|
||||
return self.population in BOT_POPULATIONS
|
||||
|
||||
|
||||
def _style(rng: random.Random, efficient: bool) -> Style:
|
||||
"""One cover. The ranges deliberately overlap between the two corners — 520-620 ms of
|
||||
median, 0.50-0.60 of sigma, 210-260 minutes of session, two or three routes — so that
|
||||
even the corner itself is not cleanly recoverable, let alone the population behind it."""
|
||||
if efficient:
|
||||
return Style(
|
||||
True,
|
||||
rng.randint(360, 620),
|
||||
rng.uniform(0.28, 0.60),
|
||||
[rng.randint(210, 520) for _ in range(SESSIONS)],
|
||||
rng.randint(1, 3),
|
||||
)
|
||||
return Style(
|
||||
False,
|
||||
rng.randint(520, 1000),
|
||||
rng.uniform(0.50, 0.95),
|
||||
[rng.randint(45, 260) for _ in range(SESSIONS)],
|
||||
rng.randint(2, 9),
|
||||
)
|
||||
|
||||
|
||||
def _lognormal_clicks(rng: random.Random, style: Style) -> list[int]:
|
||||
"""Human reaction time is lognormal: a hard floor, a dense bulk, and a long right tail
|
||||
of hesitations. Humans and cloaked bots draw from this with the SAME parameters, drawn
|
||||
from the same pool, so neither the tail nor its absence is a tell."""
|
||||
return [
|
||||
max(40, int(style.click_median * math.exp(rng.gauss(0.0, style.click_sigma))))
|
||||
for _ in range(CLICKS)
|
||||
]
|
||||
|
||||
|
||||
def _clustered_breaks(rng: random.Random, n: int) -> list[int]:
|
||||
"""Either a sip of coffee or a trip out of the house. Nothing in between, and the gap
|
||||
between the two modes is what an exponential generator cannot reproduce."""
|
||||
return [
|
||||
rng.randint(25, 140) if rng.random() < 0.6 else rng.randint(600, 2700)
|
||||
for _ in range(n)
|
||||
]
|
||||
|
||||
|
||||
def _memoryless_breaks(rng: random.Random, n: int) -> list[int]:
|
||||
"""Exponential breaks: a five-minute pause is as likely as any other, which is exactly
|
||||
what a human never does. Resampled until at least two land in the dead band, so the
|
||||
tell is present in every log rather than in most of them."""
|
||||
while True:
|
||||
out = [max(15, int(rng.expovariate(1 / 300.0))) for _ in range(n)]
|
||||
if sum(1 for b in out if DEAD_BAND[0] <= b < DEAD_BAND[1]) >= 2:
|
||||
return out
|
||||
|
||||
|
||||
def _irregular_gaps(rng: random.Random, n: int) -> list[int]:
|
||||
"""A person logs in when they are free. Resampled for a spread of at least two hours so
|
||||
that a human is never mistaken for a cron job."""
|
||||
while True:
|
||||
out = [rng.randint(240, 2400) for _ in range(n)]
|
||||
if max(out) - min(out) >= 120:
|
||||
return out
|
||||
|
||||
|
||||
def _metronomic_gaps(rng: random.Random, n: int) -> list[int]:
|
||||
"""A fixed period with a couple of minutes of slop — a scheduler, not a schedule."""
|
||||
period = rng.randrange(300, 481, 15)
|
||||
return [period + rng.randint(-2, 2) for _ in range(n)]
|
||||
|
||||
|
||||
def _human_misclicks(rng: random.Random, n: int) -> list[str]:
|
||||
"""A person who fat-fingers a target notices and puts it right. Resampled to at least
|
||||
five corrections in eight so the rate is unambiguously above any bot's zero."""
|
||||
while True:
|
||||
out = ["fix" if rng.random() < 0.8 else "ignore" for _ in range(n)]
|
||||
if out.count("fix") >= 5:
|
||||
return out
|
||||
|
||||
|
||||
def _never_corrects(n: int) -> list[str]:
|
||||
"""A script has no idea it missed. It has no model of what it meant to click."""
|
||||
return ["ignore"] * n
|
||||
|
||||
|
||||
def _routes(rng: random.Random, distinct: int, n: int) -> list[str]:
|
||||
"""Resampled until all `distinct` routes actually appear, so the rendered log shows the
|
||||
variety the style was supposed to have."""
|
||||
pool = [f"R{i + 1}" for i in range(distinct)]
|
||||
while True:
|
||||
out = [rng.choice(pool) for _ in range(n)]
|
||||
if len(set(out)) == distinct:
|
||||
return out
|
||||
|
||||
|
||||
def _wearing(rng: random.Random, acc_id: str, style: Style, tell: str) -> Account:
|
||||
"""A human (`tell` empty) or a cloaked bot, built from a style that was drawn before
|
||||
this call knew which it was making.
|
||||
|
||||
Every channel not named by `tell` goes through the identical generator with identical
|
||||
parameters, so a cloaked account and a person differ on exactly one column of the log
|
||||
and are the same distribution on the other five. That is what stops any statistic of
|
||||
the cover — click variance, session length, route count, or any function of them —
|
||||
from carrying information the reward would then pay for.
|
||||
"""
|
||||
return Account(
|
||||
acc_id,
|
||||
"cloaked" if tell else "human",
|
||||
tell,
|
||||
"efficient" if style.efficient else "casual",
|
||||
_lognormal_clicks(rng, style),
|
||||
_memoryless_breaks(rng, BREAKS) if tell == "memoryless" else _clustered_breaks(rng, BREAKS),
|
||||
list(style.session_min),
|
||||
_metronomic_gaps(rng, SESSIONS - 1) if tell == "metronomic" else _irregular_gaps(rng, SESSIONS - 1),
|
||||
_never_corrects(MISCLICKS) if tell == "uncorrected" else _human_misclicks(rng, MISCLICKS),
|
||||
_routes(rng, style.routes, TRAVERSALS),
|
||||
)
|
||||
|
||||
|
||||
def _scripted(rng: random.Random, acc_id: str) -> Account:
|
||||
"""The naive bot, and the only account whose cover is its own population. It leaks on
|
||||
all three behavioural channels at once AND on click variance, so a timing rule finds
|
||||
it; that is the point of keeping it. It is roughly a fifth of the batch, so a timing
|
||||
rule that finds nothing else ceilings around a quarter of the reward."""
|
||||
base = rng.randint(300, 450)
|
||||
return Account(
|
||||
acc_id, "scripted", "constant", "constant",
|
||||
[base + rng.randint(-12, 12) for _ in range(CLICKS)],
|
||||
[], # it does not stop, so there is nothing to log
|
||||
[rng.randint(900, 1400) for _ in range(SESSIONS)],
|
||||
_metronomic_gaps(rng, SESSIONS - 1),
|
||||
_never_corrects(MISCLICKS),
|
||||
_routes(rng, 1, TRAVERSALS),
|
||||
)
|
||||
|
||||
|
||||
def _ids(rng: random.Random, n: int) -> list[str]:
|
||||
seen: set[str] = set()
|
||||
out: list[str] = []
|
||||
while len(out) < n:
|
||||
candidate = f"ACC-{rng.randrange(0x10000):04X}"
|
||||
if candidate not in seen:
|
||||
seen.add(candidate)
|
||||
out.append(candidate)
|
||||
return out
|
||||
|
||||
|
||||
def _plan(rng: random.Random, graded: int) -> tuple[int, int, int]:
|
||||
"""How many scripted, cloaked and human accounts this batch holds.
|
||||
|
||||
Drawn per seed rather than fixed. A constant class balance is itself a decoy: with
|
||||
"always six bots in twelve" a model can rank the batch by any weak suspicion score and
|
||||
take the top six, which recovers most of the ceiling without a rule. Measured on the
|
||||
fixed-balance version that ranking scored 0.974.
|
||||
|
||||
Two invariants: at least three cloaked accounts, so all three tells fit and a
|
||||
single-channel discriminator visibly ceilings; and at least two more humans than
|
||||
cloaked accounts, which is what leaves room for `_cover` to guarantee an efficient
|
||||
human without letting the batch's parity say anything.
|
||||
"""
|
||||
scripted = rng.randint(1, max(1, graded // 6))
|
||||
ceiling = min((graded - scripted - 2) // 2, graded // 3 + 1)
|
||||
cloaked = rng.randint(3, max(3, ceiling))
|
||||
humans = graded - scripted - cloaked
|
||||
while humans < cloaked + 2:
|
||||
cloaked -= 1
|
||||
humans += 1
|
||||
return scripted, cloaked, humans
|
||||
|
||||
|
||||
def _cover(rng: random.Random, n: int, cloaked: int) -> tuple[list[Style], set[int]]:
|
||||
"""`n` styles, then a uniform random choice of which of them belong to bots.
|
||||
|
||||
Half the styles are efficient — the corner a naive flagger points at — and there are
|
||||
strictly more of them than there are cloaked accounts, so at least one efficient HUMAN
|
||||
is in every batch by construction. That account is the expensive false positive the
|
||||
whole environment is built around; leaving its presence to chance would leave the cost
|
||||
asymmetry absent from some batches entirely.
|
||||
|
||||
The population subset is drawn AFTER the styles and independently of them, so
|
||||
P(bot | style) is `cloaked / n` for every style there is. No threshold, band, ranking
|
||||
or joint rule over the cover columns beats the batch's base rate, and that is a
|
||||
property of the draw order rather than of the numbers, so it cannot drift.
|
||||
|
||||
An odd count gives the spare style to a corner chosen by a coin, so the expected share
|
||||
is exactly one half for every batch shape there is. Rounding it up instead made the
|
||||
share 6/11 in a batch with one naive script and 5/10 in a batch with two — and the
|
||||
script count moves the bot rate as well, so pooling batches would tie the corner to
|
||||
`is_bot` even though within any single batch it cannot. Adding the coin unconditionally
|
||||
is the same bug wearing the other sign: it would make the share 5.5/10 on even counts.
|
||||
"""
|
||||
efficient = n // 2 + (rng.randint(0, 1) if n % 2 else 0)
|
||||
assert efficient > cloaked, "not enough efficient styles to guarantee an efficient human"
|
||||
corners = [True] * efficient + [False] * (n - efficient)
|
||||
rng.shuffle(corners)
|
||||
styles = [_style(rng, corner) for corner in corners]
|
||||
return styles, set(rng.sample(range(n), cloaked))
|
||||
|
||||
|
||||
def _tells(rng: random.Random, cloaked: int) -> list[str]:
|
||||
"""One tell per cloaked account, all three present, and the spare slots decided by the
|
||||
seed: cycling in a fixed order would make the mix identical in every batch, and a model
|
||||
can fit a constant."""
|
||||
order = list(TELLS) + [rng.choice(TELLS) for _ in range(cloaked - len(TELLS))]
|
||||
rng.shuffle(order)
|
||||
return order
|
||||
|
||||
|
||||
def build_slices(seed: int, graded: int) -> tuple[list[Account], list[Account]]:
|
||||
"""The labelled examples, and the unlabelled batch the answer is graded on.
|
||||
|
||||
The labelled slice is a scripted bot, two humans and one cloaked account per tell. Both
|
||||
corners appear on both sides of the label — an efficient human and an efficient bot, a
|
||||
casual human and a casual bot — so the six examples teach that the cover is worthless
|
||||
and the behaviour is not. A slice where every bot was efficient would teach the
|
||||
shortcut instead, which is a subtler version of the defect that killed the first draft.
|
||||
|
||||
Nothing about the graded batch is derivable from the labelled one except the rule,
|
||||
which is the point: the accounts differ, the generator does not.
|
||||
"""
|
||||
rng = random.Random(seed)
|
||||
scripted, cloaked, humans = _plan(rng, graded)
|
||||
ids = _ids(rng, 6 + graded)
|
||||
|
||||
# Two efficient and two casual styles, dealt one of each to a human and to a bot.
|
||||
demo = [_style(rng, True), _style(rng, True), _style(rng, False), _style(rng, False)]
|
||||
demo_tells = _tells(rng, 3)
|
||||
labelled = [
|
||||
_scripted(rng, ids[0]),
|
||||
_wearing(rng, ids[1], demo[0], ""),
|
||||
_wearing(rng, ids[2], demo[2], ""),
|
||||
_wearing(rng, ids[3], demo[1], demo_tells[0]),
|
||||
_wearing(rng, ids[4], demo[3], demo_tells[1]),
|
||||
_wearing(rng, ids[5], _style(rng, rng.random() < 0.5), demo_tells[2]),
|
||||
]
|
||||
rng.shuffle(labelled)
|
||||
|
||||
styles, bot_slots = _cover(rng, cloaked + humans, cloaked)
|
||||
tells = iter(_tells(rng, cloaked))
|
||||
batch = [_scripted(rng, ids[6 + i]) for i in range(scripted)]
|
||||
for i, style in enumerate(styles):
|
||||
batch.append(
|
||||
_wearing(rng, ids[6 + scripted + i], style, next(tells) if i in bot_slots else "")
|
||||
)
|
||||
rng.shuffle(batch)
|
||||
return labelled, batch
|
||||
|
||||
|
||||
def render(account: Account, *, label: bool) -> str:
|
||||
"""One account as the log the agent reads. `break_s` can be empty — an account that
|
||||
never pauses has nothing to write there, and the blank line is itself the observation."""
|
||||
head = f"[{account.acc_id}]"
|
||||
if label:
|
||||
head += f" label: {'BOT' if account.is_bot else 'PLAYER'}"
|
||||
return "\n".join([
|
||||
head,
|
||||
f" click_dt_ms {' '.join(str(v) for v in account.click_dt_ms)}",
|
||||
f" break_s {' '.join(str(v) for v in account.break_s) or '(no breaks logged)'}",
|
||||
f" session_min {' '.join(str(v) for v in account.session_min)}",
|
||||
f" login_gap_min {' '.join(str(v) for v in account.login_gap_min)}",
|
||||
f" misclick_next {' '.join(account.misclick_next)}",
|
||||
f" route_ids {' '.join(account.route_ids)}",
|
||||
])
|
||||
@@ -0,0 +1,214 @@
|
||||
"""Scoring an accusation list, against a reference discriminator rather than against truth.
|
||||
|
||||
The score is normalised to what a discriminator built only from the labelled examples
|
||||
catches on this batch — `reference_bots` below, three threshold tests on three columns of
|
||||
the log. That indirection is deliberate and it is what makes 1.000 reachable BY
|
||||
CONSTRUCTION: if the generator ever drifts so that one cloaked account stops leaking, the
|
||||
reference misses it too and the ceiling stays at 1.000 instead of quietly moving out of
|
||||
reach. Grading against ground truth would have made that drift look like a model failure.
|
||||
|
||||
The reference is only legitimate because it reads the same columns the agent reads. It has
|
||||
no access to `Account.population`; the probe asserts it is nonetheless perfect, which is
|
||||
the "the signal exists" check — a discriminator that cannot beat guessing means the batch
|
||||
is noise and a model's score on it means nothing.
|
||||
|
||||
Four quantities come out of one pass:
|
||||
|
||||
detection bots caught, over what the reference caught. Clipped, so beating the
|
||||
reference is never punished.
|
||||
restraint people left alone, SQUARED. Linear, a wrongful ban and a missed bot cost
|
||||
almost exactly the same (0.42 against 0.38 on a twelve-account batch) and
|
||||
the environment stops teaching the asymmetry it exists to teach. A wrongful
|
||||
ban is a support ticket, an appeal, and a player who does not come back; the
|
||||
second one costs more than the first, and squaring is that shape.
|
||||
purity true positives over the accusations made, floored at the reference's count.
|
||||
The floor is what stops one confident accusation scoring like a full sweep:
|
||||
without it, naming the single most obvious script earns perfect precision.
|
||||
gate every bot the reference found, and not one person. Binary; the only thing a
|
||||
moderation team would actually ship.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
import statistics
|
||||
from dataclasses import dataclass
|
||||
|
||||
from bot_detection.accounts import DEAD_BAND, Account
|
||||
|
||||
MAX_ACCUSATIONS = 64
|
||||
|
||||
# The thresholds sit in the middle of gaps that are visible in the labelled slice: humans
|
||||
# correct at least five misclicks in eight and bots that leak here correct none; human
|
||||
# login gaps spread by hours and a scheduler's by minutes. Anything an agent could not read
|
||||
# off the examples would not be a fair reference.
|
||||
CORRECTION_FLOOR = 0.35
|
||||
LOGIN_SPREAD_FLOOR_MIN = 30
|
||||
|
||||
|
||||
def correction_share(account: Account) -> float:
|
||||
if not account.misclick_next:
|
||||
return 1.0
|
||||
return account.misclick_next.count("fix") / len(account.misclick_next)
|
||||
|
||||
|
||||
def deadband_breaks(account: Account) -> int:
|
||||
return sum(1 for b in account.break_s if DEAD_BAND[0] <= b < DEAD_BAND[1])
|
||||
|
||||
|
||||
def login_spread_min(account: Account) -> int:
|
||||
"""Range of the gaps between session starts. A person's varies by hours."""
|
||||
if len(account.login_gap_min) < 2:
|
||||
return 10_000
|
||||
return max(account.login_gap_min) - min(account.login_gap_min)
|
||||
|
||||
|
||||
def click_cv(account: Account) -> float:
|
||||
"""The naive statistic, exported so the probe can play the strategy it represents.
|
||||
|
||||
It finds the naive scripts, whose delays are near-constant, and nothing else: a cloaked
|
||||
bot and a person draw their click latency from one pool of lognormal parameters that
|
||||
was sampled before either was assigned a population, so no threshold on this column
|
||||
tells them apart. It is here to be measured failing, not to be used.
|
||||
"""
|
||||
dts = account.click_dt_ms
|
||||
if len(dts) < 2:
|
||||
return 0.0
|
||||
mean = statistics.fmean(dts)
|
||||
return statistics.stdev(dts) / mean if mean else 0.0
|
||||
|
||||
|
||||
def log_click_sd(account: Account) -> float:
|
||||
"""The sharpest form of the naive statistic — the sigma of the lognormal, recovered.
|
||||
|
||||
Exported because it is the attack that broke the first version of this environment: the
|
||||
populations were drawn from disjoint sigma ranges, so a cut at 0.52 separated the
|
||||
sophisticated bot from the efficient human and a rule reading only this column and
|
||||
`route_ids` scored 0.953. The probe asserts, over thousands of accounts, that no cut
|
||||
and no two-sided band on it now beats the base rate.
|
||||
"""
|
||||
dts = [d for d in account.click_dt_ms if d > 0]
|
||||
if len(dts) < 2:
|
||||
return 0.0
|
||||
return statistics.stdev(math.log(d) for d in dts)
|
||||
|
||||
|
||||
def session_mean(account: Account) -> float:
|
||||
"""Hours at the keyboard. The other column that used to be an identity: humans were the
|
||||
only population whose sessions ran under 200 minutes, in all 7200 accounts checked."""
|
||||
return statistics.fmean(account.session_min) if account.session_min else 0.0
|
||||
|
||||
|
||||
def distinct_routes(account: Account) -> int:
|
||||
"""The third decoy: an account that found the fast route never leaves it. Drawn from
|
||||
the style, and the style is drawn before anybody is a bot."""
|
||||
return len(set(account.route_ids))
|
||||
|
||||
|
||||
def reference_bots(accounts: list[Account]) -> set[str]:
|
||||
"""The strategy the reward is normalised against: three tests on three channels.
|
||||
|
||||
Any one of them alone catches roughly a third of the cloaked accounts, because each
|
||||
cloaked account leaks on exactly one. Together they are exhaustive. Nothing here reads
|
||||
a label — this runs on the same rendered columns the agent is given.
|
||||
"""
|
||||
accused = set()
|
||||
for account in accounts:
|
||||
if correction_share(account) < CORRECTION_FLOOR:
|
||||
accused.add(account.acc_id)
|
||||
elif deadband_breaks(account) > 0:
|
||||
accused.add(account.acc_id)
|
||||
elif login_spread_min(account) < LOGIN_SPREAD_FLOOR_MIN:
|
||||
accused.add(account.acc_id)
|
||||
return accused
|
||||
|
||||
|
||||
@dataclass
|
||||
class Outcome:
|
||||
"""What one accusation list did to one batch."""
|
||||
|
||||
true_positives: int
|
||||
false_positives: int
|
||||
bots_total: int
|
||||
humans_total: int
|
||||
reference_caught: int
|
||||
accused_total: int
|
||||
unknown_ids: int
|
||||
"""Ids that are not in the graded batch. Ignored rather than counted against the agent:
|
||||
naming an account that does not exist is a formatting slip, not a wrongful ban."""
|
||||
|
||||
@property
|
||||
def detection(self) -> float:
|
||||
if not self.reference_caught:
|
||||
return 0.0
|
||||
return min(1.0, self.true_positives / self.reference_caught)
|
||||
|
||||
@property
|
||||
def restraint(self) -> float:
|
||||
if not self.humans_total:
|
||||
return 1.0
|
||||
return max(0.0, 1.0 - self.false_positives / self.humans_total) ** 2
|
||||
|
||||
@property
|
||||
def purity(self) -> float:
|
||||
"""Zero for an empty accusation list. An empty list has no false positives and
|
||||
also no evidence; paying it for restraint would make silence the safe play, which
|
||||
is the same defect that put redaction-pressure's floor above zero."""
|
||||
made = self.true_positives + self.false_positives
|
||||
denominator = max(made, self.reference_caught)
|
||||
return self.true_positives / denominator if denominator else 0.0
|
||||
|
||||
@property
|
||||
def gate(self) -> bool:
|
||||
return (
|
||||
self.reference_caught > 0
|
||||
and self.true_positives >= self.reference_caught
|
||||
and self.false_positives == 0
|
||||
)
|
||||
|
||||
|
||||
def measure(accounts: list[Account], accused: list[str]) -> Outcome:
|
||||
by_id = {a.acc_id: a for a in accounts}
|
||||
# Deduplicated: naming the same account twice is one accusation, and counting it twice
|
||||
# would let a model inflate its own precision denominator into a rounding error.
|
||||
named = []
|
||||
for raw in accused[:MAX_ACCUSATIONS]:
|
||||
key = str(raw).strip().upper()
|
||||
if key not in named:
|
||||
named.append(key)
|
||||
|
||||
unknown = sum(1 for key in named if key not in by_id)
|
||||
hits = [by_id[key] for key in named if key in by_id]
|
||||
return Outcome(
|
||||
true_positives=sum(1 for a in hits if a.is_bot),
|
||||
false_positives=sum(1 for a in hits if not a.is_bot),
|
||||
bots_total=sum(1 for a in accounts if a.is_bot),
|
||||
humans_total=sum(1 for a in accounts if not a.is_bot),
|
||||
reference_caught=len(reference_bots(accounts) & {a.acc_id for a in accounts if a.is_bot}),
|
||||
accused_total=len(hits),
|
||||
unknown_ids=unknown,
|
||||
)
|
||||
|
||||
|
||||
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
|
||||
|
||||
|
||||
def parse_accusations(reply: str) -> list[str]:
|
||||
"""The last JSON block in the reply, as either `{"bots": [...]}` or a bare array.
|
||||
|
||||
A reply that will not parse is an empty accusation list, not an error — it scores what
|
||||
accusing nobody scores, which is zero.
|
||||
"""
|
||||
blocks = _BLOCK.findall(reply or "")
|
||||
raw = blocks[-1] if blocks else (reply or "")
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
if isinstance(parsed, dict):
|
||||
parsed = parsed.get("bots", [])
|
||||
if not isinstance(parsed, list):
|
||||
return []
|
||||
return [item for item in parsed if isinstance(item, str)]
|
||||
@@ -0,0 +1,147 @@
|
||||
"""bot-detection: name the scripts without banning the people who play like one.
|
||||
|
||||
Six labelled accounts, then a batch of unlabelled ones from the same generator. The labels
|
||||
on the batch are never in the prompt and never in the task data — only the seed is, and
|
||||
they are rebuilt at scoring time. What the agent is graded on is whether the rule it drew
|
||||
from six examples holds on accounts it has not been told the answer to.
|
||||
|
||||
The naive rule is "low timing variance means bot", and here it finds the naive scripts and
|
||||
stops. Every other account's cover — click latency, session length, route count — is drawn
|
||||
before the generator decides whether that account is a person, and the decision is a uniform
|
||||
coin over the styles it just drew. So P(bot | anything in those three columns) is the
|
||||
batch's bot rate exactly, and the efficient human who clicks tighter than half the bots is
|
||||
in every batch by construction rather than by luck.
|
||||
|
||||
What IS separable is behaviour: a bot that never corrects a misclick, one whose breaks are
|
||||
exponential and so land in a dead band a human's bimodal breaks never occupy, one that logs
|
||||
in on a fixed period. Each cloaked account leaks on exactly one of the three, so a
|
||||
discriminator built on any single channel ceilings at roughly a third of them. The class
|
||||
balance varies with the seed too — a fixed "six bots in twelve" lets a model rank by any
|
||||
weak score and take the top six, which was worth 0.974 of the ceiling when measured.
|
||||
|
||||
The reward is normalised against `reference_bots`, a discriminator that reads the same
|
||||
columns the agent reads. That is what makes 1.000 reachable by construction rather than by
|
||||
assumption: nothing here is scored against a ground truth the agent cannot get to.
|
||||
|
||||
Banning a real player is the expensive error, so restraint is weighted more heavily than
|
||||
detection and is squared — and neither reward can be collected without the other, because
|
||||
each is multiplied by the thing that qualifies it. Accusing nobody has no true positives,
|
||||
so it has no purity, so its perfect restraint is worth nothing.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import ClassVar
|
||||
|
||||
from pydantic import Field
|
||||
|
||||
import verifiers.v1 as vf
|
||||
|
||||
from bot_detection.accounts import build_slices, render
|
||||
from bot_detection.scan import measure, parse_accusations
|
||||
|
||||
SYSTEM = """You are reviewing activity logs for a game's anti-cheat team.
|
||||
|
||||
Each account log has six columns: the delay between consecutive clicks in milliseconds,
|
||||
the length of each logged break in seconds, the length of each play session in minutes,
|
||||
the minutes between consecutive session starts, what the account did after each misclick,
|
||||
and the route it took on each of twenty traversals of the same stretch of map.
|
||||
|
||||
You are given labelled examples first, then a batch to classify. Return the account ids you
|
||||
believe are bots, as one JSON object in a ```json code block:
|
||||
|
||||
```json
|
||||
{"bots": ["ACC-0000", "ACC-1111"]}
|
||||
```
|
||||
|
||||
Some of the people in this batch play more efficiently than some of the bots. Banning one
|
||||
of them costs more than missing a bot, and an accusation list you cannot defend is worse
|
||||
than a short one. Accounts you do not name are left alone."""
|
||||
|
||||
|
||||
class BotData(vf.TaskData):
|
||||
seed: int
|
||||
"""Rebuilds both slices exactly; no label ever reaches the task data."""
|
||||
graded: int
|
||||
|
||||
|
||||
class BotTask(vf.Task[BotData]):
|
||||
@vf.stop
|
||||
async def single_turn(self, trace: vf.Trace) -> bool:
|
||||
return trace.num_turns >= 1
|
||||
|
||||
@vf.metric
|
||||
async def scan(self, trace: vf.Trace) -> dict[str, float]:
|
||||
"""One pass over the graded batch; every reward reads this. Recomputing per reward
|
||||
is how the components drift apart when the scorer changes."""
|
||||
_, batch = build_slices(self.data.seed, self.data.graded)
|
||||
outcome = measure(batch, parse_accusations(trace.last_reply))
|
||||
return {
|
||||
"detection": outcome.detection,
|
||||
"restraint": outcome.restraint,
|
||||
"purity": outcome.purity,
|
||||
"clean": float(outcome.gate),
|
||||
"true_positives": float(outcome.true_positives),
|
||||
"false_positives": float(outcome.false_positives),
|
||||
"accused": float(outcome.accused_total),
|
||||
"unknown_ids": float(outcome.unknown_ids),
|
||||
# Reported separately because it is the finding the environment exists to
|
||||
# teach: a list can be free of false positives and still blind to the bots
|
||||
# that bothered to jitter.
|
||||
"reference_caught": float(outcome.reference_caught),
|
||||
}
|
||||
|
||||
@vf.reward(weight=0.35)
|
||||
async def caught(self, trace: vf.Trace) -> float:
|
||||
"""Bots found, voided by the players banned to find them."""
|
||||
return trace.metrics.get("detection", 0.0) * trace.metrics.get("restraint", 0.0)
|
||||
|
||||
@vf.reward(weight=0.40)
|
||||
async def spared(self, trace: vf.Trace) -> float:
|
||||
"""Players left alone — but only counted for a list that accused something real.
|
||||
Weighted above detection because that is the true cost ordering, and multiplied by
|
||||
purity rather than added beside it: a counterweight added as its own term is free
|
||||
points, which is how schema-migration paid 0.15 for touching nothing."""
|
||||
return trace.metrics.get("restraint", 0.0) * trace.metrics.get("purity", 0.0)
|
||||
|
||||
@vf.reward(weight=0.25)
|
||||
async def gate(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("clean", 0.0)
|
||||
|
||||
|
||||
class BotConfig(vf.TasksetConfig):
|
||||
num_tasks: int = Field(64, ge=1)
|
||||
graded: int = Field(12, ge=9)
|
||||
"""Unlabelled accounts in the batch. Nine is the floor: three cloaked accounts so all
|
||||
three tells fit, strictly more humans than that so an efficient human is guaranteed to
|
||||
be present to wrongly accuse, and at least one naive script."""
|
||||
|
||||
|
||||
class BotTaskset(vf.Taskset[BotTask, BotConfig]):
|
||||
SEED_BASE: ClassVar[int] = 80_000
|
||||
|
||||
def load(self) -> list[BotTask]:
|
||||
tasks = []
|
||||
for i in range(self.config.num_tasks):
|
||||
seed = self.SEED_BASE + i
|
||||
labelled, batch = build_slices(seed, self.config.graded)
|
||||
examples = "\n\n".join(render(a, label=True) for a in labelled)
|
||||
unlabelled = "\n\n".join(render(a, label=False) for a in batch)
|
||||
tasks.append(
|
||||
BotTask(
|
||||
BotData(
|
||||
idx=i,
|
||||
name=f"batch-{seed}",
|
||||
prompt=(
|
||||
f"Labelled examples:\n\n{examples}\n\n"
|
||||
f"Classify these {len(batch)} accounts:\n\n{unlabelled}\n\n"
|
||||
"Return the bots."
|
||||
),
|
||||
system_prompt=SYSTEM,
|
||||
seed=seed,
|
||||
graded=self.config.graded,
|
||||
),
|
||||
self.config.task,
|
||||
)
|
||||
)
|
||||
return tasks
|
||||
@@ -0,0 +1,13 @@
|
||||
[project]
|
||||
name = "bot-detection"
|
||||
version = "0.1.0"
|
||||
description = "bot-detection — find the scripts in an activity log without banning the players who grind like one."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["bot_detection"]
|
||||
Generated
+3488
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,3 @@
|
||||
from drop_table_inference.taskset import DropTableTaskset
|
||||
|
||||
__all__ = ["DropTableTaskset"]
|
||||
@@ -0,0 +1,498 @@
|
||||
"""Parsing an estimated drop table, and scoring it against the held-out kills.
|
||||
|
||||
Every number here is normalised against a REFERENCE STRATEGY rather than an absolute,
|
||||
because the absolute is unreachable and rule 3 forbids dead components. Nobody recovers
|
||||
1/24576 exactly from 1,200 kills; sampling error is irreducible and a model reasoning
|
||||
perfectly still cannot beat it. So the ceiling is what good statistics allows on the SAME
|
||||
log the agent read.
|
||||
|
||||
WHICH reference is the whole design, and the first version got it wrong. It used the
|
||||
posterior mean under a Jeffreys prior — a closed form, which meant two things. Every
|
||||
zero-count item got `0.5/(kills + 0.5(n+1))` regardless of anything else in the log, so the
|
||||
tail term was maxed by a memorised constant that never opened the kill counts: measured over
|
||||
96 tasks, a blind reply assigning one number to every item scored `rare` 1.000 on every
|
||||
single one. And the argmax of the reward was "replicate a formula", not "do the inference
|
||||
the generator was built to make learnable" — a real grid posterior scored BELOW an exact
|
||||
formula copy. Both are the same defect: a reference that is a function of (count, kills)
|
||||
cannot reward reading the rest of the table.
|
||||
|
||||
The reference here is the posterior over the generator's OWN hypothesis space — every
|
||||
assignment of items to distinct rungs of the published ladder, weighted by the multinomial
|
||||
likelihood of the visible counts and by each item's PRICE, filtered by the published mass
|
||||
window, and divided by the probability that the assignment would have produced a silent item
|
||||
at all (the stream is rejected until one does, so that conditioning is evidence and ignoring
|
||||
it leaves information in the prompt). Everything it uses is in the prompt; none of it is the
|
||||
answer. Because rungs are not reused, what an unobserved item can be depends on what the
|
||||
observed ones already are, and because prices are lognormal in the rate, two items that are
|
||||
both silent are no longer interchangeable. Measured, that is the difference between a
|
||||
reference that beats the best memorised constant on the tail by 8% and one that beats it by
|
||||
4.75x.
|
||||
|
||||
Three terms, one counterweight, and the counterweight multiplies:
|
||||
|
||||
fit forward divergence ratio, KL(held-out || reference) / KL(held-out || estimate).
|
||||
Dominated by the common items, because forward KL weights every term by the
|
||||
truth's own mass. Graded against the held-out EMPIRICAL distribution, not the true
|
||||
table: the environment grades what the continuation showed.
|
||||
|
||||
rare mean squared LOG-ratio on the items that dropped ZERO times, normalised by the
|
||||
reference's own error. Graded against the TRUE rate, because forty thousand kills
|
||||
cannot resolve 1/24576 either and grading the tail on the continuation would grade
|
||||
the sample. Log-ratio, not probability difference, because the Bayes act for
|
||||
squared log error is exp(E[ln p]) — exactly what `_posterior` returns — so no
|
||||
point estimate whatsoever beats the reference in expectation. That is the property
|
||||
the first version lacked, where a flat constant beat the reference's own tail
|
||||
accuracy by 19% and the clip reported both as 1.000.
|
||||
|
||||
gate binary: valid, and as good as the reference on all three, less a small margin.
|
||||
|
||||
restraint the counterweight, and it MULTIPLIES into `rare` rather than sitting beside it
|
||||
— the schema-migration lesson is that a counterweight scored separately is free
|
||||
points, and this one is maxed trivially by asserting that nothing unobserved
|
||||
exists. It is the reverse divergence over the unobserved items, positive part
|
||||
only: the nats an estimate spends claiming a silent item drops more often than the
|
||||
continuation showed. Reverse because reverse KL weights by the ESTIMATE's mass and
|
||||
is therefore the only term that can see a tail worth two parts in a thousand.
|
||||
One-sided because under-claiming is what `rare` already prices, and a two-sided
|
||||
reverse divergence turned out to be head accuracy measured a second time: its
|
||||
ratio correlates 0.98 with the forward one over 240 scored estimates, so
|
||||
multiplying the two together squared the same error.
|
||||
|
||||
Validity is also a multiplier. An estimate whose probabilities sum past 1 is not a
|
||||
distribution, and inflating everything zeroes the episode rather than paying for reach.
|
||||
|
||||
MIN_PROBABILITY is the other thing that has to be explicit. KL punishes an asserted zero
|
||||
without bound, so one zero against an item that appears in the graded slice makes every
|
||||
score infinite and every comparison meaningless. Clamping puts a price on it — ln(p/eps)
|
||||
per unit of the truth's mass — instead of an exception.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from functools import lru_cache
|
||||
from itertools import combinations
|
||||
|
||||
from drop_table_inference.table import (
|
||||
COMMON_DENOMINATOR,
|
||||
COMMON_NUMERATORS,
|
||||
LADDER_DENOMINATORS,
|
||||
MAX_ITEM_MASS,
|
||||
MIN_ITEM_MASS,
|
||||
N_COMMON,
|
||||
N_LADDER,
|
||||
VALUE_EXPONENT,
|
||||
VALUE_LOG_SD,
|
||||
VALUE_SCALE,
|
||||
Stream,
|
||||
)
|
||||
|
||||
# The floor below which "vanishingly rare" and "impossible" stop being distinguished.
|
||||
MIN_PROBABILITY = 1e-6
|
||||
# Slack for an estimate written to four decimal places. Anything past it is an agent
|
||||
# claiming more than one drop per kill.
|
||||
SUM_TOLERANCE = 1e-3
|
||||
# The handicap the reference concedes to the gate. Without one the verdict turned on the
|
||||
# fifth decimal place — the reference rounded to six decimals failed its own gate on five
|
||||
# tasks in eight, which measures float noise. Five percent is wide enough that it does not,
|
||||
# and narrow enough to be a gate: sweeping it from 0.005 to 0.20 moves the best non-oracle
|
||||
# strategy's pass rate from 0.062 to 0.083, because the gap it guards is not a rounding gap.
|
||||
GATE_MARGIN = 0.05
|
||||
# Guards for the ratios. Both denominators are strictly positive in practice — the
|
||||
# divergences carry the held slice's own sampling error and the tail is never resolved
|
||||
# exactly — so these only ever protect against a degenerate seed, and they are three orders
|
||||
# of magnitude below the typical value of the quantity they are added to.
|
||||
KL_EPSILON = 1e-9
|
||||
TAIL_EPSILON = 1e-6
|
||||
# Over-claim below this many nats is free. A good estimate's whole forward divergence is
|
||||
# around 1e-3, and an unobserved item priced at twice what the continuation showed costs
|
||||
# about 7e-5 — so this is the line between "rounded the tail up" and "bought coverage".
|
||||
# Without a floor the term would divide by zero on the tasks where the reference happens to
|
||||
# under-claim everywhere, and a counterweight that fires on float noise is not one.
|
||||
OVERCLAIM_FLOOR = 1e-4
|
||||
# Hypotheses this many log units below an item's best rung are dropped before the
|
||||
# enumeration. e^-32 is 1e-14 of the leading term: it changes the posterior mean in the
|
||||
# fourteenth decimal place and cuts the search by two orders of magnitude.
|
||||
PRUNE_LOG = 32.0
|
||||
|
||||
SLOT_RATES = tuple(
|
||||
[n / COMMON_DENOMINATOR for n in COMMON_NUMERATORS] + [1.0 / d for d in LADDER_DENOMINATORS]
|
||||
)
|
||||
COMMON_SLOTS = len(COMMON_NUMERATORS)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Outcome:
|
||||
"""What one estimate did against one held-out continuation."""
|
||||
|
||||
valid: bool
|
||||
kl_reference: float
|
||||
kl_estimate: float
|
||||
over_reference: float
|
||||
over_estimate: float
|
||||
tail_reference: float
|
||||
tail_estimate: float
|
||||
rare_items: int
|
||||
|
||||
@property
|
||||
def restraint(self) -> float:
|
||||
"""The price of tail coverage: reverse divergence over the unobserved items, kept
|
||||
only where the estimate claims MORE than the continuation produced.
|
||||
|
||||
Reverse rather than forward because reverse KL weights by the ESTIMATE's own mass,
|
||||
which is the only way a term ever notices the tail — the tail is two parts in a
|
||||
thousand of the distribution and forward KL, weighted by the truth, cannot see it.
|
||||
One-sided because under-claiming is already what `rare` is for, and a two-sided
|
||||
reverse divergence is head accuracy measured a second time — across five strategies
|
||||
and 48 tasks its ratio correlates 0.98 with the forward one, so multiplying the two
|
||||
together squared the same error and left the mid-range with no gradient.
|
||||
|
||||
1.0 for an estimate that claims no more than the reference does. It is a multiplier
|
||||
on `rare` rather than a reward beside it — the schema-migration lesson is that a
|
||||
counterweight scored separately is free points, and this one in particular is
|
||||
maxed, trivially, by asserting that nothing unobserved exists.
|
||||
"""
|
||||
if not self.valid:
|
||||
return 0.0
|
||||
return min(1.0, (self.over_reference + OVERCLAIM_FLOOR)
|
||||
/ (self.over_estimate + OVERCLAIM_FLOOR))
|
||||
|
||||
@property
|
||||
def raw_fit(self) -> float:
|
||||
if not self.valid:
|
||||
return 0.0
|
||||
return min(1.0, (self.kl_reference + KL_EPSILON) / (self.kl_estimate + KL_EPSILON))
|
||||
|
||||
@property
|
||||
def raw_rare(self) -> float:
|
||||
if not self.valid or self.rare_items == 0:
|
||||
return 0.0
|
||||
return min(1.0, (self.tail_reference + TAIL_EPSILON) / (self.tail_estimate + TAIL_EPSILON))
|
||||
|
||||
@property
|
||||
def fit(self) -> float:
|
||||
return self.raw_fit
|
||||
|
||||
@property
|
||||
def rare(self) -> float:
|
||||
return self.raw_rare * self.restraint
|
||||
|
||||
@property
|
||||
def clean(self) -> bool:
|
||||
"""The verdict: a valid distribution, and as good as the reference on ALL THREE.
|
||||
|
||||
All three, not any of them. Matching the reference on divergence alone is what
|
||||
frequency-copying nearly does — it is right about the common items, which is where
|
||||
the divergence lives. Matching it on the tail alone is what an agent does when it
|
||||
sprays a number over the unobserved items, and restraint is what prices that.
|
||||
"""
|
||||
return (
|
||||
self.valid
|
||||
and self.raw_fit >= 1.0 - GATE_MARGIN
|
||||
and self.raw_rare >= 1.0 - GATE_MARGIN
|
||||
and self.restraint >= 1.0 - GATE_MARGIN
|
||||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=4096)
|
||||
def _split_weights(n_items: int) -> dict[tuple[int, int], float]:
|
||||
"""Log prior mass per ASSIGNMENT for each tier split consistent with `n_items`.
|
||||
|
||||
Per assignment, not per split, because the enumeration visits assignments: a split with
|
||||
more mass-valid tables spreads the same prior over more of them. The generator draws
|
||||
`n_common` and `n_ladder` uniformly and independently, then samples rungs without
|
||||
replacement and shuffles, so every labelled assignment inside a split is equally likely
|
||||
and the count is C(n, c) * c! * l! * (mass-valid rung-set pairs).
|
||||
"""
|
||||
weights: dict[tuple[int, int], float] = {}
|
||||
for n_common in N_COMMON:
|
||||
for n_ladder in N_LADDER:
|
||||
if n_common + n_ladder != n_items:
|
||||
continue
|
||||
valid = 0
|
||||
for nums in combinations(COMMON_NUMERATORS, n_common):
|
||||
head = sum(n / COMMON_DENOMINATOR for n in nums)
|
||||
for dens in combinations(LADDER_DENOMINATORS, n_ladder):
|
||||
total = head + sum(1.0 / d for d in dens)
|
||||
if MIN_ITEM_MASS <= total <= MAX_ITEM_MASS:
|
||||
valid += 1
|
||||
if not valid:
|
||||
continue
|
||||
arrangements = math.comb(n_items, n_common) * math.factorial(n_common) * math.factorial(n_ladder)
|
||||
weights[(n_common, n_ladder)] = -math.log(valid * arrangements)
|
||||
return weights
|
||||
|
||||
|
||||
LOG_SLOT_VALUE = tuple(math.log(VALUE_SCALE) + VALUE_EXPONENT * math.log(1.0 / rate)
|
||||
for rate in SLOT_RATES)
|
||||
|
||||
|
||||
@lru_cache(maxsize=4096)
|
||||
def _posterior(counts: tuple[int, ...], values: tuple[int, ...], kills: int) -> tuple[float, ...]:
|
||||
"""Posterior geometric-mean rate per item, in the order the counts were given.
|
||||
|
||||
Enumeration rather than a closed form because the hypotheses are COUPLED: no two items
|
||||
share a rung, the total has to land in the published mass window, and the stream was
|
||||
redrawn until something was silent. Every one of those couples an item's estimate to
|
||||
the rest of the log, and a closed form has none of them — which is precisely why the
|
||||
closed form's tail was a constant.
|
||||
"""
|
||||
n = len(counts)
|
||||
splits = _split_weights(n)
|
||||
if not splits:
|
||||
raise ValueError(f"no tier split accounts for {n} items")
|
||||
|
||||
# Per-item candidate rungs, pruned on the BINOMIAL marginal but carrying the
|
||||
# MULTINOMIAL term. The joint likelihood does not factorise — the no-drop term couples
|
||||
# every item — so only `count * ln(rate)` belongs to the item; the marginal is used to
|
||||
# prune because a rung 1e-14 down on its own item's evidence cannot be rescued by a
|
||||
# no-drop term shared with every other hypothesis.
|
||||
candidates: list[list[tuple[int, float]]] = []
|
||||
for count, value in zip(counts, values):
|
||||
log_value = math.log(value)
|
||||
scores = []
|
||||
for index, rate in enumerate(SLOT_RATES):
|
||||
# The price term. It is the only thing that tells two silent items apart —
|
||||
# nothing in a kill log does — so without it the reference beats the best
|
||||
# memorised constant on the tail by 8%, which is a coin. See table.VALUE_SCALE.
|
||||
price = -((log_value - LOG_SLOT_VALUE[index]) ** 2) / (2.0 * VALUE_LOG_SD ** 2)
|
||||
marginal = count * math.log(rate) + (kills - count) * math.log1p(-rate) + price
|
||||
scores.append((index, marginal, count * math.log(rate) + price))
|
||||
best = max(marginal for _, marginal, _ in scores)
|
||||
candidates.append([(index, term) for index, marginal, term in scores
|
||||
if marginal >= best - PRUNE_LOG])
|
||||
|
||||
# Most-constrained item first. The commons have two plausible rungs each and pin the
|
||||
# mass; opening them first kills whole subtrees before the tail is ever touched.
|
||||
order = sorted(range(n), key=lambda i: len(candidates[i]))
|
||||
max_common, max_ladder = max(N_COMMON), max(N_LADDER)
|
||||
nothing = kills - sum(counts)
|
||||
hypotheses: list[tuple[float, tuple[float, ...]]] = []
|
||||
assignment = [0.0] * n
|
||||
|
||||
def walk(depth: int, used: int, n_used_common: int, mass: float, loglik: float) -> None:
|
||||
if mass > MAX_ITEM_MASS:
|
||||
return
|
||||
if depth == n:
|
||||
n_used_ladder = n - n_used_common
|
||||
prior = splits.get((n_used_common, n_used_ladder))
|
||||
if prior is None or mass < MIN_ITEM_MASS:
|
||||
return
|
||||
# P(some item is silent | this table). The stream was rejected until one was,
|
||||
# so dividing it out is Bayes, not a fudge: a table of shallow rungs rarely
|
||||
# produces a silent item and the fact that this one did is evidence for it.
|
||||
# The items are weakly negatively correlated under the multinomial; treating
|
||||
# the silences as independent is accurate to a fraction of a percent at these
|
||||
# rates and is the only closed form available.
|
||||
quiet = 1.0
|
||||
for rate in assignment:
|
||||
quiet *= 1.0 - math.exp(kills * math.log1p(-rate))
|
||||
# A table under which every item was near-certain to appear cannot have
|
||||
# produced a log that was accepted for containing a silent one. Float rounds
|
||||
# that probability to exactly 1 for an all-common assignment, so the hypothesis
|
||||
# is dropped rather than divided by zero.
|
||||
if quiet >= 1.0:
|
||||
return
|
||||
total = loglik + nothing * math.log1p(-mass) + prior - math.log1p(-quiet)
|
||||
hypotheses.append((total, tuple(assignment)))
|
||||
return
|
||||
item = order[depth]
|
||||
for index, term in candidates[item]:
|
||||
if used >> index & 1:
|
||||
continue
|
||||
is_common = index < COMMON_SLOTS
|
||||
if is_common and n_used_common == max_common:
|
||||
continue
|
||||
if not is_common and depth - n_used_common == max_ladder:
|
||||
continue
|
||||
rate = SLOT_RATES[index]
|
||||
assignment[item] = rate
|
||||
walk(depth + 1, used | 1 << index, n_used_common + is_common,
|
||||
mass + rate, loglik + term)
|
||||
assignment[item] = 0.0
|
||||
|
||||
walk(0, 0, 0, 0.0, 0.0)
|
||||
if not hypotheses:
|
||||
# Nothing in the published structure explains this log. Cannot happen for a stream
|
||||
# this module's own generator produced; falling back to the smoothed frequency
|
||||
# keeps a caller who reached here with hand-built counts from getting an exception
|
||||
# instead of a number.
|
||||
denominator = kills + 0.5 * (n + 1)
|
||||
return tuple((count + 0.5) / denominator for count in counts)
|
||||
|
||||
peak = max(score for score, _ in hypotheses)
|
||||
weight_total = 0.0
|
||||
means = [0.0] * n
|
||||
for score, rates in hypotheses:
|
||||
weight = math.exp(score - peak)
|
||||
weight_total += weight
|
||||
for i, rate in enumerate(rates):
|
||||
means[i] += weight * math.log(rate)
|
||||
# The GEOMETRIC posterior mean, exp(E[ln p]). Arithmetic would be the Bayes act for
|
||||
# squared error in probability, and on the tail that is a loss no scoring rule should
|
||||
# use: an item that could be 1/512 or 1/32768 has an arithmetic mean pinned by the top
|
||||
# rung, so the estimate is wrong by a factor of thirty whenever the deep rung is the
|
||||
# truth. Rates live on a log axis — a drop table is quoted as "one in n" — and the
|
||||
# tail term below is squared log-ratio, whose Bayes act is exactly this. That pairing
|
||||
# is what makes "some constant beats the reference" false by construction rather than
|
||||
# by luck.
|
||||
return tuple(math.exp(mean / weight_total) for mean in means)
|
||||
|
||||
|
||||
def reference_estimate(items: list[str], counts: dict[str, int], values: dict[str, int],
|
||||
kills: int) -> dict[str, float]:
|
||||
"""The reference strategy, as an estimate an agent could have written.
|
||||
|
||||
Computable from the prompt alone — that is the requirement it exists to satisfy. A
|
||||
reference that needed the true table would put the ceiling somewhere the agent cannot
|
||||
reason its way to, which is the unreachable-ceiling failure wearing a different hat.
|
||||
"""
|
||||
means = _posterior(tuple(counts.get(item, 0) for item in items),
|
||||
tuple(values[item] for item in items), kills)
|
||||
return dict(zip(items, means))
|
||||
|
||||
|
||||
def _distribution(items: list[str], estimate: dict[str, float]) -> list[float]:
|
||||
"""The estimate as a full distribution over items plus no-drop, clamped and renormalised.
|
||||
|
||||
Renormalising after the clamp keeps this a probability vector so the divergence stays a
|
||||
divergence; it shifts nothing meaningfully, since the clamp only ever moves mass on the
|
||||
order of 1e-6.
|
||||
"""
|
||||
values = [max(estimate.get(item, 0.0), MIN_PROBABILITY) for item in items]
|
||||
values.append(max(1.0 - sum(estimate.get(item, 0.0) for item in items), MIN_PROBABILITY))
|
||||
total = sum(values)
|
||||
return [value / total for value in values]
|
||||
|
||||
|
||||
def _divergence(observed: list[float], predicted: list[float]) -> float:
|
||||
return sum(p * math.log(p / q) for p, q in zip(observed, predicted) if p > 0.0)
|
||||
|
||||
|
||||
def _overclaim(predicted: list[float], observed: list[float], tail: list[int]) -> float:
|
||||
"""Reverse divergence over the unobserved items, positive part only: the nats an
|
||||
estimate spends asserting that a silent item drops more often than it does."""
|
||||
return sum(predicted[i] * math.log(predicted[i] / observed[i])
|
||||
for i in tail if predicted[i] > observed[i])
|
||||
|
||||
|
||||
def _tail_loss(truth: list[float], predicted: list[float]) -> float:
|
||||
"""Mean squared LOG-ratio on the tail: how many factors of e the estimate is out by.
|
||||
|
||||
Squared log rather than squared probability, for two reasons that are the same reason.
|
||||
A drop rate is a quantity on a log axis — one in five hundred against one in thirty
|
||||
thousand is a factor of sixty, and a difference of 0.002 — so a linear loss would grade
|
||||
the shallowest zero-count item and ignore the rest. And the Bayes act for this loss is
|
||||
exp(E[ln p]), which is exactly what `_posterior` returns, so the reference cannot be
|
||||
beaten in expectation by any point estimate whatsoever. That is the property the first
|
||||
version lacked, where a flat 0.0002 beat the reference's own tail accuracy by 19% and
|
||||
the clip reported both as 1.000.
|
||||
"""
|
||||
if not truth:
|
||||
return 0.0
|
||||
return sum(math.log(p / t) ** 2 for t, p in zip(truth, predicted)) / len(truth)
|
||||
|
||||
|
||||
def measure(stream: Stream, estimate: dict[str, float] | None) -> Outcome:
|
||||
items = stream.table.items
|
||||
observed = [stream.held[item] / stream.held_kills for item in items]
|
||||
observed.append(stream.held_nothing / stream.held_kills)
|
||||
# Clamped for the same reason the estimate is: the reverse divergence divides by the
|
||||
# continuation, and an item the held slice happened to miss would make it infinite.
|
||||
observed = [max(value, MIN_PROBABILITY) for value in observed]
|
||||
observed = [value / sum(observed) for value in observed]
|
||||
|
||||
reference = _distribution(items, reference_estimate(
|
||||
items, stream.visible, stream.table.values, stream.visible_kills))
|
||||
# The tail is the items that dropped ZERO times in the visible slice — the generator
|
||||
# guarantees there is at least one, and it is the only place an estimate cannot be read
|
||||
# off the log. Defining it by true rate instead diluted it with 1/256 items that did
|
||||
# appear, where frequency-copying is fine, and the term stopped measuring anything.
|
||||
tail = [i for i, item in enumerate(items) if stream.visible[item] == 0]
|
||||
rates = [stream.table.rates[items[i]] for i in tail]
|
||||
base = Outcome(
|
||||
valid=False,
|
||||
kl_reference=_divergence(observed, reference),
|
||||
kl_estimate=0.0,
|
||||
over_reference=_overclaim(reference, observed, tail),
|
||||
over_estimate=0.0,
|
||||
tail_reference=_tail_loss(rates, [reference[i] for i in tail]),
|
||||
tail_estimate=0.0,
|
||||
rare_items=len(tail),
|
||||
)
|
||||
if estimate is None:
|
||||
return base
|
||||
predicted = _distribution(items, estimate)
|
||||
base.valid = True
|
||||
base.kl_estimate = _divergence(observed, predicted)
|
||||
base.over_estimate = _overclaim(predicted, observed, tail)
|
||||
base.tail_estimate = _tail_loss(rates, [predicted[i] for i in tail])
|
||||
return base
|
||||
|
||||
|
||||
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
|
||||
_FRACTION = re.compile(r"^\s*(\d+(?:\.\d+)?)\s*(?:/|\s+in\s+)\s*(\d+(?:\.\d+)?)\s*$")
|
||||
|
||||
|
||||
def _rate(value: object) -> float | None:
|
||||
"""A rate as a number, or as the fraction an agent that spotted the grid would write.
|
||||
`1/128` and `1 in 128` are the natural way to state a drop rate and refusing them would
|
||||
punish exactly the reasoning the environment is trying to reward."""
|
||||
if isinstance(value, bool):
|
||||
return None
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
if isinstance(value, str):
|
||||
match = _FRACTION.match(value)
|
||||
if match:
|
||||
numerator, denominator = float(match.group(1)), float(match.group(2))
|
||||
return numerator / denominator if denominator else None
|
||||
try:
|
||||
return float(value.strip())
|
||||
except ValueError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def parse_estimate(reply: str, items: list[str]) -> dict[str, float] | None:
|
||||
"""The last JSON object in the reply, keyed by item name.
|
||||
|
||||
None means "no usable estimate" and scores what doing nothing scores — a reply that
|
||||
cannot be parsed never raises, and an estimate that is not a sub-distribution is
|
||||
rejected here rather than being quietly renormalised into a valid one, because
|
||||
renormalising would make inflating every rate free.
|
||||
"""
|
||||
blocks = _BLOCK.findall(reply or "")
|
||||
raw = blocks[-1] if blocks else (reply or "")
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
if isinstance(parsed, list):
|
||||
pairs = {}
|
||||
for row in parsed:
|
||||
if isinstance(row, dict):
|
||||
name = row.get("item", row.get("name"))
|
||||
rate = row.get("rate", row.get("probability", row.get("p")))
|
||||
if isinstance(name, str):
|
||||
pairs[name] = rate
|
||||
parsed = pairs
|
||||
if not isinstance(parsed, dict):
|
||||
return None
|
||||
|
||||
lookup = {item.lower(): item for item in items}
|
||||
estimate: dict[str, float] = {}
|
||||
for key, value in parsed.items():
|
||||
item = lookup.get(str(key).strip().lower())
|
||||
if item is None:
|
||||
continue
|
||||
rate = _rate(value)
|
||||
if rate is None or not math.isfinite(rate) or rate < 0.0 or rate > 1.0:
|
||||
return None
|
||||
estimate[item] = rate
|
||||
if not estimate or sum(estimate.values()) > 1.0 + SUM_TOLERANCE:
|
||||
return None
|
||||
return estimate
|
||||
@@ -0,0 +1,252 @@
|
||||
"""The monster, its drop table, and the kills that came out of it.
|
||||
|
||||
A task is one monster. Its table is a set of invented items, each with a rate the agent
|
||||
never sees, and the log is a sample from it: a visible slice the agent reads and a held-out
|
||||
continuation it is graded on. Both come out of one kill stream, so the graded kills are
|
||||
literally the *next* ones the monster would have dropped — not a differently-seeded
|
||||
population that could differ in ways the agent could not have anticipated.
|
||||
|
||||
Four properties of the generator are load-bearing, and none is decoration:
|
||||
|
||||
A LADDER WITH NO REPEATS. Rates are not merely "structured", they are drawn WITHOUT
|
||||
REPLACEMENT from two published sets: commons are distinct multiples of 4/128, and the rest
|
||||
are distinct unit fractions from 1/128 down to 1/24576. Distinctness is the first half of
|
||||
why the tail is learnable. Nothing in a kill log distinguishes one item that dropped zero
|
||||
times from another — any estimator that is a function of (count, kills) alone must give them
|
||||
the same number, and if that number is also the same across TASKS then a memorised constant
|
||||
is a perfect tail estimate and the term is inert. That is exactly what the first version of
|
||||
this environment did: its reference gave every zero-count item (0+0.5)/(kills+0.5(n+1)), a
|
||||
value with four distinct settings across ninety-six tasks, so a reply that never opened the
|
||||
kill log scored 1.000 on the tail. With no repeats, the rungs the OBSERVED items occupy are
|
||||
the rungs the unobserved ones cannot: a table whose counts pin 1/128 and 1/384 leaves its
|
||||
silent item somewhere below, and one that pins nothing above 1/3072 leaves it somewhere much
|
||||
higher.
|
||||
|
||||
Distinctness alone was not enough, and the number that says so is 8%: with the ladder and
|
||||
the elimination but no price, the reference beat the best memorised constant on the tail
|
||||
loss by eight percent, and a frequency copy with a constant floor held 0.86 of the term.
|
||||
Rungs eliminate, but two SILENT items are still interchangeable, and the irreducible spread
|
||||
inside that pair swamped everything the elimination revealed. That is what VALUE_SCALE
|
||||
below is for.
|
||||
|
||||
A PRICE THAT RANKS WHAT THE LOG CANNOT. Every item carries a shop value, lognormal around
|
||||
SCALE * (1/rate) ** EXPONENT — rare drops are worth more, which is a fact about the genre
|
||||
rather than anybody's copy. It is the only channel that separates two silent items, and
|
||||
separating them is what turns the tail from a lottery into an inference: with it the
|
||||
reference beats the best memorised constant on the tail by 4.75x rather than by 8%, and the
|
||||
frequency-copy-with-a-floor attack falls from 0.86 of the term to 0.28. The scatter is
|
||||
deliberate and it is large — sigma 0.55 in log value against 3x rung spacing — so the price
|
||||
ranks the candidates and never decides them, and an agent that reads only the price scores
|
||||
0.46 where one that reads price, counts and leftover rungs together scores 1.000.
|
||||
|
||||
A ZERO-COUNT ITEM, ALWAYS. The kill stream is rejected and redrawn until at least one item
|
||||
on the list dropped zero times in the visible slice. That case is the entire environment:
|
||||
the item list says the item exists, the log says it never appeared, and an estimate of 0 is
|
||||
an infinitely strong claim about something the graded kills will demonstrate is false. A
|
||||
task where every item happened to show up does not pose the question, so it is not used.
|
||||
The rejection is part of the model — `estimate._posterior` divides it back out,
|
||||
because conditioning on "some item was silent" is evidence about the table and an estimator
|
||||
that ignores it is leaving information in the prompt.
|
||||
|
||||
HELD-OUT KILLS UNTIL EVERY ITEM APPEARS. The item list is a promise that all of these drop;
|
||||
the graded slice has to keep it, or an estimate of zero for the rarest item costs nothing.
|
||||
It is also what keeps the counterweight finite: `estimate.restraint` divides by what the
|
||||
continuation produced, and an item the continuation never produced would make it infinite.
|
||||
|
||||
No wiki content: every item, monster and word here is invented. The MECHANICS are the
|
||||
borrowed part, and mechanics are facts.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import bisect
|
||||
import math
|
||||
import random
|
||||
from dataclasses import dataclass
|
||||
|
||||
ADJECTIVES = [
|
||||
"Ashen", "Brackish", "Cindered", "Duskbound", "Ember", "Frostbit", "Gilded", "Hollow",
|
||||
"Ivory", "Jagged", "Kelpish", "Lodestone", "Murkwrought", "Nettled", "Pitchblack",
|
||||
"Quarried", "Saltworn", "Tarnished", "Umbral", "Verdigris",
|
||||
]
|
||||
NOUNS = [
|
||||
"talon", "sigil", "vertebra", "censer", "ingot", "lantern", "chitin", "warhorn",
|
||||
"tessera", "phial", "reliquary", "scale", "cog", "wick", "grimoire", "spur",
|
||||
"anvil-shard", "orb", "tabard", "quill",
|
||||
]
|
||||
MONSTER_PREFIX = ["Marrow", "Gloom", "Rime", "Slagborn", "Fenwater", "Barrow", "Cinder", "Hush"]
|
||||
MONSTER_KIND = ["Warden", "Basilisk", "Revenant", "Colossus", "Hierophant", "Drake", "Herald", "Ossifier"]
|
||||
|
||||
# The common grid, and it is coarse on purpose. Adjacent numerators sit 33-50% apart, so a
|
||||
# few hundred kills can actually tell them apart — a continuous numerator would put the
|
||||
# notches inside the sampling error, the structure would be real but unresolvable, and
|
||||
# "spot the grid" would be a slogan rather than a strategy.
|
||||
COMMON_DENOMINATOR = 128
|
||||
COMMON_NUMERATORS = (8, 12, 16, 24, 32, 40, 48)
|
||||
|
||||
# The rare ladder: unit fractions spaced by roughly a factor of two. No sample of 600 kills
|
||||
# resolves 1/1024 from 1/2048 on its own evidence, and that is the point — down here the
|
||||
# information is in which rungs the OTHER items have already taken.
|
||||
LADDER_DENOMINATORS = (128, 384, 1024, 3072, 8192, 24576)
|
||||
|
||||
# How many of each. Both are published to the agent, because the reference estimator uses
|
||||
# them and a reference the agent cannot compute is an unreachable ceiling wearing a hat.
|
||||
N_COMMON = (3, 4)
|
||||
N_LADDER = (4, 5, 6)
|
||||
|
||||
# The price law, and it is the reason the tail is a skill rather than a lottery.
|
||||
#
|
||||
# Two items that both dropped zero times are EXCHANGEABLE on the kill log: no statistic of
|
||||
# the counts distinguishes them, so every estimator must give them the same number, and
|
||||
# the reference's advantage over a memorised constant collapses to whatever the free-rung
|
||||
# set alone reveals. Measured, that was four to eight percent — the reference beat the best
|
||||
# constant on the tail by less than a tenth, which is not an environment, it is a coin.
|
||||
#
|
||||
# A price breaks the tie, and it is the OSRS-shaped way to break it: rare drops are worth
|
||||
# more. `value` is lognormal around SCALE * (1/rate) ** EXPONENT, so it ranks the silent
|
||||
# items without deciding them — at 3x rung spacing and this sigma, the price alone gets an
|
||||
# adjacent pair the right way round about six times in seven and gets nothing for free two
|
||||
# rungs down. The agent has to combine it with the counts and the leftover rungs, which is
|
||||
# the entire task.
|
||||
VALUE_SCALE = 15.0
|
||||
VALUE_EXPONENT = 0.8
|
||||
VALUE_LOG_SD = 0.55
|
||||
|
||||
# The share of kills that drop something. Bounded away from 1 so "no drop" is always a
|
||||
# substantial category: it is where the mass an agent does not assign to items has to go,
|
||||
# and it is what makes the sum-to-at-most-one constraint bite instead of being cosmetic.
|
||||
MIN_ITEM_MASS = 0.55
|
||||
MAX_ITEM_MASS = 0.90
|
||||
|
||||
# Held-out kills stop when every item has been seen; this caps the walk in case a table
|
||||
# with a 1/24576 tertiary draws a very long silence. Hitting it costs an item's evidence
|
||||
# rather than anything worse, and at twelve times the requested slice it is a 1e-8 event.
|
||||
HELD_OUT_CEILING = 12
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DropTable:
|
||||
"""One monster's table. `rates` is in DISPLAY order, deliberately shuffled: listing
|
||||
items by rarity would hand over the ordering the agent is being asked to recover."""
|
||||
|
||||
monster: str
|
||||
rates: dict[str, float]
|
||||
values: dict[str, int]
|
||||
|
||||
@property
|
||||
def items(self) -> list[str]:
|
||||
return list(self.rates)
|
||||
|
||||
@property
|
||||
def nothing(self) -> float:
|
||||
return 1.0 - sum(self.rates.values())
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Stream:
|
||||
"""One task: the table, and both slices as tallies.
|
||||
|
||||
Tallies rather than kill sequences because the tally is the sufficient statistic — the
|
||||
order kills came in carries no information about the table, so serialising 5,000 lines
|
||||
would cost tokens and teach nothing.
|
||||
"""
|
||||
|
||||
table: DropTable
|
||||
visible: dict[str, int]
|
||||
visible_kills: int
|
||||
held: dict[str, int]
|
||||
held_kills: int
|
||||
|
||||
@property
|
||||
def visible_nothing(self) -> int:
|
||||
return self.visible_kills - sum(self.visible.values())
|
||||
|
||||
@property
|
||||
def held_nothing(self) -> int:
|
||||
return self.held_kills - sum(self.held.values())
|
||||
|
||||
|
||||
def _names(rng: random.Random, count: int) -> list[str]:
|
||||
seen: list[str] = []
|
||||
while len(seen) < count:
|
||||
name = f"{rng.choice(ADJECTIVES)} {rng.choice(NOUNS)}"
|
||||
if name not in seen:
|
||||
seen.append(name)
|
||||
return seen
|
||||
|
||||
|
||||
def build_table(seed: int) -> DropTable:
|
||||
"""The table alone, from the seed. Stable across kill-stream redraws so that a task's
|
||||
identity is its monster, not whichever sample happened to be accepted.
|
||||
|
||||
`sample` rather than `choices` in both tiers: with replacement, two items could share a
|
||||
rung, the leftover-rung deduction would be unsound, and the tail would go back to being
|
||||
a constant nobody has to read the log for.
|
||||
"""
|
||||
rng = random.Random(seed)
|
||||
monster = f"{rng.choice(MONSTER_PREFIX)} {rng.choice(MONSTER_KIND)}"
|
||||
n_common = rng.choice(N_COMMON)
|
||||
n_ladder = rng.choice(N_LADDER)
|
||||
names = _names(rng, n_common + n_ladder)
|
||||
|
||||
# Rejection on total mass, with the tier COUNTS fixed outside the loop: redrawing the
|
||||
# shape as well would make the split's prior depend on how often that shape lands in
|
||||
# the mass window, and the reference's prior — which is P(n_common) * P(n_ladder) —
|
||||
# would then be subtly wrong on every task.
|
||||
while True:
|
||||
numerators = rng.sample(COMMON_NUMERATORS, n_common)
|
||||
denominators = rng.sample(LADDER_DENOMINATORS, n_ladder)
|
||||
rates = [n / COMMON_DENOMINATOR for n in numerators] + [1.0 / d for d in denominators]
|
||||
if MIN_ITEM_MASS <= sum(rates) <= MAX_ITEM_MASS:
|
||||
break
|
||||
order = list(zip(names, rates))
|
||||
rng.shuffle(order)
|
||||
table = dict(order)
|
||||
# Prices drawn AFTER the shuffle so the display order carries nothing, and from the
|
||||
# rate rather than the tier, so a common at 8/128 and a ladder item at 1/128 are priced
|
||||
# identically — the price is evidence about the RATE, never about which tier an item
|
||||
# came from.
|
||||
values = {
|
||||
name: max(1, round(VALUE_SCALE * (1.0 / rate) ** VALUE_EXPONENT
|
||||
* math.exp(VALUE_LOG_SD * rng.gauss(0.0, 1.0))))
|
||||
for name, rate in table.items()
|
||||
}
|
||||
return DropTable(monster=monster, rates=table, values=values)
|
||||
|
||||
|
||||
def _tally(rng: random.Random, table: DropTable, kills: int, until_covered: bool) -> tuple[dict[str, int], int]:
|
||||
"""Roll kills, counting drops. With `until_covered`, keep rolling past `kills` until
|
||||
every item has appeared — see the module docstring for why the graded slice owes the
|
||||
item list that."""
|
||||
items = table.items
|
||||
cumulative, running = [], 0.0
|
||||
for item in items:
|
||||
running += table.rates[item]
|
||||
cumulative.append(running)
|
||||
counts = {item: 0 for item in items}
|
||||
rolled, ceiling = 0, kills * HELD_OUT_CEILING
|
||||
while rolled < kills or (until_covered and not all(counts.values()) and rolled < ceiling):
|
||||
index = bisect.bisect(cumulative, rng.random())
|
||||
if index < len(items):
|
||||
counts[items[index]] += 1
|
||||
rolled += 1
|
||||
return counts, rolled
|
||||
|
||||
|
||||
def build_slices(seed: int, visible_kills: int, held_out_kills: int) -> Stream:
|
||||
"""The two slices of one task, from one seed.
|
||||
|
||||
The kill stream is redrawn — table held fixed — until the visible slice has a
|
||||
zero-count item, because a task where every item showed up does not pose the question
|
||||
this environment exists to ask. Rejection is on the visible slice only; the held-out
|
||||
continuation is whatever the accepted stream produced next.
|
||||
"""
|
||||
table = build_table(seed)
|
||||
for attempt in range(256):
|
||||
rng = random.Random((seed << 9) + attempt)
|
||||
visible, _ = _tally(rng, table, visible_kills, until_covered=False)
|
||||
if all(visible.values()):
|
||||
continue
|
||||
held, held_kills = _tally(rng, table, held_out_kills, until_covered=True)
|
||||
return Stream(table, visible, visible_kills, held, held_kills)
|
||||
raise RuntimeError(f"no stream with an unobserved item for seed {seed}")
|
||||
@@ -0,0 +1,205 @@
|
||||
"""drop-table-inference: recover a drop table from a kill log, graded on the next kills.
|
||||
|
||||
The agent gets a monster's item list — each item with a price and a drop count over a few
|
||||
hundred kills — and never gets the rates. It returns an estimate, and the estimate is scored
|
||||
against the continuation of that same kill stream, thirty times as long and never shown.
|
||||
|
||||
The reward is normalised against a REFERENCE STRATEGY rather than an absolute, and that is
|
||||
not a softening. Sampling error is irreducible: nobody recovers 1/24576 from 1,200 kills, so
|
||||
a reward measured against the true table has a ceiling no amount of reasoning reaches, which
|
||||
is the dead-component failure rule 3 exists to catch.
|
||||
|
||||
WHICH reference is the design, and the first version of this environment got it wrong. It
|
||||
used a closed form — the posterior mean under a Jeffreys prior — and a closed form is a
|
||||
function of (count, kills) alone, so every item that dropped zero times got the same number
|
||||
on every task. A reply that never opened the kill log scored 1.000 on the term built to
|
||||
catch frequency-copying, and frequency-copying plus that one memorised constant reached 90%
|
||||
of the ceiling. The reference here is the posterior over the generator's own hypothesis
|
||||
space: items assigned to distinct rungs of a published ladder, weighted by the multinomial
|
||||
likelihood of the counts AND by each item's price, filtered by the published mass window,
|
||||
and corrected for the fact that the log was selected to contain a silent item. Everything it
|
||||
uses is in the prompt. Nothing it uses is a constant.
|
||||
|
||||
The structure below is published deliberately. A reference the agent cannot compute is an
|
||||
unreachable ceiling wearing a hat, and rule 3 forbids those. What is not published is the
|
||||
answer — which item sits on which rung — and that is a joint inference over every item at
|
||||
once, because no two items share a rung and the leftovers are the only evidence about the
|
||||
ones the log never showed.
|
||||
|
||||
Three terms:
|
||||
|
||||
fit forward divergence ratio against the held-out continuation. Dominated by the
|
||||
common items, because forward KL weights every term by the truth's own mass.
|
||||
rare squared log-ratio on the items that dropped ZERO times, against the true rate,
|
||||
normalised by the reference's own error and multiplied by `restraint`.
|
||||
gate binary: valid, and as good as the reference on all three.
|
||||
|
||||
and the counterweight, `restraint`, which multiplies into `rare` rather than sitting beside
|
||||
it: the nats an estimate spends claiming that a silent item drops more often than the
|
||||
continuation showed. Forward KL rewards covering what you cannot see; restraint prices the
|
||||
covering. Measured, over a uniform hedge mixed into frequency-copying at weight a: fit rises
|
||||
0.136 -> 0.186 -> 0.198 as a goes 0 -> 0.0005 -> 0.002 while restraint falls 1.000 -> 0.996
|
||||
-> 0.443. There is no setting of a where both are at their best, which is what rule 2 asks
|
||||
for and what the first version did not have.
|
||||
|
||||
Probabilities must sum to at most 1; the remainder is the chance of no drop, and it is not
|
||||
small. An estimate that sums past 1 is not a distribution and scores zero.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pydantic import Field
|
||||
|
||||
import verifiers.v1 as vf
|
||||
|
||||
from drop_table_inference.estimate import measure, parse_estimate
|
||||
from drop_table_inference.table import (
|
||||
COMMON_DENOMINATOR,
|
||||
COMMON_NUMERATORS,
|
||||
LADDER_DENOMINATORS,
|
||||
MAX_ITEM_MASS,
|
||||
MIN_ITEM_MASS,
|
||||
N_COMMON,
|
||||
N_LADDER,
|
||||
VALUE_EXPONENT,
|
||||
VALUE_LOG_SD,
|
||||
VALUE_SCALE,
|
||||
build_slices,
|
||||
)
|
||||
|
||||
|
||||
def _english(counts: tuple[int, ...]) -> str:
|
||||
"""The tier counts as a person would write them: 3 or 4, and 4, 5 or 6. Spelled out
|
||||
rather than given as a range because the agent needs the exact set — the reference sums
|
||||
over the splits it allows, and a split it does not know about is one it cannot weigh."""
|
||||
return " or ".join(filter(None, (", ".join(str(n) for n in counts[:-1]), str(counts[-1]))))
|
||||
|
||||
|
||||
_COMMONS = ", ".join(f"{n}/{COMMON_DENOMINATOR}" for n in COMMON_NUMERATORS)
|
||||
_LADDER = ", ".join(f"1/{d}" for d in LADDER_DENOMINATORS)
|
||||
|
||||
SYSTEM = f"""You are estimating a monster's drop table from a kill log.
|
||||
|
||||
You are given every item on the table, its shop value in gp, and how many times it dropped.
|
||||
You are NOT given the rates. Return your estimate as one JSON object in a ```json code
|
||||
block, mapping each item name to its probability per kill:
|
||||
|
||||
```json
|
||||
{{"Ashen talon": 0.125, "Ivory sigil": "1/3072"}}
|
||||
```
|
||||
|
||||
A value may be a decimal or a fraction like "1/3072". Give at least four significant
|
||||
figures — your estimate is compared against a well-reasoned one and rounding throws that
|
||||
comparison away.
|
||||
|
||||
How these tables are built, exactly:
|
||||
|
||||
- {_english(N_COMMON)} COMMON items, each at one of {_COMMONS}, no two the same.
|
||||
- {_english(N_LADDER)} RARE items, each at one of {_LADDER}, no two the same.
|
||||
- The rates sum to between {MIN_ITEM_MASS:.2f} and {MAX_ITEM_MASS:.2f}. The remainder is
|
||||
the chance a kill drops nothing, and it is not small.
|
||||
- An item's value in gp is lognormal around {VALUE_SCALE:g} * (1/rate) ** {VALUE_EXPONENT:g},
|
||||
with standard deviation {VALUE_LOG_SD:g} in natural log. Rarer is worth more, with real
|
||||
scatter: the price ranks the items the counts cannot identify, it does not name them.
|
||||
- This log was chosen because at least one item dropped ZERO times in it. Every item
|
||||
listed does drop. A rate of 0 is the worst answer available for any of them.
|
||||
|
||||
Because no two items share a rate, the rates the observed items take are rates the silent
|
||||
ones cannot. That, the price, and the count are the whole of the evidence.
|
||||
|
||||
You are graded on the next several tens of thousands of kills from this same monster, which
|
||||
you have not seen — on how well your table predicts them, and separately on how close you
|
||||
got for the items this log never showed you."""
|
||||
|
||||
|
||||
class DropTableData(vf.TaskData):
|
||||
seed: int
|
||||
"""Rebuilds the table and both slices exactly; no rate is ever serialized here."""
|
||||
visible_kills: int
|
||||
held_out_kills: int
|
||||
|
||||
|
||||
class DropTableTask(vf.Task[DropTableData]):
|
||||
@vf.stop
|
||||
async def single_turn(self, trace: vf.Trace) -> bool:
|
||||
return trace.num_turns >= 1
|
||||
|
||||
@vf.metric
|
||||
async def scan(self, trace: vf.Trace) -> dict[str, float]:
|
||||
"""Rebuild the stream and score the estimate once; every reward reads this."""
|
||||
stream = build_slices(self.data.seed, self.data.visible_kills, self.data.held_out_kills)
|
||||
estimate = parse_estimate(trace.last_reply, stream.table.items)
|
||||
outcome = measure(stream, estimate)
|
||||
return {
|
||||
"fit": outcome.fit,
|
||||
"rare": outcome.rare,
|
||||
"clean": float(outcome.clean),
|
||||
"valid": float(outcome.valid),
|
||||
"restraint": outcome.restraint,
|
||||
"raw_fit": outcome.raw_fit,
|
||||
"raw_rare": outcome.raw_rare,
|
||||
"kl_estimate": outcome.kl_estimate,
|
||||
"kl_reference": outcome.kl_reference,
|
||||
"tail_estimate": outcome.tail_estimate,
|
||||
"tail_reference": outcome.tail_reference,
|
||||
"rare_items": float(outcome.rare_items),
|
||||
}
|
||||
|
||||
@vf.reward(weight=0.45)
|
||||
async def fit(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("fit", 0.0)
|
||||
|
||||
@vf.reward(weight=0.35)
|
||||
async def rare(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("rare", 0.0)
|
||||
|
||||
@vf.reward(weight=0.20)
|
||||
async def gate(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("clean", 0.0)
|
||||
|
||||
|
||||
class DropTableConfig(vf.TasksetConfig):
|
||||
num_tasks: int = Field(64, ge=1)
|
||||
visible_kills: int = Field(1_200, ge=50)
|
||||
"""Kills the agent reads. Long enough that the common tier is resolvable against the
|
||||
grid, short enough that the deep rungs show up zero times — which is the case the
|
||||
environment exists to pose."""
|
||||
held_out_kills: int = Field(40_000, ge=200)
|
||||
"""Kills it is graded on and never sees. Thirty times the visible slice, so the graded
|
||||
frequencies are the estimate's problem rather than the sample's, and long enough that a
|
||||
1/24576 item is expected to appear."""
|
||||
|
||||
|
||||
class DropTableTaskset(vf.Taskset[DropTableTask, DropTableConfig]):
|
||||
SEED_BASE = 20_000
|
||||
|
||||
def load(self) -> list[DropTableTask]:
|
||||
tasks = []
|
||||
for i in range(self.config.num_tasks):
|
||||
seed = self.SEED_BASE + i
|
||||
stream = build_slices(seed, self.config.visible_kills, self.config.held_out_kills)
|
||||
width = max(len(item) for item in stream.table.items) + 2
|
||||
rows = "\n".join(
|
||||
f" {item:<{width}}{stream.table.values[item]:>9}{stream.visible[item]:>8}"
|
||||
for item in stream.table.items
|
||||
)
|
||||
rows += f"\n {'(no drop)':<{width}}{'':>9}{stream.visible_nothing:>8}"
|
||||
tasks.append(
|
||||
DropTableTask(
|
||||
DropTableData(
|
||||
idx=i,
|
||||
name=f"{stream.table.monster.lower().replace(' ', '-')}-{seed}",
|
||||
prompt=(
|
||||
f"{stream.table.monster} — {stream.visible_kills} kills logged.\n\n"
|
||||
f" {'item':<{width}}{'value':>9}{'drops':>8}\n{rows}\n\n"
|
||||
f"Estimate the drop rate of every item."
|
||||
),
|
||||
system_prompt=SYSTEM,
|
||||
seed=seed,
|
||||
visible_kills=self.config.visible_kills,
|
||||
held_out_kills=self.config.held_out_kills,
|
||||
),
|
||||
self.config.task,
|
||||
)
|
||||
)
|
||||
return tasks
|
||||
@@ -0,0 +1,13 @@
|
||||
[project]
|
||||
name = "drop-table-inference"
|
||||
version = "0.1.0"
|
||||
description = "drop-table-inference — estimate a drop table from a kill log, graded on the kills you never saw."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["drop_table_inference"]
|
||||
+3488
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,3 @@
|
||||
from grand_exchange.taskset import ExchangeTaskset
|
||||
|
||||
__all__ = ["ExchangeTaskset"]
|
||||
@@ -0,0 +1,525 @@
|
||||
"""Executing the agent's orders against the held-out window, and scoring the result.
|
||||
|
||||
Realised profit is unbounded above and has no theoretical optimum the data supports, so
|
||||
scoring it against one would put the ceiling out of reach — which house rule 3 forbids.
|
||||
Every ratio here is therefore taken against a REFERENCE STRATEGY (`reference_orders`)
|
||||
that is computed from the visible half and nothing else, and executed through the same
|
||||
engine as the agent's orders. That is what makes 1.000 reachable by construction, and it
|
||||
is honest: the model is asked to match a strategy available to anything that can read the
|
||||
prompt, not to beat hindsight it was never shown.
|
||||
|
||||
A reference in the denominator has its own failure mode, and the first cut of this file had
|
||||
it. If the ceiling is the reference EXACTLY, then above the clip excess profit is worth
|
||||
nothing while every deviation still costs, so the reward stops being a profit metric and
|
||||
becomes an imitation score for a strategy whose constants are nowhere in the prompt. It was
|
||||
measurable: a wider sell band earned sixteen percent more gp and scored 0.812, and one
|
||||
wasted unit of capital with identical realised profit cost 0.146. Two changes answer it.
|
||||
The reference's own constants are now the profit-maximising point of its family, swept on
|
||||
seeds no task is built from. And the ceiling is a BAND — TARGET_SHARE of the reference —
|
||||
so the whole plateau around it, and everything above it, scores 1.000.
|
||||
|
||||
Three numbers come out of a run:
|
||||
|
||||
profit_ratio clip(realised / (TARGET_SHARE x reference realised), 0, 1). No orders is
|
||||
zero, and a round trip that does not clear the tax is negative and also
|
||||
zero.
|
||||
efficiency realised profit per coin LOCKED behind an offer, against the reference's
|
||||
own. This is where the volume limit bites: a fat margin on an item
|
||||
carrying three thousand gp a tick is an offer that sits there, and a
|
||||
sitting offer is capital a better item did not get. It also catches the
|
||||
plan that fills perfectly at prices that were never worth reaching, which
|
||||
a plain fill rate scores as discipline.
|
||||
clean cleared the bar on both. Binary, because a trading run either was worth
|
||||
doing or was not.
|
||||
|
||||
`efficiency` is never a reward on its own — one tiny order that traded perfectly would earn
|
||||
it in full for near-inaction, which is the free-points defect schema-migration shipped with.
|
||||
It multiplies into profit instead, so it can only ever qualify profit that exists.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
import statistics
|
||||
from dataclasses import dataclass
|
||||
|
||||
from grand_exchange.market import (
|
||||
DUMP_BASE,
|
||||
DUMP_CAP,
|
||||
DUMP_IMPACT,
|
||||
FILL_SHARE,
|
||||
TAX,
|
||||
Item,
|
||||
Market,
|
||||
build_market,
|
||||
)
|
||||
|
||||
MAX_ORDERS = 12
|
||||
BUY_BAND = 0.05
|
||||
SELL_BAND = 0.05
|
||||
"""The band the reference buys under its anchor and sells over it.
|
||||
|
||||
These two numbers are the reward's denominator, so where they sit is not a taste question.
|
||||
They are the profit-maximising point of the reference's own family, found by sweeping
|
||||
(buy band, sell band, crossings threshold, purse cap, size multiplier) over 160 baskets
|
||||
drawn from SEED 400,000 ONWARD — a range no task is ever built from — and they are within
|
||||
three percent of the sweep's best cell over the whole plateau.
|
||||
|
||||
The first cut used a sell band of 0.01, and that was the defect a reviewer found: a strategy
|
||||
with a wider sell band earned sixteen percent more gp than the reference and scored 0.812,
|
||||
because every ratio here divides by the reference and clips at one. When the denominator is
|
||||
a strategy that leaves money on the table, the reward's argmax is the denominator's
|
||||
hyperparameters rather than the profit. Tuning the reference to its own family's optimum is
|
||||
half the fix; TARGET_SHARE below is the other half."""
|
||||
|
||||
TARGET_SHARE = 0.90
|
||||
"""What counts as a full score, as a share of the reference's realised profit.
|
||||
|
||||
A ratio that divides by the reference EXACTLY makes the ceiling a single point, and a point
|
||||
ceiling turns the reward into an imitation score: excess profit is worth nothing above the
|
||||
clip while any deviation is punished, so the gradient near the top points at replicating a
|
||||
strategy the prompt does not contain rather than at making money. Measured on the first cut,
|
||||
one wasted unit of capital with byte-identical realised profit cost 0.146 of total reward,
|
||||
and a one-gp change to every buy limit that EARNED 463 gp a basket more cost 0.195.
|
||||
|
||||
Normalising against nine tenths of the reference makes the ceiling a BAND. Everything from
|
||||
"ten percent short of the reference" upward scores 1.000, so the whole plateau around the
|
||||
reference — and everything above it — is the argmax, and a rounding difference costs
|
||||
nothing. The reference still scores exactly 1.000, so house rule 3 is unchanged."""
|
||||
|
||||
MIN_CROSSINGS = 0.25
|
||||
"""Below this share of ticks crossing the mean, the reference will not trade the item at
|
||||
all. Over 4,000 baskets a reverting series recrosses its own mean 0.362 of the time and a
|
||||
random walk 0.113, and the threshold sits in the gap: it lets 4.2% of walks through and
|
||||
turns away 4.1% of reverters. This one line is the difference between a strategy and a
|
||||
superstition — without it the reference buys the widest-swinging line in the basket, which
|
||||
is exactly the line that has no anchor to revert to."""
|
||||
|
||||
MAX_ITEM_SHARE = 0.40
|
||||
"""No more than this share of the purse behind one name. The anchor is an ESTIMATE, and one
|
||||
bad estimate carrying the whole purse is the only way this strategy loses money over a
|
||||
window. Spread over the 3.4 names it trades on average it profits in 3,976 of 4,000
|
||||
baskets, and `viable_market` refuses the other twenty-four."""
|
||||
|
||||
VIABILITY_TRIES = 64
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Order:
|
||||
item: str
|
||||
quantity: int
|
||||
buy: int
|
||||
sell: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class Fills:
|
||||
"""What one plan actually did in the window."""
|
||||
|
||||
realised: float = 0.0
|
||||
"""Closing capital minus starting capital: the only number that pays."""
|
||||
paper: float = 0.0
|
||||
"""What the orders CLAIM, filled in full at the limit prices. Not part of any reward —
|
||||
it exists so `arith_ok` can ask whether the agent's stated expected profit matches the
|
||||
sum its own orders imply."""
|
||||
bought: int = 0
|
||||
sold: int = 0
|
||||
planned: int = 0
|
||||
spent: float = 0.0
|
||||
dumped: int = 0
|
||||
orders: int = 0
|
||||
committed: float = 0.0
|
||||
"""Coins actually locked behind offers. Capital left idle earns nothing, which is the
|
||||
only reason allocation is a decision."""
|
||||
offered: int = 0
|
||||
"""Units the purse could fund, against `planned` units asked for."""
|
||||
|
||||
@property
|
||||
def roc(self) -> float:
|
||||
"""Realised profit per coin locked behind an offer.
|
||||
|
||||
This is the counterweight the profit term is qualified by, and it replaced a plain
|
||||
fill rate. A fill rate only asks whether an offer was reachable; it says nothing
|
||||
about whether reaching it was worth doing, so a plan that bought badly at prices it
|
||||
was always going to reach scored as disciplined as one that bought well. Return on
|
||||
the coins actually committed asks both questions at once, and it cannot be won by
|
||||
committing nothing: it multiplies into profit, which needs the coins.
|
||||
"""
|
||||
if self.committed <= 0.0:
|
||||
return 0.0
|
||||
return self.realised / self.committed
|
||||
|
||||
@property
|
||||
def conversion(self) -> float:
|
||||
"""Of the coins locked behind offers, the share that actually bought stock.
|
||||
|
||||
The first version of this was realised profit over CLAIMED profit, and it was
|
||||
gameable in one line: set the sell limit a hair over the buy limit and the claim
|
||||
goes to nearly nothing, so the ratio goes to nearly anything. Coins converted
|
||||
cannot be talked down — the only way to raise it is to offer a quantity the volume
|
||||
supports at a price the market reaches, which is the lesson the number is for.
|
||||
"""
|
||||
if self.committed <= 0.0:
|
||||
return 0.0
|
||||
return min(1.0, self.spent / self.committed)
|
||||
|
||||
|
||||
def paper_profit(orders: list[Order]) -> float:
|
||||
"""The profit the orders assert, at their own limit prices, net of tax on the sale.
|
||||
Deliberately the naive multiplication — it is the sum the agent is claiming, and
|
||||
`arith_ok` checks whether the agent can do it."""
|
||||
return sum(o.quantity * (o.sell * (1.0 - TAX) - o.buy) for o in orders)
|
||||
|
||||
|
||||
def execute(market: Market, orders: list[Order]) -> Fills:
|
||||
"""Walk the held-out ticks once, filling what the purse and the book support.
|
||||
|
||||
Rules, all quoted to the agent:
|
||||
- COINS ARE LOCKED WHEN THE OFFER IS PLACED, quantity times buy price, in the order
|
||||
the orders were listed. This is the exchange's own behaviour and it is what makes
|
||||
the task an allocation: an offer that never fills has still spent the capital a
|
||||
better offer needed, so ordering the whole basket at its buy limit is not free.
|
||||
It is also why a buy limit of a billion buys nothing — nobody can fund that offer.
|
||||
- a buy fills at the tick's price when it is at or under the limit; a sell fills at
|
||||
the tick's price when it is at or over. You are never filled at your own limit
|
||||
when the market is better than it, because that is not how a limit works.
|
||||
- at most FILL_SHARE of a tick's volume per side, per item, and only on the ticks the
|
||||
offer was eligible on — flow you were not in the market for is not yours to bank.
|
||||
- stock bought this tick cannot be sold this tick. Without that, a buy limit above a
|
||||
sell limit is a free round trip on a single price.
|
||||
- sale proceeds land in the purse but fund nothing: every offer was placed up front.
|
||||
- stock still held at the close is forced out at DUMP_BASE under the last price plus
|
||||
DUMP_IMPACT for every tick's worth of the item's median volume being pushed through.
|
||||
Depth is taken from the window the stock is actually being sold into, which is the
|
||||
window the agent estimated from the visible median.
|
||||
"""
|
||||
fills = Fills(orders=len(orders), planned=sum(o.quantity for o in orders))
|
||||
fills.paper = paper_profit(orders)
|
||||
|
||||
# Placement: fund each offer in submission order out of one purse.
|
||||
purse = float(market.capital)
|
||||
live: list[tuple[Order, Item, int]] = []
|
||||
for order in orders:
|
||||
item = market.item(order.item)
|
||||
if item is None or order.buy <= 0:
|
||||
continue
|
||||
qty = min(order.quantity, item.buy_limit, int(purse // order.buy))
|
||||
if qty <= 0:
|
||||
continue
|
||||
purse -= qty * order.buy
|
||||
live.append((order, item, qty))
|
||||
fills.committed = market.capital - purse
|
||||
fills.offered = sum(q for _, _, q in live)
|
||||
|
||||
remaining = [q for _, _, q in live]
|
||||
available = [0] * len(live)
|
||||
pending = [0] * len(live)
|
||||
# Capacity carries between eligible ticks instead of being truncated at each one. An
|
||||
# item that trades two units a tick would otherwise be untradeable rather than thin,
|
||||
# because a quarter of two is zero every time — and thin is what this is about.
|
||||
buy_room = [0.0] * len(live)
|
||||
sell_room = [0.0] * len(live)
|
||||
proceeds = 0.0
|
||||
|
||||
for t in range(market.visible, market.visible + market.held_out):
|
||||
for i in range(len(live)):
|
||||
available[i] += pending[i]
|
||||
pending[i] = 0
|
||||
for i, (order, item, _) in enumerate(live):
|
||||
price = item.prices[t]
|
||||
flow = FILL_SHARE * item.volumes[t]
|
||||
# Room accrues only on the ticks the order was actually eligible on. Accruing it
|
||||
# every tick banks the flow of ticks the price never reached, which would let an
|
||||
# offer fill far past the volume that was ever available to it — and the volume
|
||||
# limit is the whole reason allocation is a decision here.
|
||||
if price <= order.buy:
|
||||
buy_room[i] += flow
|
||||
if price >= order.sell:
|
||||
sell_room[i] += flow
|
||||
if price <= order.buy and remaining[i] > 0:
|
||||
qty = min(remaining[i], int(buy_room[i]))
|
||||
if qty > 0:
|
||||
buy_room[i] -= qty
|
||||
remaining[i] -= qty
|
||||
pending[i] += qty
|
||||
fills.spent += qty * price
|
||||
fills.bought += qty
|
||||
if price >= order.sell and available[i] > 0:
|
||||
qty = min(available[i], int(sell_room[i]))
|
||||
if qty > 0:
|
||||
sell_room[i] -= qty
|
||||
proceeds += qty * price * (1.0 - TAX)
|
||||
available[i] -= qty
|
||||
fills.sold += qty
|
||||
|
||||
for i, (_, item, _) in enumerate(live):
|
||||
left = available[i] + pending[i]
|
||||
if left:
|
||||
depth = statistics.median(item.volumes[market.visible:])
|
||||
haircut = min(DUMP_CAP, DUMP_BASE + DUMP_IMPACT * (left / max(1.0, depth)))
|
||||
fills.dumped += left
|
||||
proceeds += left * item.prices[-1] * (1.0 - haircut) * (1.0 - TAX)
|
||||
# Coins locked behind an offer that never filled come back when the window closes, so
|
||||
# the only thing a wasted offer costs is the profit the capital did not make. That is
|
||||
# the right price for it: an opportunity cost, not a fine.
|
||||
fills.realised = proceeds - fills.spent
|
||||
return fills
|
||||
|
||||
|
||||
def crossings(prices: list[int]) -> float:
|
||||
"""Share of consecutive ticks that straddle the series' own mean.
|
||||
|
||||
The whole discrimination, in one countable number, so it is available to anything that
|
||||
can read the prompt — no variance ratio, no regression, just how often the line cuts
|
||||
its own average.
|
||||
"""
|
||||
if len(prices) < 2:
|
||||
return 0.0
|
||||
anchor = statistics.fmean(prices)
|
||||
above = [p > anchor for p in prices]
|
||||
return sum(1 for i in range(1, len(above)) if above[i] != above[i - 1]) / (len(above) - 1)
|
||||
|
||||
|
||||
def reference_orders(market: Market) -> list[Order]:
|
||||
"""The strategy the reward is normalised against, computed from the visible half only.
|
||||
|
||||
Per item: anchor on the mean of the visible prices, buy a band under it, sell a band
|
||||
over it. Then three judgements, and each of them is a way the reward discriminates:
|
||||
|
||||
does it revert `crossings` over MIN_CROSSINGS, or the item is skipped. The decoy
|
||||
swings widest and is worth nothing.
|
||||
what it pays the average visible price BELOW the buy limit against the average
|
||||
ABOVE the sell limit — because a limit fills at the market, not at
|
||||
the limit, so a wide reverting item pays far more than its band.
|
||||
Ranking by the band alone ranks every item identically.
|
||||
what it can hold a quarter of a typical tick's volume, over the ticks that touched
|
||||
the buy limit. Capital committed beyond that is capital locked
|
||||
behind an offer that will not fill.
|
||||
|
||||
Best return on capital first, until the purse is gone. Nothing here reads a held-out
|
||||
tick; every input is a column the agent was shown.
|
||||
"""
|
||||
plans = []
|
||||
for item in market.items:
|
||||
prices = item.visible_prices(market.visible)
|
||||
volumes = item.visible_volumes(market.visible)
|
||||
if crossings(prices) < MIN_CROSSINGS:
|
||||
continue
|
||||
anchor = statistics.fmean(prices)
|
||||
buy = max(1, round(anchor * (1.0 - BUY_BAND)))
|
||||
sell = max(buy + 1, round(anchor * (1.0 + SELL_BAND)))
|
||||
lows = [p for p in prices if p <= buy]
|
||||
highs = [p for p in prices if p >= sell]
|
||||
if not lows or not highs:
|
||||
continue
|
||||
entry, exit_ = statistics.fmean(lows), statistics.fmean(highs)
|
||||
expected = (exit_ * (1.0 - TAX) - entry) / entry
|
||||
if expected <= 0:
|
||||
continue
|
||||
reachable = int(
|
||||
FILL_SHARE * statistics.median(volumes) * (len(lows) / len(prices)) * market.held_out
|
||||
)
|
||||
qty = min(item.buy_limit, reachable, int(MAX_ITEM_SHARE * market.capital // buy))
|
||||
if qty <= 0:
|
||||
continue
|
||||
plans.append((expected, item.name, qty, buy, sell))
|
||||
|
||||
plans.sort(key=lambda p: -p[0])
|
||||
orders, purse = [], float(market.capital)
|
||||
for _, name, qty, buy, sell in plans:
|
||||
qty = min(qty, int(purse // buy))
|
||||
if qty <= 0:
|
||||
continue
|
||||
purse -= qty * buy
|
||||
orders.append(Order(item=name, quantity=qty, buy=buy, sell=sell))
|
||||
return orders
|
||||
|
||||
|
||||
def viable_market(seed: int, num_items: int, visible: int, held_out: int) -> Market:
|
||||
"""The next basket from `seed` onward in which the reference strategy makes money.
|
||||
|
||||
Every ratio in the reward divides by the reference's realised profit, so a basket where
|
||||
the reference loses has no reachable ceiling and would quietly break house rule 3 for
|
||||
that task — the oracle would score below 1.000 and nothing would say why. About one
|
||||
basket in two thousand is like that, from an anchor the visible half happened to
|
||||
mis-estimate. Skipping it is a guard, not a crutch, and the seed that was used travels
|
||||
on the Market so scoring rebuilds exactly the basket that was shown.
|
||||
"""
|
||||
for offset in range(VIABILITY_TRIES):
|
||||
market = build_market(seed + offset, num_items, visible, held_out)
|
||||
if execute(market, reference_orders(market)).realised > 0:
|
||||
return market
|
||||
return build_market(seed, num_items, visible, held_out)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Outcome:
|
||||
"""One run, beside the reference run it is normalised against."""
|
||||
|
||||
fills: Fills
|
||||
reference: Fills
|
||||
stated: float | None
|
||||
dropped: int = 0
|
||||
"""Orders that named nothing on the board, or asked for a quantity or price of zero."""
|
||||
|
||||
@property
|
||||
def target(self) -> float:
|
||||
"""The gp a full score is worth in this basket. See TARGET_SHARE."""
|
||||
return TARGET_SHARE * self.reference.realised
|
||||
|
||||
@property
|
||||
def profit_ratio(self) -> float:
|
||||
if self.target <= 0.0:
|
||||
return 0.0
|
||||
return min(1.0, max(0.0, self.fills.realised / self.target))
|
||||
|
||||
@property
|
||||
def efficiency(self) -> float:
|
||||
"""Return on committed capital, against the reference's return on its own.
|
||||
|
||||
Not discounted by TARGET_SHARE: it is already a rate rather than a total, so a
|
||||
strategy that trades well on a small book is not penalised for being small, and
|
||||
giving it a band as well would hand out the term for free."""
|
||||
if self.reference.roc <= 0.0:
|
||||
return 0.0
|
||||
return min(1.0, max(0.0, self.fills.roc / self.reference.roc))
|
||||
|
||||
@property
|
||||
def conversion_ratio(self) -> float:
|
||||
"""Of the coins locked, the share that bought stock, against the reference's share.
|
||||
Recorded, never rewarded — `efficiency` is what qualifies profit now. It stays in
|
||||
the trace because it separates the two ways `efficiency` falls: offers that never
|
||||
filled, and offers that filled at prices not worth reaching."""
|
||||
if self.reference.conversion <= 0.0:
|
||||
return 0.0
|
||||
return min(1.0, self.fills.conversion / self.reference.conversion)
|
||||
|
||||
@property
|
||||
def discipline(self) -> float:
|
||||
"""Profit, qualified by what each committed coin earned. A product, not a term
|
||||
beside it: a term would pay in full for one tiny order that traded perfectly, which
|
||||
is inaction with a receipt — the free-points defect schema-migration shipped with."""
|
||||
return self.profit_ratio * self.efficiency
|
||||
|
||||
@property
|
||||
def clean(self) -> bool:
|
||||
"""A bar, not a knife-edge.
|
||||
|
||||
It used to require matching the reference on both ratios to within 1e-9, which is a
|
||||
demand for replication rather than for a good run: a reference plan with one extra
|
||||
one-unit order that never fills has identical realised profit and lost 0.146 of
|
||||
total reward. The bar is now TARGET_SHARE of the reference on the money AND on the
|
||||
return that money made, which is a run that was worth doing however it got there.
|
||||
"""
|
||||
return (
|
||||
self.fills.realised >= TARGET_SHARE * self.reference.realised
|
||||
and self.fills.roc >= TARGET_SHARE * self.reference.roc
|
||||
)
|
||||
|
||||
@property
|
||||
def arith_ok(self) -> bool:
|
||||
"""Did the agent's stated expected profit match what its own ACCEPTED orders imply.
|
||||
|
||||
Not a reward — a probe into the trace. The target model gets simple arithmetic wrong
|
||||
with thinking off, and without this line a bad multiplication and a bad strategy are
|
||||
the same low number and nothing in the trace tells them apart. Measured against the
|
||||
orders that survived parsing, so claiming profit from an order for an item that is
|
||||
not on the board reads as the arithmetic error it is.
|
||||
"""
|
||||
if self.stated is None:
|
||||
return False
|
||||
return abs(self.stated - self.fills.paper) <= max(50.0, 0.02 * abs(self.fills.paper))
|
||||
|
||||
@property
|
||||
def arith_error(self) -> float:
|
||||
if self.stated is None or self.fills.paper == 0.0:
|
||||
return 0.0
|
||||
return abs(self.stated - self.fills.paper) / abs(self.fills.paper)
|
||||
|
||||
|
||||
def measure(market: Market, orders: list[Order], stated: float | None) -> Outcome:
|
||||
"""Run the agent's plan and the reference plan through the same engine."""
|
||||
clean, seen, dropped = [], set(), 0
|
||||
for order in orders[:MAX_ORDERS]:
|
||||
item = market.item(order.item)
|
||||
# One order per item: holdings pool per item, so two orders on one name would make
|
||||
# "which sell limit does this unit belong to" a question the engine has to invent an
|
||||
# answer to. The first one submitted is the one that counts.
|
||||
if item is None or order.quantity <= 0 or order.buy <= 0 or order.sell <= 0:
|
||||
dropped += 1
|
||||
continue
|
||||
if item.name in seen:
|
||||
dropped += 1
|
||||
continue
|
||||
seen.add(item.name)
|
||||
clean.append(Order(item.name, order.quantity, order.buy, order.sell))
|
||||
return Outcome(
|
||||
fills=execute(market, clean),
|
||||
reference=execute(market, reference_orders(market)),
|
||||
stated=stated,
|
||||
dropped=dropped,
|
||||
)
|
||||
|
||||
|
||||
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
|
||||
|
||||
|
||||
def parse_orders(reply: str) -> tuple[list[Order], float | None]:
|
||||
"""The last JSON block in the reply: an object with "orders" and, optionally,
|
||||
"expected_profit". A bare array of orders is accepted too — a model that answers the
|
||||
question and skips the arithmetic has traded, and should be graded on the trade.
|
||||
|
||||
A reply that parses to nothing is an empty plan, not an error: it scores what doing
|
||||
nothing scores. Raising here would turn a formatting slip into a crashed rollout — and
|
||||
the reward runs inside the metric, so a raise takes the whole rollout with it rather
|
||||
than scoring zero.
|
||||
|
||||
Which is why the except clauses below are wider than they look like they need to be.
|
||||
Python's `json.loads` is not strict JSON: it accepts the bare literals Infinity,
|
||||
-Infinity and NaN, and it overflows 1e309 to inf rather than refusing it. RecursionError
|
||||
is not a ValueError, so twenty thousand nested arrays crashed the decoder; OverflowError
|
||||
is not a ValueError either, so `int(float("inf"))` crashed the row loop. Both were live
|
||||
on the first cut and both were reachable from a reply a model can actually emit.
|
||||
"""
|
||||
blocks = _BLOCK.findall(reply or "")
|
||||
raw = blocks[-1] if blocks else (reply or "")
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
except (ValueError, RecursionError):
|
||||
return [], None
|
||||
|
||||
stated: float | None = None
|
||||
if isinstance(parsed, dict):
|
||||
rows = parsed.get("orders")
|
||||
value = parsed.get("expected_profit")
|
||||
# Finite only. `Infinity` and `NaN` parse, and either one propagates through
|
||||
# `arith_error` into the trace as a non-finite metric, which is a corrupted training
|
||||
# signal rather than a bad answer. An unusable claim is no claim.
|
||||
if isinstance(value, (int, float)) and not isinstance(value, bool):
|
||||
try:
|
||||
stated = float(value) if math.isfinite(value) else None
|
||||
except OverflowError:
|
||||
stated = None
|
||||
else:
|
||||
rows = parsed
|
||||
if not isinstance(rows, list):
|
||||
return [], stated
|
||||
|
||||
orders = []
|
||||
for row in rows:
|
||||
if not isinstance(row, dict) or not isinstance(row.get("item"), str):
|
||||
continue
|
||||
try:
|
||||
orders.append(
|
||||
Order(
|
||||
item=row["item"],
|
||||
quantity=int(row.get("quantity", 0)),
|
||||
buy=int(row.get("buy", 0)),
|
||||
sell=int(row.get("sell", 0)),
|
||||
)
|
||||
)
|
||||
except (TypeError, ValueError, OverflowError):
|
||||
continue
|
||||
return orders, stated
|
||||
@@ -0,0 +1,244 @@
|
||||
"""The market generator: price and volume streams with structure a reader can find.
|
||||
|
||||
If prices were a random walk this environment would be worthless and would still look
|
||||
fine — expected profit is zero for every strategy, so the reward is noise, no oracle
|
||||
exists, and the numbers coming out of it would be plausible and meaningless. So the
|
||||
structure is put in deliberately and is the whole design:
|
||||
|
||||
price_t = fundamental_t * (1 + x_t)
|
||||
|
||||
`fundamental_t` drifts by FUND_DRIFT a tick, worth about one percent over a whole stream.
|
||||
`x_t` is an AR(1) around zero with PHI decay and a stationary spread of NOISE_SD — nine
|
||||
percent on the deep items, fifteen or sixteen on the thin ones. Noise dominates drift by an
|
||||
order of magnitude, which is what makes the mean of the visible prices a usable estimate of
|
||||
the fundamental, and "buy under the estimate, sell over it" a real strategy rather than a
|
||||
superstition. `book.reference_orders` is that strategy; `probe.py` measures how far it beats
|
||||
trading at random, and refuses to pass if the margin is thin.
|
||||
|
||||
The visible and held-out windows are two halves of ONE stream from ONE seed: the graded
|
||||
ticks are the next ticks the generator would have produced, not a differently-seeded
|
||||
population that could have moved somewhere the visible half gave no warning of.
|
||||
|
||||
Two things are here to punish reading the price column alone. One or two items per basket do
|
||||
not revert at all (see DRIFT_STEP and WALKS) and swing widest of everything on screen. And
|
||||
liquidity — price times volume, gp a tick — is uncorrelated with price, so the fattest
|
||||
visible margins sit on the items that can absorb the least of the purse.
|
||||
|
||||
Item names are invented. Formulas and market mechanics are facts about a kind of game;
|
||||
item tables are somebody's copyrighted content, and none of it is here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import random
|
||||
from dataclasses import dataclass
|
||||
|
||||
# --- the rules of the exchange, quoted to the agent verbatim in the prompt --------------
|
||||
TAX = 0.01
|
||||
"""Charged on every sale, the GE's own. It is the counterweight that kills thin flips:
|
||||
a round trip has to clear it twice over before it is worth doing, so "trade everything"
|
||||
is a losing strategy rather than a neutral one."""
|
||||
|
||||
FILL_SHARE = 0.25
|
||||
"""Share of a tick's volume one participant can take, and it accrues only on the ticks an
|
||||
offer was actually eligible on. Without it a fat margin on an item carrying three thousand
|
||||
gp a tick is worth as much as one carrying a hundred thousand, and the environment stops
|
||||
measuring allocation."""
|
||||
|
||||
DUMP_BASE = 0.03
|
||||
DUMP_IMPACT = 0.05
|
||||
DUMP_CAP = 0.40
|
||||
"""Stock still held when the window closes is forced out at the last price, minus a haircut
|
||||
of DUMP_BASE plus DUMP_IMPACT for every full tick's worth of the item's median volume that
|
||||
has to be pushed through, capped at DUMP_CAP.
|
||||
|
||||
A FLAT haircut was the first cut of this and it was the wrong shape. Under a flat five
|
||||
percent, the cost of holding stock at the close was the same whether the leftovers were
|
||||
twenty units of an item that trades thirty a tick or four thousand units of one that trades
|
||||
six hundred — so oversizing was only ever punished through the purse, and the purse punishes
|
||||
it as a cliff: the first over-large offer eats the whole budget and everything after it is
|
||||
never placed. A cliff is not a gradient. Scaling the haircut by position-over-depth prices
|
||||
the thing that is actually true — forcing size out costs you in proportion to how much of
|
||||
the book you are pushing through — and it makes reading the volume column pay smoothly,
|
||||
which is what the column is here to teach.
|
||||
|
||||
It also prices a bad anchor. A buy limit set too high fills fast and leaves the sell limit
|
||||
out of reach, so the position that a sloppy anchor builds is exactly the position that has
|
||||
to be dumped, and now it is dumped at a price that scales with its size."""
|
||||
|
||||
STARTING_CAPITAL = 250_000
|
||||
|
||||
WALKS = (1, 2)
|
||||
"""How many of the basket's items do not mean-revert, drawn uniformly from this range.
|
||||
|
||||
It used to be exactly one, and a fixed count is a free prior: "drop the single widest line
|
||||
on the board" scores what computing `book.crossings` scores, without computing anything.
|
||||
It is the same defect `bot_detection` ships an assertion against — a class balance the model
|
||||
can count on is a class balance it will use instead of the discriminator. With the count
|
||||
unknown the shape statistic is the only thing that answers the question, and a basket can
|
||||
punish both over- and under-rejection."""
|
||||
|
||||
SEED_BASE = 60_000
|
||||
"""Where task seeds start. Lives here rather than on the taskset so `probe.py`, which
|
||||
cannot import the taskset without `verifiers`, grades the baskets a run would actually
|
||||
serve rather than a different set that happens to share a generator."""
|
||||
|
||||
# --- stream parameters -----------------------------------------------------------------
|
||||
PHI = 0.45
|
||||
"""AR(1) decay of the mispricing. Half-life under a tick, so a visible window holds many
|
||||
independent draws around the fundamental — which is what makes the mean of it an estimate
|
||||
rather than a guess — and a held-out window holds many excursions, so the reward is not one
|
||||
lucky draw."""
|
||||
FUND_DRIFT = 0.0008
|
||||
"""Per-tick drift of the fundamental. Small on purpose: the fundamental has to be
|
||||
ESTIMABLE from the visible half or there is nothing to learn."""
|
||||
VOLUME_SD = 0.35
|
||||
|
||||
DRIFT_STEP = 0.055
|
||||
"""Per-tick step of the items that do not mean-revert at all — their price is a pure random
|
||||
walk, so the fundamental IS wherever it last was.
|
||||
|
||||
This is the trap the whole environment is built around, put inside the task instead of
|
||||
left as a hazard the designer has to avoid. A random walk has no anchor, so buying under
|
||||
its moving average is not a discount, it is a coin flip that pays the tax and the dump
|
||||
slippage every time. On screen it is the widest-swinging line in the basket and therefore
|
||||
the most attractive one, because amplitude is what a careless reader ranks by. The two can
|
||||
only be told apart by SHAPE: a reverting series crosses its own mean constantly, a walk
|
||||
wanders on one side of it for a dozen ticks at a time. `book.crossings` is that statistic
|
||||
and the reference strategy will not trade an item that fails it."""
|
||||
|
||||
# (names, base price range, base volume range, buy limit, mispricing spread)
|
||||
#
|
||||
# What separates these is LIQUIDITY IN GP PER TICK — base price times base volume — and it
|
||||
# is deliberately uncorrelated with the price. That is the allocation problem: the purse is
|
||||
# 250,000 gp and a quarter of the flow over thirty ticks is what any one offer can absorb,
|
||||
# so a deep tier can take a third of the purse and a thin one can take a twentieth of it no
|
||||
# matter how good the margin looks. An earlier cut of this file made the expensive items the
|
||||
# thin ones, which sounds right and is not: eight units a tick of a 46,000 gp item is 368,000
|
||||
# gp of flow, the deepest thing on the board. Thin means small in coins, not small in units.
|
||||
#
|
||||
# The two thin tiers also carry the widest mispricing spread, so they show the fattest margin
|
||||
# and can absorb the least. That is the trap, and it is the same trap either way an agent
|
||||
# falls into it: ignore the volume column and either the offers sit unfilled or the purse
|
||||
# sits idle.
|
||||
TIERS = [
|
||||
# deep, ~40k-130k gp a tick: this is where the purse actually goes
|
||||
(["Thornroot poultice", "Chipped bone charm", "Bogwater draught", "Coarse fletching feather"],
|
||||
(90, 170), (400, 800), 5000, 0.09),
|
||||
(["Emberglass shard", "Stormrune tablet", "Marrowsteel nail", "Pale grimoire page"],
|
||||
(900, 1700), (40, 90), 400, 0.09),
|
||||
# thin, ~3k-15k gp a tick, and the widest swings on the board
|
||||
(["Gilded harpoon head", "Cinderweave cloak", "Wyrmbone talisman", "Frostbitten ledger"],
|
||||
(200, 420), (12, 30), 800, 0.15),
|
||||
(["Duskforged sigil", "Heart of the sunken cairn", "Voidglass lens", "Tideworn crown"],
|
||||
(6000, 14000), (1.2, 3.0), 40, 0.16),
|
||||
]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Item:
|
||||
name: str
|
||||
reverting: bool
|
||||
"""Whether this item has an anchor at all. Never shown to the agent — it is here so the
|
||||
probe can assert that the trap is a trap, and that the reference avoids it for a reason
|
||||
rather than by luck."""
|
||||
buy_limit: int
|
||||
"""Units per item per window, the GE's own limit. With finite capital it is what turns
|
||||
the task into an allocation problem instead of a single pick."""
|
||||
prices: list[int]
|
||||
volumes: list[int]
|
||||
|
||||
def visible_prices(self, visible: int) -> list[int]:
|
||||
return self.prices[:visible]
|
||||
|
||||
def visible_volumes(self, visible: int) -> list[int]:
|
||||
return self.volumes[:visible]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Market:
|
||||
seed: int
|
||||
"""Carried on the basket so the taskset can store the seed it actually used. Baskets are
|
||||
skipped when the reference strategy is not profitable in them (see `book.viable_market`),
|
||||
so the seed a task was built from is not always the one it was asked for."""
|
||||
items: list[Item]
|
||||
visible: int
|
||||
held_out: int
|
||||
capital: int
|
||||
|
||||
def item(self, name: str) -> Item | None:
|
||||
wanted = name.strip().casefold()
|
||||
for item in self.items:
|
||||
if item.name.casefold() == wanted:
|
||||
return item
|
||||
return None
|
||||
|
||||
|
||||
def _volumes(rng: random.Random, volume: float, ticks: int) -> list[int]:
|
||||
return [max(1, round(volume * math.exp(rng.gauss(0.0, VOLUME_SD)))) for _ in range(ticks)]
|
||||
|
||||
|
||||
def _walk(rng: random.Random, base: float, ticks: int) -> list[int]:
|
||||
"""The decoy: no anchor, no reversion, just a wide random walk. Whatever a moving
|
||||
average says about where this price belongs is a statement about the past only."""
|
||||
price = base
|
||||
out = []
|
||||
for _ in range(ticks):
|
||||
price *= 1.0 + rng.gauss(0.0, DRIFT_STEP)
|
||||
out.append(max(1, round(price)))
|
||||
return out
|
||||
|
||||
|
||||
def _stream(rng: random.Random, base: float, volume: float, noise_sd: float, ticks: int
|
||||
) -> tuple[list[int], list[int]]:
|
||||
"""One item's price and volume history, visible and held-out ticks together."""
|
||||
# eps is scaled so the AR(1) settles at exactly noise_sd rather than drifting toward it
|
||||
# over the first few ticks; the visible half would otherwise be quieter than the graded
|
||||
# half and every anchor estimated from it would be too tight.
|
||||
eps_sd = noise_sd * math.sqrt(1.0 - PHI * PHI)
|
||||
x = rng.gauss(0.0, noise_sd)
|
||||
fundamental = base
|
||||
prices = []
|
||||
for _ in range(ticks):
|
||||
fundamental *= 1.0 + rng.gauss(0.0, FUND_DRIFT)
|
||||
x = PHI * x + rng.gauss(0.0, eps_sd)
|
||||
prices.append(max(1, round(fundamental * (1.0 + x))))
|
||||
return prices, _volumes(rng, volume, ticks)
|
||||
|
||||
|
||||
def build_market(seed: int, num_items: int, visible: int, held_out: int) -> Market:
|
||||
"""One basket, from one seed.
|
||||
|
||||
Every tier is represented before any tier repeats, so a thin item and a deep one are
|
||||
always both on the table: the allocation choice is the task, and a basket that happened
|
||||
to be all-deep or all-thin would not pose it.
|
||||
"""
|
||||
rng = random.Random(seed)
|
||||
ticks = visible + held_out
|
||||
order = list(range(len(TIERS)))
|
||||
rng.shuffle(order)
|
||||
picks = [order[i % len(order)] for i in range(num_items)]
|
||||
|
||||
# Which slots are walks, and how many, are both drawn here. The tier is drawn
|
||||
# independently of the walk flag, so a basket where the walk was always the cheap item —
|
||||
# solvable by reading the price column and never the shape — cannot arise.
|
||||
decoys = set(rng.sample(range(num_items), rng.randint(*WALKS)))
|
||||
|
||||
used: set[str] = set()
|
||||
items = []
|
||||
for slot, tier_idx in enumerate(picks):
|
||||
names, (lo, hi), (vlo, vhi), limit, noise_sd = TIERS[tier_idx]
|
||||
choices = [n for n in names if n not in used] or names
|
||||
name = rng.choice(choices)
|
||||
used.add(name)
|
||||
base, volume = rng.uniform(lo, hi), rng.uniform(vlo, vhi)
|
||||
if slot in decoys:
|
||||
prices, volumes = _walk(rng, base, ticks), _volumes(rng, volume, ticks)
|
||||
else:
|
||||
prices, volumes = _stream(rng, base, volume, noise_sd, ticks)
|
||||
items.append(Item(name=name, reverting=slot not in decoys, buy_limit=limit,
|
||||
prices=prices, volumes=volumes))
|
||||
items.sort(key=lambda i: i.name)
|
||||
return Market(seed=seed, items=items, visible=visible, held_out=held_out,
|
||||
capital=STARTING_CAPITAL)
|
||||
@@ -0,0 +1,230 @@
|
||||
"""grand-exchange: place orders that are executed against a window you have not seen.
|
||||
|
||||
The agent reads 56 ticks of price and volume for five items and submits limit orders. The
|
||||
orders are then run against the NEXT 30 ticks of the same streams — the ones the generator
|
||||
would have produced next — against that window's actual volume, with a 1% sale tax, a
|
||||
per-item buy limit and one finite purse. Realised profit is what pays.
|
||||
|
||||
Four judgements separate a good plan from a plan, and every one of them is available from
|
||||
the columns on screen:
|
||||
|
||||
the anchor the mean of 56 visible ticks estimates the fundamental about four times
|
||||
more tightly than the last tick does, and a limit set off a bad anchor
|
||||
fills fast into stock the sell limit never reaches.
|
||||
the trap one or two items per basket are a random walk, not a reverting series.
|
||||
They swing widest, which is what a careless reader ranks by, and they have
|
||||
no anchor to revert to. `book.crossings` is the only thing that tells them
|
||||
apart, and the COUNT is not fixed, so "drop the widest one" is not a
|
||||
substitute for computing it.
|
||||
the size a quarter of a tick's volume, over the ticks that will reach your limit —
|
||||
offers beyond that are coins locked behind something that cannot fill, and
|
||||
stock beyond that is stock the close-out haircut charges you for.
|
||||
the allocation finite purse, funded in submission order, so the best return has to go
|
||||
first and the rest has to fit.
|
||||
|
||||
Realised profit has no ceiling and no optimum the data supports, so scoring it against one
|
||||
would put 1.000 out of reach. Every ratio is taken against a REFERENCE STRATEGY instead,
|
||||
which makes those four judgements and nothing else, from the visible half only, and runs
|
||||
through the same execution engine. Its constants are the profit-maximising point of its own
|
||||
family, swept on seeds no task is built from — a denominator that leaves money on the table
|
||||
makes the reward an imitation score rather than a profit metric — and the ceiling is nine
|
||||
tenths of it, so the plateau around it and everything above it all score 1.000.
|
||||
|
||||
`probe.py` measures how far the reference beats trading at random, and refuses to pass if
|
||||
that margin closes, because a reward normalised against a reference that is no better than
|
||||
chance is noise wearing a number. It also refuses to pass if a plan that never estimates the
|
||||
anchor and never applies the trap filter scores as well as one that does.
|
||||
|
||||
Three rewards:
|
||||
|
||||
profit clip(realised / (0.90 × reference realised), 0, 1). No orders is exactly zero,
|
||||
and so is any round trip that fails to clear the tax.
|
||||
discipline profit × realised-per-coin-committed, relative to the reference's. A PRODUCT:
|
||||
as a term beside profit it would pay in full for one tiny order that traded
|
||||
perfectly, which is inaction with a receipt.
|
||||
gate cleared both bars. Binary — a trading run was worth doing or it was not. A
|
||||
bar, not a match: demanding the reference's exact result made a wasted unit of
|
||||
capital cost 0.146 with byte-identical profit, which is a gradient pointing at
|
||||
replication instead of at money.
|
||||
|
||||
`arith_ok` is recorded and never rewarded: whether the agent's stated expected profit
|
||||
matches what its own orders imply. The target model gets simple arithmetic wrong with
|
||||
thinking off, and without this line in the trace a bad multiplication and a bad strategy
|
||||
are the same low score.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import ClassVar
|
||||
|
||||
from pydantic import Field
|
||||
|
||||
import verifiers.v1 as vf
|
||||
|
||||
from grand_exchange.book import MAX_ORDERS, TARGET_SHARE, measure, parse_orders, viable_market
|
||||
from grand_exchange.market import (
|
||||
DUMP_BASE,
|
||||
DUMP_CAP,
|
||||
DUMP_IMPACT,
|
||||
FILL_SHARE,
|
||||
SEED_BASE,
|
||||
STARTING_CAPITAL,
|
||||
TAX,
|
||||
build_market,
|
||||
)
|
||||
|
||||
SYSTEM = f"""You are trading on the Grand Exchange.
|
||||
|
||||
You will be shown recent price and volume history for a basket of items. You submit limit
|
||||
orders. They are executed tick by tick against a FUTURE window you have not seen, drawn
|
||||
from the same streams.
|
||||
|
||||
The rules of the exchange:
|
||||
- A buy fills at the tick's price when that price is at or below your buy limit. A sell
|
||||
fills at the tick's price when that price is at or above your sell limit.
|
||||
- PLACING an offer locks quantity × buy price out of your purse straight away, whether
|
||||
or not it ever fills. Offers are funded in the order you list them; once the purse is
|
||||
gone the rest are not placed at all. You start with {STARTING_CAPITAL:,} gp and that is
|
||||
the entire budget — sale proceeds return to you but fund no further offers.
|
||||
- You can take at most {FILL_SHARE:.0%} of a tick's traded volume, per item, per side, and only
|
||||
on ticks where your limit was actually reached.
|
||||
- Each item has a buy limit: the most units of it you may buy in the whole window.
|
||||
- {TAX:.0%} tax is charged on every sale.
|
||||
- Stock bought on a tick cannot be sold on the same tick.
|
||||
- Stock you still hold when the window closes is forced out at the last price minus {DUMP_BASE:.0%},
|
||||
plus a further {DUMP_IMPACT:.0%} for every full tick's worth of that item's median volume you have
|
||||
to push through, capped at {DUMP_CAP:.0%}; then taxed. Buying what you cannot sell is a loss,
|
||||
not a hold, and it is a bigger loss the more of it there is.
|
||||
|
||||
Return ONE ```json code block containing an object with:
|
||||
"expected_profit": the profit you expect your orders to make, in gp, as a number
|
||||
"orders": a list of at most {MAX_ORDERS} objects, each with "item" (exactly as named above),
|
||||
"quantity", "buy" and "sell"
|
||||
|
||||
At most one order per item; later duplicates are discarded. Example shape:
|
||||
```json
|
||||
{{"expected_profit": 12000,
|
||||
"orders": [{{"item": "Bogwater draught", "quantity": 400, "buy": 118, "sell": 129}}]}}
|
||||
```"""
|
||||
|
||||
|
||||
class ExchangeData(vf.TaskData):
|
||||
seed: int
|
||||
"""Rebuilds both windows exactly. The held-out ticks are never serialized here — they
|
||||
are regenerated at scoring time from this integer, so the task data cannot leak them."""
|
||||
num_items: int
|
||||
visible: int
|
||||
held_out: int
|
||||
|
||||
|
||||
class ExchangeTask(vf.Task[ExchangeData]):
|
||||
@vf.stop
|
||||
async def single_turn(self, trace: vf.Trace) -> bool:
|
||||
return trace.num_turns >= 1
|
||||
|
||||
@vf.metric
|
||||
async def scan(self, trace: vf.Trace) -> dict[str, float]:
|
||||
"""Execute the plan and the reference once; every reward reads this. Recomputing
|
||||
per reward is how two rewards end up disagreeing about the same episode."""
|
||||
market = build_market(
|
||||
self.data.seed, self.data.num_items, self.data.visible, self.data.held_out
|
||||
)
|
||||
orders, stated = parse_orders(trace.last_reply)
|
||||
outcome = measure(market, orders, stated)
|
||||
return {
|
||||
"profit_ratio": outcome.profit_ratio,
|
||||
"efficiency": outcome.efficiency,
|
||||
"discipline": outcome.discipline,
|
||||
"clean": float(outcome.clean),
|
||||
"realised": outcome.fills.realised,
|
||||
"reference_realised": outcome.reference.realised,
|
||||
"target_realised": outcome.target,
|
||||
"roc": outcome.fills.roc,
|
||||
"reference_roc": outcome.reference.roc,
|
||||
# Recorded, not rewarded. `efficiency` falls for two different reasons and this
|
||||
# is what tells them apart in the trace: offers that never filled show here,
|
||||
# offers that filled at prices not worth reaching do not.
|
||||
"conversion_ratio": outcome.conversion_ratio,
|
||||
"paper": outcome.fills.paper,
|
||||
"filled": float(outcome.fills.bought),
|
||||
"planned": float(outcome.fills.planned),
|
||||
"dumped": float(outcome.fills.dumped),
|
||||
"committed": outcome.fills.committed / max(market.capital, 1),
|
||||
"conversion": outcome.fills.conversion,
|
||||
"orders_dropped": float(outcome.dropped),
|
||||
# Recorded, never rewarded: bad multiplication and bad strategy are different
|
||||
# failures and score the same without this.
|
||||
"arith_ok": float(outcome.arith_ok),
|
||||
"arith_error": outcome.arith_error,
|
||||
}
|
||||
|
||||
@vf.reward(weight=0.45)
|
||||
async def profit(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("profit_ratio", 0.0)
|
||||
|
||||
@vf.reward(weight=0.30)
|
||||
async def discipline(self, trace: vf.Trace) -> float:
|
||||
"""profit_ratio × efficiency, computed once in `scan`. Read from the metric rather
|
||||
than multiplied here: two rewards that recompute the same product are two rewards
|
||||
that will eventually disagree about the same episode."""
|
||||
return trace.metrics.get("discipline", 0.0)
|
||||
|
||||
@vf.reward(weight=0.25)
|
||||
async def gate(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("clean", 0.0)
|
||||
|
||||
|
||||
class ExchangeConfig(vf.TasksetConfig):
|
||||
num_tasks: int = Field(48, ge=1)
|
||||
num_items: int = Field(5, ge=1)
|
||||
visible: int = Field(56, ge=8)
|
||||
"""Ticks the agent reads. Enough of them that the mean is an anchor and not a rumour:
|
||||
shorten this and the reference strategy stops being reliably profitable, which would
|
||||
take the reward's denominator with it."""
|
||||
held_out: int = Field(30, ge=1)
|
||||
"""Ticks it is executed against and never sees."""
|
||||
|
||||
|
||||
class ExchangeTaskset(vf.Taskset[ExchangeTask, ExchangeConfig]):
|
||||
SEED_BASE: ClassVar[int] = SEED_BASE
|
||||
|
||||
def load(self) -> list[ExchangeTask]:
|
||||
tasks, seed = [], self.SEED_BASE
|
||||
for i in range(self.config.num_tasks):
|
||||
# `viable_market` may skip a basket the reference loses money in, so the seed
|
||||
# stored is the one it landed on and the next scan starts after it — otherwise
|
||||
# a skip would hand two tasks the same basket.
|
||||
market = viable_market(
|
||||
seed, self.config.num_items, self.config.visible, self.config.held_out
|
||||
)
|
||||
seed = market.seed + 1
|
||||
blocks = []
|
||||
for item in market.items:
|
||||
prices = " ".join(str(p) for p in item.visible_prices(market.visible))
|
||||
volumes = " ".join(str(v) for v in item.visible_volumes(market.visible))
|
||||
blocks.append(
|
||||
f"{item.name} (buy limit {item.buy_limit} per window)\n"
|
||||
f" price {prices}\n"
|
||||
f" volume {volumes}"
|
||||
)
|
||||
board = "\n\n".join(blocks)
|
||||
tasks.append(
|
||||
ExchangeTask(
|
||||
ExchangeData(
|
||||
idx=i,
|
||||
name=f"basket-{market.seed}",
|
||||
prompt=(
|
||||
f"Last {market.visible} ticks, oldest first. Your orders will be "
|
||||
f"executed against the next {market.held_out}.\n\n{board}\n\n"
|
||||
f"Purse: {market.capital:,} gp. Place your orders."
|
||||
),
|
||||
system_prompt=SYSTEM,
|
||||
seed=market.seed,
|
||||
num_items=self.config.num_items,
|
||||
visible=self.config.visible,
|
||||
held_out=self.config.held_out,
|
||||
),
|
||||
self.config.task,
|
||||
)
|
||||
)
|
||||
return tasks
|
||||
@@ -0,0 +1,13 @@
|
||||
[project]
|
||||
name = "grand-exchange"
|
||||
version = "0.1.0"
|
||||
description = "grand-exchange — place orders executed against a market window you never saw."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["grand_exchange"]
|
||||
Generated
+3488
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user