Three OSRS environments, and the first real model scores
grand-exchange, bot-detection and drop-table-inference. All three produce unbounded or irreducible-error scores, unlike the first four, so all three normalise against a REFERENCE STRATEGY rather than an absolute -- which is what makes 1.000 reachable rather than aspirational. drop-table-inference is the clearest case. Scoring KL against the true drop table would put the ceiling out of reach, because sampling error is irreducible: ground truth scores 0.900 against the Bayesian posterior's 1.000, and that ordering is correct. The best estimate available from 1,200 kills is not the true table, and an environment that demands it is measuring luck. grand-exchange needed the market to carry structure a model can actually infer, or profit is noise and no oracle exists. Measured over 120 baskets: the reference earns 38,821 gp and is profitable in 120/120, random orders lose 561, and trading only the random-walk items -- which look like the widest-swinging lines on the board -- loses 16,860. bot-detection generates naive bots, cloaked bots that jitter on purpose, and efficient humans who look bot-like on every naive statistic. Timing features are drawn BEFORE the generator decides who is a bot, so every latency rule sits at chance. Reference discriminator F1 1.000, random 0.430. Each was built by one agent then attacked by two independent reviewers on exploitability and soundness. They earned their keep: drop-table-inference's first reward let a memorised constant score 0.901 without opening the kill log, and its reference was 19% worse than a flat number. probe.py now carries three named fences asserting those attacks stay dead. First evaluation also lands. Nemotron 3.5 Lightning, thinking off, 32 rollouts per environment: 0.570 down to 0.059, every gate at or near zero, nothing solved and nothing unsolvable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,3 @@
|
||||
from bot_detection.taskset import BotTaskset
|
||||
|
||||
__all__ = ["BotTaskset"]
|
||||
@@ -0,0 +1,367 @@
|
||||
"""Generating one batch of accounts: a cover drawn before anybody knows who is a bot.
|
||||
|
||||
The naive discriminator for this task is "low timing variance means bot". An environment
|
||||
that only generates metronomic scripts teaches that rule and nothing else, so the cover
|
||||
statistics — how fast an account clicks, how long it plays, how far it wanders — are drawn
|
||||
BEFORE the population is assigned, and the assignment is a uniform coin the style never
|
||||
sees. That ordering is the whole design. The first version of this file drew the cover per
|
||||
population instead, and gave a grinder `sigma ~ U(0.30, 0.45)` where a cloaked bot got
|
||||
`U(0.65, 0.90)`; the two ranges do not touch, so `sd(log click_dt_ms)` cut cleanly at 0.52
|
||||
and a two-line rule that never opened `misclick_next`, `break_s` or `login_gap_min` scored
|
||||
0.953 against the oracle's 1.000. Session length and route count were worse than that: they
|
||||
were EXACT labels — every account with four or more routes was a casual human, all 7200 of
|
||||
them — so half the humans in every batch were free and the false positive the environment
|
||||
is built around cost nothing to avoid.
|
||||
|
||||
Drawing the style first makes P(bot | any statistic of the style) exactly the batch's bot
|
||||
rate, for every statistic, with no distributional argument required and nothing to drift.
|
||||
There are three populations:
|
||||
|
||||
human a person. Long-tailed click latency, breaks that cluster into micro-pauses and
|
||||
long absences, corrects most misclicks, logs in when they are free.
|
||||
scripted a naive bot. Near-constant delays, no breaks at all, one route, never corrects
|
||||
anything. Caught by everything, and meant to be — it is the floor of the task,
|
||||
not the task. A click-variance rule finds these and nothing else, which is
|
||||
worth about a quarter of the ceiling.
|
||||
cloaked a sophisticated bot wearing a style drawn from the same pool a human's came
|
||||
from. It leaks on exactly ONE of three behavioural channels, chosen per
|
||||
account:
|
||||
|
||||
uncorrected it never corrects a misclick. A person does.
|
||||
memoryless its breaks are exponential, so they land in the 3-8 minute dead
|
||||
band that a human's bimodal breaks never occupy.
|
||||
metronomic it logs in on a fixed period, so every gap between session
|
||||
starts is the same to within a couple of minutes.
|
||||
|
||||
Exactly one leak per cloaked account is the difficulty: a discriminator built on any single
|
||||
channel catches a third of them, and the labelled slice shows one of each so the three are
|
||||
learnable rather than guessable.
|
||||
|
||||
Every separation is guaranteed at generation time rather than hoped for — the resample
|
||||
loops below are why — and so is every NON-separation, which is what the style-first
|
||||
ordering buys. A ceiling that is unreachable on one seed in two hundred is still an
|
||||
unreachable ceiling, and house rule 3 is measured, not assumed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import random
|
||||
from dataclasses import dataclass
|
||||
|
||||
CLICKS = 32
|
||||
BREAKS = 12
|
||||
SESSIONS = 4
|
||||
MISCLICKS = 8
|
||||
TRAVERSALS = 20
|
||||
|
||||
# Seconds. Human breaks are bimodal — a micro-pause or a real absence — so nothing of
|
||||
# theirs lands in here. An exponential break generator fills it, which is the whole tell.
|
||||
DEAD_BAND = (180, 480)
|
||||
|
||||
BOT_POPULATIONS = ("scripted", "cloaked")
|
||||
TELLS = ("uncorrected", "memoryless", "metronomic")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Style:
|
||||
"""The cover an account wears, drawn before it is decided whether it is a person.
|
||||
|
||||
`efficient` is the corner every naive flagger points at: tight fast clicking, one or
|
||||
two memorised routes, sessions measured in hours. Half the styles in a batch are drawn
|
||||
there and the population coin does not know which, so an efficient account is exactly
|
||||
as likely to be a bot as a rambling one. The two corners OVERLAP on all three
|
||||
statistics as well, so there is not even a clean latent to recover.
|
||||
"""
|
||||
|
||||
efficient: bool
|
||||
click_median: int
|
||||
click_sigma: float
|
||||
session_min: list[int]
|
||||
routes: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Account:
|
||||
"""One account's activity log. Every field is raw per-event data rather than a summary,
|
||||
because the summary IS the answer — handing over a variance or a correction rate would
|
||||
leave nothing to find."""
|
||||
|
||||
acc_id: str
|
||||
population: str
|
||||
tell: str
|
||||
"""Which channel a cloaked account leaks on; empty for everyone else."""
|
||||
style: str
|
||||
"""`efficient`, `casual`, or `constant` for a naive script. Never rendered — it is here
|
||||
so the probe can assert it carries no information about `is_bot`."""
|
||||
click_dt_ms: list[int]
|
||||
break_s: list[int]
|
||||
session_min: list[int]
|
||||
login_gap_min: list[int]
|
||||
"""Minutes between consecutive session starts."""
|
||||
misclick_next: list[str]
|
||||
"""What the account did after each misclick: `fix` or `ignore`."""
|
||||
route_ids: list[str]
|
||||
|
||||
@property
|
||||
def is_bot(self) -> bool:
|
||||
return self.population in BOT_POPULATIONS
|
||||
|
||||
|
||||
def _style(rng: random.Random, efficient: bool) -> Style:
|
||||
"""One cover. The ranges deliberately overlap between the two corners — 520-620 ms of
|
||||
median, 0.50-0.60 of sigma, 210-260 minutes of session, two or three routes — so that
|
||||
even the corner itself is not cleanly recoverable, let alone the population behind it."""
|
||||
if efficient:
|
||||
return Style(
|
||||
True,
|
||||
rng.randint(360, 620),
|
||||
rng.uniform(0.28, 0.60),
|
||||
[rng.randint(210, 520) for _ in range(SESSIONS)],
|
||||
rng.randint(1, 3),
|
||||
)
|
||||
return Style(
|
||||
False,
|
||||
rng.randint(520, 1000),
|
||||
rng.uniform(0.50, 0.95),
|
||||
[rng.randint(45, 260) for _ in range(SESSIONS)],
|
||||
rng.randint(2, 9),
|
||||
)
|
||||
|
||||
|
||||
def _lognormal_clicks(rng: random.Random, style: Style) -> list[int]:
|
||||
"""Human reaction time is lognormal: a hard floor, a dense bulk, and a long right tail
|
||||
of hesitations. Humans and cloaked bots draw from this with the SAME parameters, drawn
|
||||
from the same pool, so neither the tail nor its absence is a tell."""
|
||||
return [
|
||||
max(40, int(style.click_median * math.exp(rng.gauss(0.0, style.click_sigma))))
|
||||
for _ in range(CLICKS)
|
||||
]
|
||||
|
||||
|
||||
def _clustered_breaks(rng: random.Random, n: int) -> list[int]:
|
||||
"""Either a sip of coffee or a trip out of the house. Nothing in between, and the gap
|
||||
between the two modes is what an exponential generator cannot reproduce."""
|
||||
return [
|
||||
rng.randint(25, 140) if rng.random() < 0.6 else rng.randint(600, 2700)
|
||||
for _ in range(n)
|
||||
]
|
||||
|
||||
|
||||
def _memoryless_breaks(rng: random.Random, n: int) -> list[int]:
|
||||
"""Exponential breaks: a five-minute pause is as likely as any other, which is exactly
|
||||
what a human never does. Resampled until at least two land in the dead band, so the
|
||||
tell is present in every log rather than in most of them."""
|
||||
while True:
|
||||
out = [max(15, int(rng.expovariate(1 / 300.0))) for _ in range(n)]
|
||||
if sum(1 for b in out if DEAD_BAND[0] <= b < DEAD_BAND[1]) >= 2:
|
||||
return out
|
||||
|
||||
|
||||
def _irregular_gaps(rng: random.Random, n: int) -> list[int]:
|
||||
"""A person logs in when they are free. Resampled for a spread of at least two hours so
|
||||
that a human is never mistaken for a cron job."""
|
||||
while True:
|
||||
out = [rng.randint(240, 2400) for _ in range(n)]
|
||||
if max(out) - min(out) >= 120:
|
||||
return out
|
||||
|
||||
|
||||
def _metronomic_gaps(rng: random.Random, n: int) -> list[int]:
|
||||
"""A fixed period with a couple of minutes of slop — a scheduler, not a schedule."""
|
||||
period = rng.randrange(300, 481, 15)
|
||||
return [period + rng.randint(-2, 2) for _ in range(n)]
|
||||
|
||||
|
||||
def _human_misclicks(rng: random.Random, n: int) -> list[str]:
|
||||
"""A person who fat-fingers a target notices and puts it right. Resampled to at least
|
||||
five corrections in eight so the rate is unambiguously above any bot's zero."""
|
||||
while True:
|
||||
out = ["fix" if rng.random() < 0.8 else "ignore" for _ in range(n)]
|
||||
if out.count("fix") >= 5:
|
||||
return out
|
||||
|
||||
|
||||
def _never_corrects(n: int) -> list[str]:
|
||||
"""A script has no idea it missed. It has no model of what it meant to click."""
|
||||
return ["ignore"] * n
|
||||
|
||||
|
||||
def _routes(rng: random.Random, distinct: int, n: int) -> list[str]:
|
||||
"""Resampled until all `distinct` routes actually appear, so the rendered log shows the
|
||||
variety the style was supposed to have."""
|
||||
pool = [f"R{i + 1}" for i in range(distinct)]
|
||||
while True:
|
||||
out = [rng.choice(pool) for _ in range(n)]
|
||||
if len(set(out)) == distinct:
|
||||
return out
|
||||
|
||||
|
||||
def _wearing(rng: random.Random, acc_id: str, style: Style, tell: str) -> Account:
|
||||
"""A human (`tell` empty) or a cloaked bot, built from a style that was drawn before
|
||||
this call knew which it was making.
|
||||
|
||||
Every channel not named by `tell` goes through the identical generator with identical
|
||||
parameters, so a cloaked account and a person differ on exactly one column of the log
|
||||
and are the same distribution on the other five. That is what stops any statistic of
|
||||
the cover — click variance, session length, route count, or any function of them —
|
||||
from carrying information the reward would then pay for.
|
||||
"""
|
||||
return Account(
|
||||
acc_id,
|
||||
"cloaked" if tell else "human",
|
||||
tell,
|
||||
"efficient" if style.efficient else "casual",
|
||||
_lognormal_clicks(rng, style),
|
||||
_memoryless_breaks(rng, BREAKS) if tell == "memoryless" else _clustered_breaks(rng, BREAKS),
|
||||
list(style.session_min),
|
||||
_metronomic_gaps(rng, SESSIONS - 1) if tell == "metronomic" else _irregular_gaps(rng, SESSIONS - 1),
|
||||
_never_corrects(MISCLICKS) if tell == "uncorrected" else _human_misclicks(rng, MISCLICKS),
|
||||
_routes(rng, style.routes, TRAVERSALS),
|
||||
)
|
||||
|
||||
|
||||
def _scripted(rng: random.Random, acc_id: str) -> Account:
|
||||
"""The naive bot, and the only account whose cover is its own population. It leaks on
|
||||
all three behavioural channels at once AND on click variance, so a timing rule finds
|
||||
it; that is the point of keeping it. It is roughly a fifth of the batch, so a timing
|
||||
rule that finds nothing else ceilings around a quarter of the reward."""
|
||||
base = rng.randint(300, 450)
|
||||
return Account(
|
||||
acc_id, "scripted", "constant", "constant",
|
||||
[base + rng.randint(-12, 12) for _ in range(CLICKS)],
|
||||
[], # it does not stop, so there is nothing to log
|
||||
[rng.randint(900, 1400) for _ in range(SESSIONS)],
|
||||
_metronomic_gaps(rng, SESSIONS - 1),
|
||||
_never_corrects(MISCLICKS),
|
||||
_routes(rng, 1, TRAVERSALS),
|
||||
)
|
||||
|
||||
|
||||
def _ids(rng: random.Random, n: int) -> list[str]:
|
||||
seen: set[str] = set()
|
||||
out: list[str] = []
|
||||
while len(out) < n:
|
||||
candidate = f"ACC-{rng.randrange(0x10000):04X}"
|
||||
if candidate not in seen:
|
||||
seen.add(candidate)
|
||||
out.append(candidate)
|
||||
return out
|
||||
|
||||
|
||||
def _plan(rng: random.Random, graded: int) -> tuple[int, int, int]:
|
||||
"""How many scripted, cloaked and human accounts this batch holds.
|
||||
|
||||
Drawn per seed rather than fixed. A constant class balance is itself a decoy: with
|
||||
"always six bots in twelve" a model can rank the batch by any weak suspicion score and
|
||||
take the top six, which recovers most of the ceiling without a rule. Measured on the
|
||||
fixed-balance version that ranking scored 0.974.
|
||||
|
||||
Two invariants: at least three cloaked accounts, so all three tells fit and a
|
||||
single-channel discriminator visibly ceilings; and at least two more humans than
|
||||
cloaked accounts, which is what leaves room for `_cover` to guarantee an efficient
|
||||
human without letting the batch's parity say anything.
|
||||
"""
|
||||
scripted = rng.randint(1, max(1, graded // 6))
|
||||
ceiling = min((graded - scripted - 2) // 2, graded // 3 + 1)
|
||||
cloaked = rng.randint(3, max(3, ceiling))
|
||||
humans = graded - scripted - cloaked
|
||||
while humans < cloaked + 2:
|
||||
cloaked -= 1
|
||||
humans += 1
|
||||
return scripted, cloaked, humans
|
||||
|
||||
|
||||
def _cover(rng: random.Random, n: int, cloaked: int) -> tuple[list[Style], set[int]]:
|
||||
"""`n` styles, then a uniform random choice of which of them belong to bots.
|
||||
|
||||
Half the styles are efficient — the corner a naive flagger points at — and there are
|
||||
strictly more of them than there are cloaked accounts, so at least one efficient HUMAN
|
||||
is in every batch by construction. That account is the expensive false positive the
|
||||
whole environment is built around; leaving its presence to chance would leave the cost
|
||||
asymmetry absent from some batches entirely.
|
||||
|
||||
The population subset is drawn AFTER the styles and independently of them, so
|
||||
P(bot | style) is `cloaked / n` for every style there is. No threshold, band, ranking
|
||||
or joint rule over the cover columns beats the batch's base rate, and that is a
|
||||
property of the draw order rather than of the numbers, so it cannot drift.
|
||||
|
||||
An odd count gives the spare style to a corner chosen by a coin, so the expected share
|
||||
is exactly one half for every batch shape there is. Rounding it up instead made the
|
||||
share 6/11 in a batch with one naive script and 5/10 in a batch with two — and the
|
||||
script count moves the bot rate as well, so pooling batches would tie the corner to
|
||||
`is_bot` even though within any single batch it cannot. Adding the coin unconditionally
|
||||
is the same bug wearing the other sign: it would make the share 5.5/10 on even counts.
|
||||
"""
|
||||
efficient = n // 2 + (rng.randint(0, 1) if n % 2 else 0)
|
||||
assert efficient > cloaked, "not enough efficient styles to guarantee an efficient human"
|
||||
corners = [True] * efficient + [False] * (n - efficient)
|
||||
rng.shuffle(corners)
|
||||
styles = [_style(rng, corner) for corner in corners]
|
||||
return styles, set(rng.sample(range(n), cloaked))
|
||||
|
||||
|
||||
def _tells(rng: random.Random, cloaked: int) -> list[str]:
|
||||
"""One tell per cloaked account, all three present, and the spare slots decided by the
|
||||
seed: cycling in a fixed order would make the mix identical in every batch, and a model
|
||||
can fit a constant."""
|
||||
order = list(TELLS) + [rng.choice(TELLS) for _ in range(cloaked - len(TELLS))]
|
||||
rng.shuffle(order)
|
||||
return order
|
||||
|
||||
|
||||
def build_slices(seed: int, graded: int) -> tuple[list[Account], list[Account]]:
|
||||
"""The labelled examples, and the unlabelled batch the answer is graded on.
|
||||
|
||||
The labelled slice is a scripted bot, two humans and one cloaked account per tell. Both
|
||||
corners appear on both sides of the label — an efficient human and an efficient bot, a
|
||||
casual human and a casual bot — so the six examples teach that the cover is worthless
|
||||
and the behaviour is not. A slice where every bot was efficient would teach the
|
||||
shortcut instead, which is a subtler version of the defect that killed the first draft.
|
||||
|
||||
Nothing about the graded batch is derivable from the labelled one except the rule,
|
||||
which is the point: the accounts differ, the generator does not.
|
||||
"""
|
||||
rng = random.Random(seed)
|
||||
scripted, cloaked, humans = _plan(rng, graded)
|
||||
ids = _ids(rng, 6 + graded)
|
||||
|
||||
# Two efficient and two casual styles, dealt one of each to a human and to a bot.
|
||||
demo = [_style(rng, True), _style(rng, True), _style(rng, False), _style(rng, False)]
|
||||
demo_tells = _tells(rng, 3)
|
||||
labelled = [
|
||||
_scripted(rng, ids[0]),
|
||||
_wearing(rng, ids[1], demo[0], ""),
|
||||
_wearing(rng, ids[2], demo[2], ""),
|
||||
_wearing(rng, ids[3], demo[1], demo_tells[0]),
|
||||
_wearing(rng, ids[4], demo[3], demo_tells[1]),
|
||||
_wearing(rng, ids[5], _style(rng, rng.random() < 0.5), demo_tells[2]),
|
||||
]
|
||||
rng.shuffle(labelled)
|
||||
|
||||
styles, bot_slots = _cover(rng, cloaked + humans, cloaked)
|
||||
tells = iter(_tells(rng, cloaked))
|
||||
batch = [_scripted(rng, ids[6 + i]) for i in range(scripted)]
|
||||
for i, style in enumerate(styles):
|
||||
batch.append(
|
||||
_wearing(rng, ids[6 + scripted + i], style, next(tells) if i in bot_slots else "")
|
||||
)
|
||||
rng.shuffle(batch)
|
||||
return labelled, batch
|
||||
|
||||
|
||||
def render(account: Account, *, label: bool) -> str:
|
||||
"""One account as the log the agent reads. `break_s` can be empty — an account that
|
||||
never pauses has nothing to write there, and the blank line is itself the observation."""
|
||||
head = f"[{account.acc_id}]"
|
||||
if label:
|
||||
head += f" label: {'BOT' if account.is_bot else 'PLAYER'}"
|
||||
return "\n".join([
|
||||
head,
|
||||
f" click_dt_ms {' '.join(str(v) for v in account.click_dt_ms)}",
|
||||
f" break_s {' '.join(str(v) for v in account.break_s) or '(no breaks logged)'}",
|
||||
f" session_min {' '.join(str(v) for v in account.session_min)}",
|
||||
f" login_gap_min {' '.join(str(v) for v in account.login_gap_min)}",
|
||||
f" misclick_next {' '.join(account.misclick_next)}",
|
||||
f" route_ids {' '.join(account.route_ids)}",
|
||||
])
|
||||
@@ -0,0 +1,214 @@
|
||||
"""Scoring an accusation list, against a reference discriminator rather than against truth.
|
||||
|
||||
The score is normalised to what a discriminator built only from the labelled examples
|
||||
catches on this batch — `reference_bots` below, three threshold tests on three columns of
|
||||
the log. That indirection is deliberate and it is what makes 1.000 reachable BY
|
||||
CONSTRUCTION: if the generator ever drifts so that one cloaked account stops leaking, the
|
||||
reference misses it too and the ceiling stays at 1.000 instead of quietly moving out of
|
||||
reach. Grading against ground truth would have made that drift look like a model failure.
|
||||
|
||||
The reference is only legitimate because it reads the same columns the agent reads. It has
|
||||
no access to `Account.population`; the probe asserts it is nonetheless perfect, which is
|
||||
the "the signal exists" check — a discriminator that cannot beat guessing means the batch
|
||||
is noise and a model's score on it means nothing.
|
||||
|
||||
Four quantities come out of one pass:
|
||||
|
||||
detection bots caught, over what the reference caught. Clipped, so beating the
|
||||
reference is never punished.
|
||||
restraint people left alone, SQUARED. Linear, a wrongful ban and a missed bot cost
|
||||
almost exactly the same (0.42 against 0.38 on a twelve-account batch) and
|
||||
the environment stops teaching the asymmetry it exists to teach. A wrongful
|
||||
ban is a support ticket, an appeal, and a player who does not come back; the
|
||||
second one costs more than the first, and squaring is that shape.
|
||||
purity true positives over the accusations made, floored at the reference's count.
|
||||
The floor is what stops one confident accusation scoring like a full sweep:
|
||||
without it, naming the single most obvious script earns perfect precision.
|
||||
gate every bot the reference found, and not one person. Binary; the only thing a
|
||||
moderation team would actually ship.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
import statistics
|
||||
from dataclasses import dataclass
|
||||
|
||||
from bot_detection.accounts import DEAD_BAND, Account
|
||||
|
||||
MAX_ACCUSATIONS = 64
|
||||
|
||||
# The thresholds sit in the middle of gaps that are visible in the labelled slice: humans
|
||||
# correct at least five misclicks in eight and bots that leak here correct none; human
|
||||
# login gaps spread by hours and a scheduler's by minutes. Anything an agent could not read
|
||||
# off the examples would not be a fair reference.
|
||||
CORRECTION_FLOOR = 0.35
|
||||
LOGIN_SPREAD_FLOOR_MIN = 30
|
||||
|
||||
|
||||
def correction_share(account: Account) -> float:
|
||||
if not account.misclick_next:
|
||||
return 1.0
|
||||
return account.misclick_next.count("fix") / len(account.misclick_next)
|
||||
|
||||
|
||||
def deadband_breaks(account: Account) -> int:
|
||||
return sum(1 for b in account.break_s if DEAD_BAND[0] <= b < DEAD_BAND[1])
|
||||
|
||||
|
||||
def login_spread_min(account: Account) -> int:
|
||||
"""Range of the gaps between session starts. A person's varies by hours."""
|
||||
if len(account.login_gap_min) < 2:
|
||||
return 10_000
|
||||
return max(account.login_gap_min) - min(account.login_gap_min)
|
||||
|
||||
|
||||
def click_cv(account: Account) -> float:
|
||||
"""The naive statistic, exported so the probe can play the strategy it represents.
|
||||
|
||||
It finds the naive scripts, whose delays are near-constant, and nothing else: a cloaked
|
||||
bot and a person draw their click latency from one pool of lognormal parameters that
|
||||
was sampled before either was assigned a population, so no threshold on this column
|
||||
tells them apart. It is here to be measured failing, not to be used.
|
||||
"""
|
||||
dts = account.click_dt_ms
|
||||
if len(dts) < 2:
|
||||
return 0.0
|
||||
mean = statistics.fmean(dts)
|
||||
return statistics.stdev(dts) / mean if mean else 0.0
|
||||
|
||||
|
||||
def log_click_sd(account: Account) -> float:
|
||||
"""The sharpest form of the naive statistic — the sigma of the lognormal, recovered.
|
||||
|
||||
Exported because it is the attack that broke the first version of this environment: the
|
||||
populations were drawn from disjoint sigma ranges, so a cut at 0.52 separated the
|
||||
sophisticated bot from the efficient human and a rule reading only this column and
|
||||
`route_ids` scored 0.953. The probe asserts, over thousands of accounts, that no cut
|
||||
and no two-sided band on it now beats the base rate.
|
||||
"""
|
||||
dts = [d for d in account.click_dt_ms if d > 0]
|
||||
if len(dts) < 2:
|
||||
return 0.0
|
||||
return statistics.stdev(math.log(d) for d in dts)
|
||||
|
||||
|
||||
def session_mean(account: Account) -> float:
|
||||
"""Hours at the keyboard. The other column that used to be an identity: humans were the
|
||||
only population whose sessions ran under 200 minutes, in all 7200 accounts checked."""
|
||||
return statistics.fmean(account.session_min) if account.session_min else 0.0
|
||||
|
||||
|
||||
def distinct_routes(account: Account) -> int:
|
||||
"""The third decoy: an account that found the fast route never leaves it. Drawn from
|
||||
the style, and the style is drawn before anybody is a bot."""
|
||||
return len(set(account.route_ids))
|
||||
|
||||
|
||||
def reference_bots(accounts: list[Account]) -> set[str]:
|
||||
"""The strategy the reward is normalised against: three tests on three channels.
|
||||
|
||||
Any one of them alone catches roughly a third of the cloaked accounts, because each
|
||||
cloaked account leaks on exactly one. Together they are exhaustive. Nothing here reads
|
||||
a label — this runs on the same rendered columns the agent is given.
|
||||
"""
|
||||
accused = set()
|
||||
for account in accounts:
|
||||
if correction_share(account) < CORRECTION_FLOOR:
|
||||
accused.add(account.acc_id)
|
||||
elif deadband_breaks(account) > 0:
|
||||
accused.add(account.acc_id)
|
||||
elif login_spread_min(account) < LOGIN_SPREAD_FLOOR_MIN:
|
||||
accused.add(account.acc_id)
|
||||
return accused
|
||||
|
||||
|
||||
@dataclass
|
||||
class Outcome:
|
||||
"""What one accusation list did to one batch."""
|
||||
|
||||
true_positives: int
|
||||
false_positives: int
|
||||
bots_total: int
|
||||
humans_total: int
|
||||
reference_caught: int
|
||||
accused_total: int
|
||||
unknown_ids: int
|
||||
"""Ids that are not in the graded batch. Ignored rather than counted against the agent:
|
||||
naming an account that does not exist is a formatting slip, not a wrongful ban."""
|
||||
|
||||
@property
|
||||
def detection(self) -> float:
|
||||
if not self.reference_caught:
|
||||
return 0.0
|
||||
return min(1.0, self.true_positives / self.reference_caught)
|
||||
|
||||
@property
|
||||
def restraint(self) -> float:
|
||||
if not self.humans_total:
|
||||
return 1.0
|
||||
return max(0.0, 1.0 - self.false_positives / self.humans_total) ** 2
|
||||
|
||||
@property
|
||||
def purity(self) -> float:
|
||||
"""Zero for an empty accusation list. An empty list has no false positives and
|
||||
also no evidence; paying it for restraint would make silence the safe play, which
|
||||
is the same defect that put redaction-pressure's floor above zero."""
|
||||
made = self.true_positives + self.false_positives
|
||||
denominator = max(made, self.reference_caught)
|
||||
return self.true_positives / denominator if denominator else 0.0
|
||||
|
||||
@property
|
||||
def gate(self) -> bool:
|
||||
return (
|
||||
self.reference_caught > 0
|
||||
and self.true_positives >= self.reference_caught
|
||||
and self.false_positives == 0
|
||||
)
|
||||
|
||||
|
||||
def measure(accounts: list[Account], accused: list[str]) -> Outcome:
|
||||
by_id = {a.acc_id: a for a in accounts}
|
||||
# Deduplicated: naming the same account twice is one accusation, and counting it twice
|
||||
# would let a model inflate its own precision denominator into a rounding error.
|
||||
named = []
|
||||
for raw in accused[:MAX_ACCUSATIONS]:
|
||||
key = str(raw).strip().upper()
|
||||
if key not in named:
|
||||
named.append(key)
|
||||
|
||||
unknown = sum(1 for key in named if key not in by_id)
|
||||
hits = [by_id[key] for key in named if key in by_id]
|
||||
return Outcome(
|
||||
true_positives=sum(1 for a in hits if a.is_bot),
|
||||
false_positives=sum(1 for a in hits if not a.is_bot),
|
||||
bots_total=sum(1 for a in accounts if a.is_bot),
|
||||
humans_total=sum(1 for a in accounts if not a.is_bot),
|
||||
reference_caught=len(reference_bots(accounts) & {a.acc_id for a in accounts if a.is_bot}),
|
||||
accused_total=len(hits),
|
||||
unknown_ids=unknown,
|
||||
)
|
||||
|
||||
|
||||
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
|
||||
|
||||
|
||||
def parse_accusations(reply: str) -> list[str]:
|
||||
"""The last JSON block in the reply, as either `{"bots": [...]}` or a bare array.
|
||||
|
||||
A reply that will not parse is an empty accusation list, not an error — it scores what
|
||||
accusing nobody scores, which is zero.
|
||||
"""
|
||||
blocks = _BLOCK.findall(reply or "")
|
||||
raw = blocks[-1] if blocks else (reply or "")
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
if isinstance(parsed, dict):
|
||||
parsed = parsed.get("bots", [])
|
||||
if not isinstance(parsed, list):
|
||||
return []
|
||||
return [item for item in parsed if isinstance(item, str)]
|
||||
@@ -0,0 +1,147 @@
|
||||
"""bot-detection: name the scripts without banning the people who play like one.
|
||||
|
||||
Six labelled accounts, then a batch of unlabelled ones from the same generator. The labels
|
||||
on the batch are never in the prompt and never in the task data — only the seed is, and
|
||||
they are rebuilt at scoring time. What the agent is graded on is whether the rule it drew
|
||||
from six examples holds on accounts it has not been told the answer to.
|
||||
|
||||
The naive rule is "low timing variance means bot", and here it finds the naive scripts and
|
||||
stops. Every other account's cover — click latency, session length, route count — is drawn
|
||||
before the generator decides whether that account is a person, and the decision is a uniform
|
||||
coin over the styles it just drew. So P(bot | anything in those three columns) is the
|
||||
batch's bot rate exactly, and the efficient human who clicks tighter than half the bots is
|
||||
in every batch by construction rather than by luck.
|
||||
|
||||
What IS separable is behaviour: a bot that never corrects a misclick, one whose breaks are
|
||||
exponential and so land in a dead band a human's bimodal breaks never occupy, one that logs
|
||||
in on a fixed period. Each cloaked account leaks on exactly one of the three, so a
|
||||
discriminator built on any single channel ceilings at roughly a third of them. The class
|
||||
balance varies with the seed too — a fixed "six bots in twelve" lets a model rank by any
|
||||
weak score and take the top six, which was worth 0.974 of the ceiling when measured.
|
||||
|
||||
The reward is normalised against `reference_bots`, a discriminator that reads the same
|
||||
columns the agent reads. That is what makes 1.000 reachable by construction rather than by
|
||||
assumption: nothing here is scored against a ground truth the agent cannot get to.
|
||||
|
||||
Banning a real player is the expensive error, so restraint is weighted more heavily than
|
||||
detection and is squared — and neither reward can be collected without the other, because
|
||||
each is multiplied by the thing that qualifies it. Accusing nobody has no true positives,
|
||||
so it has no purity, so its perfect restraint is worth nothing.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import ClassVar
|
||||
|
||||
from pydantic import Field
|
||||
|
||||
import verifiers.v1 as vf
|
||||
|
||||
from bot_detection.accounts import build_slices, render
|
||||
from bot_detection.scan import measure, parse_accusations
|
||||
|
||||
SYSTEM = """You are reviewing activity logs for a game's anti-cheat team.
|
||||
|
||||
Each account log has six columns: the delay between consecutive clicks in milliseconds,
|
||||
the length of each logged break in seconds, the length of each play session in minutes,
|
||||
the minutes between consecutive session starts, what the account did after each misclick,
|
||||
and the route it took on each of twenty traversals of the same stretch of map.
|
||||
|
||||
You are given labelled examples first, then a batch to classify. Return the account ids you
|
||||
believe are bots, as one JSON object in a ```json code block:
|
||||
|
||||
```json
|
||||
{"bots": ["ACC-0000", "ACC-1111"]}
|
||||
```
|
||||
|
||||
Some of the people in this batch play more efficiently than some of the bots. Banning one
|
||||
of them costs more than missing a bot, and an accusation list you cannot defend is worse
|
||||
than a short one. Accounts you do not name are left alone."""
|
||||
|
||||
|
||||
class BotData(vf.TaskData):
|
||||
seed: int
|
||||
"""Rebuilds both slices exactly; no label ever reaches the task data."""
|
||||
graded: int
|
||||
|
||||
|
||||
class BotTask(vf.Task[BotData]):
|
||||
@vf.stop
|
||||
async def single_turn(self, trace: vf.Trace) -> bool:
|
||||
return trace.num_turns >= 1
|
||||
|
||||
@vf.metric
|
||||
async def scan(self, trace: vf.Trace) -> dict[str, float]:
|
||||
"""One pass over the graded batch; every reward reads this. Recomputing per reward
|
||||
is how the components drift apart when the scorer changes."""
|
||||
_, batch = build_slices(self.data.seed, self.data.graded)
|
||||
outcome = measure(batch, parse_accusations(trace.last_reply))
|
||||
return {
|
||||
"detection": outcome.detection,
|
||||
"restraint": outcome.restraint,
|
||||
"purity": outcome.purity,
|
||||
"clean": float(outcome.gate),
|
||||
"true_positives": float(outcome.true_positives),
|
||||
"false_positives": float(outcome.false_positives),
|
||||
"accused": float(outcome.accused_total),
|
||||
"unknown_ids": float(outcome.unknown_ids),
|
||||
# Reported separately because it is the finding the environment exists to
|
||||
# teach: a list can be free of false positives and still blind to the bots
|
||||
# that bothered to jitter.
|
||||
"reference_caught": float(outcome.reference_caught),
|
||||
}
|
||||
|
||||
@vf.reward(weight=0.35)
|
||||
async def caught(self, trace: vf.Trace) -> float:
|
||||
"""Bots found, voided by the players banned to find them."""
|
||||
return trace.metrics.get("detection", 0.0) * trace.metrics.get("restraint", 0.0)
|
||||
|
||||
@vf.reward(weight=0.40)
|
||||
async def spared(self, trace: vf.Trace) -> float:
|
||||
"""Players left alone — but only counted for a list that accused something real.
|
||||
Weighted above detection because that is the true cost ordering, and multiplied by
|
||||
purity rather than added beside it: a counterweight added as its own term is free
|
||||
points, which is how schema-migration paid 0.15 for touching nothing."""
|
||||
return trace.metrics.get("restraint", 0.0) * trace.metrics.get("purity", 0.0)
|
||||
|
||||
@vf.reward(weight=0.25)
|
||||
async def gate(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("clean", 0.0)
|
||||
|
||||
|
||||
class BotConfig(vf.TasksetConfig):
|
||||
num_tasks: int = Field(64, ge=1)
|
||||
graded: int = Field(12, ge=9)
|
||||
"""Unlabelled accounts in the batch. Nine is the floor: three cloaked accounts so all
|
||||
three tells fit, strictly more humans than that so an efficient human is guaranteed to
|
||||
be present to wrongly accuse, and at least one naive script."""
|
||||
|
||||
|
||||
class BotTaskset(vf.Taskset[BotTask, BotConfig]):
|
||||
SEED_BASE: ClassVar[int] = 80_000
|
||||
|
||||
def load(self) -> list[BotTask]:
|
||||
tasks = []
|
||||
for i in range(self.config.num_tasks):
|
||||
seed = self.SEED_BASE + i
|
||||
labelled, batch = build_slices(seed, self.config.graded)
|
||||
examples = "\n\n".join(render(a, label=True) for a in labelled)
|
||||
unlabelled = "\n\n".join(render(a, label=False) for a in batch)
|
||||
tasks.append(
|
||||
BotTask(
|
||||
BotData(
|
||||
idx=i,
|
||||
name=f"batch-{seed}",
|
||||
prompt=(
|
||||
f"Labelled examples:\n\n{examples}\n\n"
|
||||
f"Classify these {len(batch)} accounts:\n\n{unlabelled}\n\n"
|
||||
"Return the bots."
|
||||
),
|
||||
system_prompt=SYSTEM,
|
||||
seed=seed,
|
||||
graded=self.config.graded,
|
||||
),
|
||||
self.config.task,
|
||||
)
|
||||
)
|
||||
return tasks
|
||||
Reference in New Issue
Block a user