Three OSRS environments, and the first real model scores
grand-exchange, bot-detection and drop-table-inference. All three produce unbounded or irreducible-error scores, unlike the first four, so all three normalise against a REFERENCE STRATEGY rather than an absolute -- which is what makes 1.000 reachable rather than aspirational. drop-table-inference is the clearest case. Scoring KL against the true drop table would put the ceiling out of reach, because sampling error is irreducible: ground truth scores 0.900 against the Bayesian posterior's 1.000, and that ordering is correct. The best estimate available from 1,200 kills is not the true table, and an environment that demands it is measuring luck. grand-exchange needed the market to carry structure a model can actually infer, or profit is noise and no oracle exists. Measured over 120 baskets: the reference earns 38,821 gp and is profitable in 120/120, random orders lose 561, and trading only the random-walk items -- which look like the widest-swinging lines on the board -- loses 16,860. bot-detection generates naive bots, cloaked bots that jitter on purpose, and efficient humans who look bot-like on every naive statistic. Timing features are drawn BEFORE the generator decides who is a bot, so every latency rule sits at chance. Reference discriminator F1 1.000, random 0.430. Each was built by one agent then attacked by two independent reviewers on exploitability and soundness. They earned their keep: drop-table-inference's first reward let a memorised constant score 0.901 without opening the kill log, and its reference was 19% worse than a flat number. probe.py now carries three named fences asserting those attacks stay dead. First evaluation also lands. Nemotron 3.5 Lightning, thinking off, 32 rollouts per environment: 0.570 down to 0.059, every gate at or near zero, nothing solved and nothing unsolvable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,3 @@
|
||||
from drop_table_inference.taskset import DropTableTaskset
|
||||
|
||||
__all__ = ["DropTableTaskset"]
|
||||
@@ -0,0 +1,498 @@
|
||||
"""Parsing an estimated drop table, and scoring it against the held-out kills.
|
||||
|
||||
Every number here is normalised against a REFERENCE STRATEGY rather than an absolute,
|
||||
because the absolute is unreachable and rule 3 forbids dead components. Nobody recovers
|
||||
1/24576 exactly from 1,200 kills; sampling error is irreducible and a model reasoning
|
||||
perfectly still cannot beat it. So the ceiling is what good statistics allows on the SAME
|
||||
log the agent read.
|
||||
|
||||
WHICH reference is the whole design, and the first version got it wrong. It used the
|
||||
posterior mean under a Jeffreys prior — a closed form, which meant two things. Every
|
||||
zero-count item got `0.5/(kills + 0.5(n+1))` regardless of anything else in the log, so the
|
||||
tail term was maxed by a memorised constant that never opened the kill counts: measured over
|
||||
96 tasks, a blind reply assigning one number to every item scored `rare` 1.000 on every
|
||||
single one. And the argmax of the reward was "replicate a formula", not "do the inference
|
||||
the generator was built to make learnable" — a real grid posterior scored BELOW an exact
|
||||
formula copy. Both are the same defect: a reference that is a function of (count, kills)
|
||||
cannot reward reading the rest of the table.
|
||||
|
||||
The reference here is the posterior over the generator's OWN hypothesis space — every
|
||||
assignment of items to distinct rungs of the published ladder, weighted by the multinomial
|
||||
likelihood of the visible counts and by each item's PRICE, filtered by the published mass
|
||||
window, and divided by the probability that the assignment would have produced a silent item
|
||||
at all (the stream is rejected until one does, so that conditioning is evidence and ignoring
|
||||
it leaves information in the prompt). Everything it uses is in the prompt; none of it is the
|
||||
answer. Because rungs are not reused, what an unobserved item can be depends on what the
|
||||
observed ones already are, and because prices are lognormal in the rate, two items that are
|
||||
both silent are no longer interchangeable. Measured, that is the difference between a
|
||||
reference that beats the best memorised constant on the tail by 8% and one that beats it by
|
||||
4.75x.
|
||||
|
||||
Three terms, one counterweight, and the counterweight multiplies:
|
||||
|
||||
fit forward divergence ratio, KL(held-out || reference) / KL(held-out || estimate).
|
||||
Dominated by the common items, because forward KL weights every term by the
|
||||
truth's own mass. Graded against the held-out EMPIRICAL distribution, not the true
|
||||
table: the environment grades what the continuation showed.
|
||||
|
||||
rare mean squared LOG-ratio on the items that dropped ZERO times, normalised by the
|
||||
reference's own error. Graded against the TRUE rate, because forty thousand kills
|
||||
cannot resolve 1/24576 either and grading the tail on the continuation would grade
|
||||
the sample. Log-ratio, not probability difference, because the Bayes act for
|
||||
squared log error is exp(E[ln p]) — exactly what `_posterior` returns — so no
|
||||
point estimate whatsoever beats the reference in expectation. That is the property
|
||||
the first version lacked, where a flat constant beat the reference's own tail
|
||||
accuracy by 19% and the clip reported both as 1.000.
|
||||
|
||||
gate binary: valid, and as good as the reference on all three, less a small margin.
|
||||
|
||||
restraint the counterweight, and it MULTIPLIES into `rare` rather than sitting beside it
|
||||
— the schema-migration lesson is that a counterweight scored separately is free
|
||||
points, and this one is maxed trivially by asserting that nothing unobserved
|
||||
exists. It is the reverse divergence over the unobserved items, positive part
|
||||
only: the nats an estimate spends claiming a silent item drops more often than the
|
||||
continuation showed. Reverse because reverse KL weights by the ESTIMATE's mass and
|
||||
is therefore the only term that can see a tail worth two parts in a thousand.
|
||||
One-sided because under-claiming is what `rare` already prices, and a two-sided
|
||||
reverse divergence turned out to be head accuracy measured a second time: its
|
||||
ratio correlates 0.98 with the forward one over 240 scored estimates, so
|
||||
multiplying the two together squared the same error.
|
||||
|
||||
Validity is also a multiplier. An estimate whose probabilities sum past 1 is not a
|
||||
distribution, and inflating everything zeroes the episode rather than paying for reach.
|
||||
|
||||
MIN_PROBABILITY is the other thing that has to be explicit. KL punishes an asserted zero
|
||||
without bound, so one zero against an item that appears in the graded slice makes every
|
||||
score infinite and every comparison meaningless. Clamping puts a price on it — ln(p/eps)
|
||||
per unit of the truth's mass — instead of an exception.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from functools import lru_cache
|
||||
from itertools import combinations
|
||||
|
||||
from drop_table_inference.table import (
|
||||
COMMON_DENOMINATOR,
|
||||
COMMON_NUMERATORS,
|
||||
LADDER_DENOMINATORS,
|
||||
MAX_ITEM_MASS,
|
||||
MIN_ITEM_MASS,
|
||||
N_COMMON,
|
||||
N_LADDER,
|
||||
VALUE_EXPONENT,
|
||||
VALUE_LOG_SD,
|
||||
VALUE_SCALE,
|
||||
Stream,
|
||||
)
|
||||
|
||||
# The floor below which "vanishingly rare" and "impossible" stop being distinguished.
|
||||
MIN_PROBABILITY = 1e-6
|
||||
# Slack for an estimate written to four decimal places. Anything past it is an agent
|
||||
# claiming more than one drop per kill.
|
||||
SUM_TOLERANCE = 1e-3
|
||||
# The handicap the reference concedes to the gate. Without one the verdict turned on the
|
||||
# fifth decimal place — the reference rounded to six decimals failed its own gate on five
|
||||
# tasks in eight, which measures float noise. Five percent is wide enough that it does not,
|
||||
# and narrow enough to be a gate: sweeping it from 0.005 to 0.20 moves the best non-oracle
|
||||
# strategy's pass rate from 0.062 to 0.083, because the gap it guards is not a rounding gap.
|
||||
GATE_MARGIN = 0.05
|
||||
# Guards for the ratios. Both denominators are strictly positive in practice — the
|
||||
# divergences carry the held slice's own sampling error and the tail is never resolved
|
||||
# exactly — so these only ever protect against a degenerate seed, and they are three orders
|
||||
# of magnitude below the typical value of the quantity they are added to.
|
||||
KL_EPSILON = 1e-9
|
||||
TAIL_EPSILON = 1e-6
|
||||
# Over-claim below this many nats is free. A good estimate's whole forward divergence is
|
||||
# around 1e-3, and an unobserved item priced at twice what the continuation showed costs
|
||||
# about 7e-5 — so this is the line between "rounded the tail up" and "bought coverage".
|
||||
# Without a floor the term would divide by zero on the tasks where the reference happens to
|
||||
# under-claim everywhere, and a counterweight that fires on float noise is not one.
|
||||
OVERCLAIM_FLOOR = 1e-4
|
||||
# Hypotheses this many log units below an item's best rung are dropped before the
|
||||
# enumeration. e^-32 is 1e-14 of the leading term: it changes the posterior mean in the
|
||||
# fourteenth decimal place and cuts the search by two orders of magnitude.
|
||||
PRUNE_LOG = 32.0
|
||||
|
||||
SLOT_RATES = tuple(
|
||||
[n / COMMON_DENOMINATOR for n in COMMON_NUMERATORS] + [1.0 / d for d in LADDER_DENOMINATORS]
|
||||
)
|
||||
COMMON_SLOTS = len(COMMON_NUMERATORS)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Outcome:
|
||||
"""What one estimate did against one held-out continuation."""
|
||||
|
||||
valid: bool
|
||||
kl_reference: float
|
||||
kl_estimate: float
|
||||
over_reference: float
|
||||
over_estimate: float
|
||||
tail_reference: float
|
||||
tail_estimate: float
|
||||
rare_items: int
|
||||
|
||||
@property
|
||||
def restraint(self) -> float:
|
||||
"""The price of tail coverage: reverse divergence over the unobserved items, kept
|
||||
only where the estimate claims MORE than the continuation produced.
|
||||
|
||||
Reverse rather than forward because reverse KL weights by the ESTIMATE's own mass,
|
||||
which is the only way a term ever notices the tail — the tail is two parts in a
|
||||
thousand of the distribution and forward KL, weighted by the truth, cannot see it.
|
||||
One-sided because under-claiming is already what `rare` is for, and a two-sided
|
||||
reverse divergence is head accuracy measured a second time — across five strategies
|
||||
and 48 tasks its ratio correlates 0.98 with the forward one, so multiplying the two
|
||||
together squared the same error and left the mid-range with no gradient.
|
||||
|
||||
1.0 for an estimate that claims no more than the reference does. It is a multiplier
|
||||
on `rare` rather than a reward beside it — the schema-migration lesson is that a
|
||||
counterweight scored separately is free points, and this one in particular is
|
||||
maxed, trivially, by asserting that nothing unobserved exists.
|
||||
"""
|
||||
if not self.valid:
|
||||
return 0.0
|
||||
return min(1.0, (self.over_reference + OVERCLAIM_FLOOR)
|
||||
/ (self.over_estimate + OVERCLAIM_FLOOR))
|
||||
|
||||
@property
|
||||
def raw_fit(self) -> float:
|
||||
if not self.valid:
|
||||
return 0.0
|
||||
return min(1.0, (self.kl_reference + KL_EPSILON) / (self.kl_estimate + KL_EPSILON))
|
||||
|
||||
@property
|
||||
def raw_rare(self) -> float:
|
||||
if not self.valid or self.rare_items == 0:
|
||||
return 0.0
|
||||
return min(1.0, (self.tail_reference + TAIL_EPSILON) / (self.tail_estimate + TAIL_EPSILON))
|
||||
|
||||
@property
|
||||
def fit(self) -> float:
|
||||
return self.raw_fit
|
||||
|
||||
@property
|
||||
def rare(self) -> float:
|
||||
return self.raw_rare * self.restraint
|
||||
|
||||
@property
|
||||
def clean(self) -> bool:
|
||||
"""The verdict: a valid distribution, and as good as the reference on ALL THREE.
|
||||
|
||||
All three, not any of them. Matching the reference on divergence alone is what
|
||||
frequency-copying nearly does — it is right about the common items, which is where
|
||||
the divergence lives. Matching it on the tail alone is what an agent does when it
|
||||
sprays a number over the unobserved items, and restraint is what prices that.
|
||||
"""
|
||||
return (
|
||||
self.valid
|
||||
and self.raw_fit >= 1.0 - GATE_MARGIN
|
||||
and self.raw_rare >= 1.0 - GATE_MARGIN
|
||||
and self.restraint >= 1.0 - GATE_MARGIN
|
||||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=4096)
|
||||
def _split_weights(n_items: int) -> dict[tuple[int, int], float]:
|
||||
"""Log prior mass per ASSIGNMENT for each tier split consistent with `n_items`.
|
||||
|
||||
Per assignment, not per split, because the enumeration visits assignments: a split with
|
||||
more mass-valid tables spreads the same prior over more of them. The generator draws
|
||||
`n_common` and `n_ladder` uniformly and independently, then samples rungs without
|
||||
replacement and shuffles, so every labelled assignment inside a split is equally likely
|
||||
and the count is C(n, c) * c! * l! * (mass-valid rung-set pairs).
|
||||
"""
|
||||
weights: dict[tuple[int, int], float] = {}
|
||||
for n_common in N_COMMON:
|
||||
for n_ladder in N_LADDER:
|
||||
if n_common + n_ladder != n_items:
|
||||
continue
|
||||
valid = 0
|
||||
for nums in combinations(COMMON_NUMERATORS, n_common):
|
||||
head = sum(n / COMMON_DENOMINATOR for n in nums)
|
||||
for dens in combinations(LADDER_DENOMINATORS, n_ladder):
|
||||
total = head + sum(1.0 / d for d in dens)
|
||||
if MIN_ITEM_MASS <= total <= MAX_ITEM_MASS:
|
||||
valid += 1
|
||||
if not valid:
|
||||
continue
|
||||
arrangements = math.comb(n_items, n_common) * math.factorial(n_common) * math.factorial(n_ladder)
|
||||
weights[(n_common, n_ladder)] = -math.log(valid * arrangements)
|
||||
return weights
|
||||
|
||||
|
||||
LOG_SLOT_VALUE = tuple(math.log(VALUE_SCALE) + VALUE_EXPONENT * math.log(1.0 / rate)
|
||||
for rate in SLOT_RATES)
|
||||
|
||||
|
||||
@lru_cache(maxsize=4096)
|
||||
def _posterior(counts: tuple[int, ...], values: tuple[int, ...], kills: int) -> tuple[float, ...]:
|
||||
"""Posterior geometric-mean rate per item, in the order the counts were given.
|
||||
|
||||
Enumeration rather than a closed form because the hypotheses are COUPLED: no two items
|
||||
share a rung, the total has to land in the published mass window, and the stream was
|
||||
redrawn until something was silent. Every one of those couples an item's estimate to
|
||||
the rest of the log, and a closed form has none of them — which is precisely why the
|
||||
closed form's tail was a constant.
|
||||
"""
|
||||
n = len(counts)
|
||||
splits = _split_weights(n)
|
||||
if not splits:
|
||||
raise ValueError(f"no tier split accounts for {n} items")
|
||||
|
||||
# Per-item candidate rungs, pruned on the BINOMIAL marginal but carrying the
|
||||
# MULTINOMIAL term. The joint likelihood does not factorise — the no-drop term couples
|
||||
# every item — so only `count * ln(rate)` belongs to the item; the marginal is used to
|
||||
# prune because a rung 1e-14 down on its own item's evidence cannot be rescued by a
|
||||
# no-drop term shared with every other hypothesis.
|
||||
candidates: list[list[tuple[int, float]]] = []
|
||||
for count, value in zip(counts, values):
|
||||
log_value = math.log(value)
|
||||
scores = []
|
||||
for index, rate in enumerate(SLOT_RATES):
|
||||
# The price term. It is the only thing that tells two silent items apart —
|
||||
# nothing in a kill log does — so without it the reference beats the best
|
||||
# memorised constant on the tail by 8%, which is a coin. See table.VALUE_SCALE.
|
||||
price = -((log_value - LOG_SLOT_VALUE[index]) ** 2) / (2.0 * VALUE_LOG_SD ** 2)
|
||||
marginal = count * math.log(rate) + (kills - count) * math.log1p(-rate) + price
|
||||
scores.append((index, marginal, count * math.log(rate) + price))
|
||||
best = max(marginal for _, marginal, _ in scores)
|
||||
candidates.append([(index, term) for index, marginal, term in scores
|
||||
if marginal >= best - PRUNE_LOG])
|
||||
|
||||
# Most-constrained item first. The commons have two plausible rungs each and pin the
|
||||
# mass; opening them first kills whole subtrees before the tail is ever touched.
|
||||
order = sorted(range(n), key=lambda i: len(candidates[i]))
|
||||
max_common, max_ladder = max(N_COMMON), max(N_LADDER)
|
||||
nothing = kills - sum(counts)
|
||||
hypotheses: list[tuple[float, tuple[float, ...]]] = []
|
||||
assignment = [0.0] * n
|
||||
|
||||
def walk(depth: int, used: int, n_used_common: int, mass: float, loglik: float) -> None:
|
||||
if mass > MAX_ITEM_MASS:
|
||||
return
|
||||
if depth == n:
|
||||
n_used_ladder = n - n_used_common
|
||||
prior = splits.get((n_used_common, n_used_ladder))
|
||||
if prior is None or mass < MIN_ITEM_MASS:
|
||||
return
|
||||
# P(some item is silent | this table). The stream was rejected until one was,
|
||||
# so dividing it out is Bayes, not a fudge: a table of shallow rungs rarely
|
||||
# produces a silent item and the fact that this one did is evidence for it.
|
||||
# The items are weakly negatively correlated under the multinomial; treating
|
||||
# the silences as independent is accurate to a fraction of a percent at these
|
||||
# rates and is the only closed form available.
|
||||
quiet = 1.0
|
||||
for rate in assignment:
|
||||
quiet *= 1.0 - math.exp(kills * math.log1p(-rate))
|
||||
# A table under which every item was near-certain to appear cannot have
|
||||
# produced a log that was accepted for containing a silent one. Float rounds
|
||||
# that probability to exactly 1 for an all-common assignment, so the hypothesis
|
||||
# is dropped rather than divided by zero.
|
||||
if quiet >= 1.0:
|
||||
return
|
||||
total = loglik + nothing * math.log1p(-mass) + prior - math.log1p(-quiet)
|
||||
hypotheses.append((total, tuple(assignment)))
|
||||
return
|
||||
item = order[depth]
|
||||
for index, term in candidates[item]:
|
||||
if used >> index & 1:
|
||||
continue
|
||||
is_common = index < COMMON_SLOTS
|
||||
if is_common and n_used_common == max_common:
|
||||
continue
|
||||
if not is_common and depth - n_used_common == max_ladder:
|
||||
continue
|
||||
rate = SLOT_RATES[index]
|
||||
assignment[item] = rate
|
||||
walk(depth + 1, used | 1 << index, n_used_common + is_common,
|
||||
mass + rate, loglik + term)
|
||||
assignment[item] = 0.0
|
||||
|
||||
walk(0, 0, 0, 0.0, 0.0)
|
||||
if not hypotheses:
|
||||
# Nothing in the published structure explains this log. Cannot happen for a stream
|
||||
# this module's own generator produced; falling back to the smoothed frequency
|
||||
# keeps a caller who reached here with hand-built counts from getting an exception
|
||||
# instead of a number.
|
||||
denominator = kills + 0.5 * (n + 1)
|
||||
return tuple((count + 0.5) / denominator for count in counts)
|
||||
|
||||
peak = max(score for score, _ in hypotheses)
|
||||
weight_total = 0.0
|
||||
means = [0.0] * n
|
||||
for score, rates in hypotheses:
|
||||
weight = math.exp(score - peak)
|
||||
weight_total += weight
|
||||
for i, rate in enumerate(rates):
|
||||
means[i] += weight * math.log(rate)
|
||||
# The GEOMETRIC posterior mean, exp(E[ln p]). Arithmetic would be the Bayes act for
|
||||
# squared error in probability, and on the tail that is a loss no scoring rule should
|
||||
# use: an item that could be 1/512 or 1/32768 has an arithmetic mean pinned by the top
|
||||
# rung, so the estimate is wrong by a factor of thirty whenever the deep rung is the
|
||||
# truth. Rates live on a log axis — a drop table is quoted as "one in n" — and the
|
||||
# tail term below is squared log-ratio, whose Bayes act is exactly this. That pairing
|
||||
# is what makes "some constant beats the reference" false by construction rather than
|
||||
# by luck.
|
||||
return tuple(math.exp(mean / weight_total) for mean in means)
|
||||
|
||||
|
||||
def reference_estimate(items: list[str], counts: dict[str, int], values: dict[str, int],
|
||||
kills: int) -> dict[str, float]:
|
||||
"""The reference strategy, as an estimate an agent could have written.
|
||||
|
||||
Computable from the prompt alone — that is the requirement it exists to satisfy. A
|
||||
reference that needed the true table would put the ceiling somewhere the agent cannot
|
||||
reason its way to, which is the unreachable-ceiling failure wearing a different hat.
|
||||
"""
|
||||
means = _posterior(tuple(counts.get(item, 0) for item in items),
|
||||
tuple(values[item] for item in items), kills)
|
||||
return dict(zip(items, means))
|
||||
|
||||
|
||||
def _distribution(items: list[str], estimate: dict[str, float]) -> list[float]:
|
||||
"""The estimate as a full distribution over items plus no-drop, clamped and renormalised.
|
||||
|
||||
Renormalising after the clamp keeps this a probability vector so the divergence stays a
|
||||
divergence; it shifts nothing meaningfully, since the clamp only ever moves mass on the
|
||||
order of 1e-6.
|
||||
"""
|
||||
values = [max(estimate.get(item, 0.0), MIN_PROBABILITY) for item in items]
|
||||
values.append(max(1.0 - sum(estimate.get(item, 0.0) for item in items), MIN_PROBABILITY))
|
||||
total = sum(values)
|
||||
return [value / total for value in values]
|
||||
|
||||
|
||||
def _divergence(observed: list[float], predicted: list[float]) -> float:
|
||||
return sum(p * math.log(p / q) for p, q in zip(observed, predicted) if p > 0.0)
|
||||
|
||||
|
||||
def _overclaim(predicted: list[float], observed: list[float], tail: list[int]) -> float:
|
||||
"""Reverse divergence over the unobserved items, positive part only: the nats an
|
||||
estimate spends asserting that a silent item drops more often than it does."""
|
||||
return sum(predicted[i] * math.log(predicted[i] / observed[i])
|
||||
for i in tail if predicted[i] > observed[i])
|
||||
|
||||
|
||||
def _tail_loss(truth: list[float], predicted: list[float]) -> float:
|
||||
"""Mean squared LOG-ratio on the tail: how many factors of e the estimate is out by.
|
||||
|
||||
Squared log rather than squared probability, for two reasons that are the same reason.
|
||||
A drop rate is a quantity on a log axis — one in five hundred against one in thirty
|
||||
thousand is a factor of sixty, and a difference of 0.002 — so a linear loss would grade
|
||||
the shallowest zero-count item and ignore the rest. And the Bayes act for this loss is
|
||||
exp(E[ln p]), which is exactly what `_posterior` returns, so the reference cannot be
|
||||
beaten in expectation by any point estimate whatsoever. That is the property the first
|
||||
version lacked, where a flat 0.0002 beat the reference's own tail accuracy by 19% and
|
||||
the clip reported both as 1.000.
|
||||
"""
|
||||
if not truth:
|
||||
return 0.0
|
||||
return sum(math.log(p / t) ** 2 for t, p in zip(truth, predicted)) / len(truth)
|
||||
|
||||
|
||||
def measure(stream: Stream, estimate: dict[str, float] | None) -> Outcome:
|
||||
items = stream.table.items
|
||||
observed = [stream.held[item] / stream.held_kills for item in items]
|
||||
observed.append(stream.held_nothing / stream.held_kills)
|
||||
# Clamped for the same reason the estimate is: the reverse divergence divides by the
|
||||
# continuation, and an item the held slice happened to miss would make it infinite.
|
||||
observed = [max(value, MIN_PROBABILITY) for value in observed]
|
||||
observed = [value / sum(observed) for value in observed]
|
||||
|
||||
reference = _distribution(items, reference_estimate(
|
||||
items, stream.visible, stream.table.values, stream.visible_kills))
|
||||
# The tail is the items that dropped ZERO times in the visible slice — the generator
|
||||
# guarantees there is at least one, and it is the only place an estimate cannot be read
|
||||
# off the log. Defining it by true rate instead diluted it with 1/256 items that did
|
||||
# appear, where frequency-copying is fine, and the term stopped measuring anything.
|
||||
tail = [i for i, item in enumerate(items) if stream.visible[item] == 0]
|
||||
rates = [stream.table.rates[items[i]] for i in tail]
|
||||
base = Outcome(
|
||||
valid=False,
|
||||
kl_reference=_divergence(observed, reference),
|
||||
kl_estimate=0.0,
|
||||
over_reference=_overclaim(reference, observed, tail),
|
||||
over_estimate=0.0,
|
||||
tail_reference=_tail_loss(rates, [reference[i] for i in tail]),
|
||||
tail_estimate=0.0,
|
||||
rare_items=len(tail),
|
||||
)
|
||||
if estimate is None:
|
||||
return base
|
||||
predicted = _distribution(items, estimate)
|
||||
base.valid = True
|
||||
base.kl_estimate = _divergence(observed, predicted)
|
||||
base.over_estimate = _overclaim(predicted, observed, tail)
|
||||
base.tail_estimate = _tail_loss(rates, [predicted[i] for i in tail])
|
||||
return base
|
||||
|
||||
|
||||
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
|
||||
_FRACTION = re.compile(r"^\s*(\d+(?:\.\d+)?)\s*(?:/|\s+in\s+)\s*(\d+(?:\.\d+)?)\s*$")
|
||||
|
||||
|
||||
def _rate(value: object) -> float | None:
|
||||
"""A rate as a number, or as the fraction an agent that spotted the grid would write.
|
||||
`1/128` and `1 in 128` are the natural way to state a drop rate and refusing them would
|
||||
punish exactly the reasoning the environment is trying to reward."""
|
||||
if isinstance(value, bool):
|
||||
return None
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
if isinstance(value, str):
|
||||
match = _FRACTION.match(value)
|
||||
if match:
|
||||
numerator, denominator = float(match.group(1)), float(match.group(2))
|
||||
return numerator / denominator if denominator else None
|
||||
try:
|
||||
return float(value.strip())
|
||||
except ValueError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def parse_estimate(reply: str, items: list[str]) -> dict[str, float] | None:
|
||||
"""The last JSON object in the reply, keyed by item name.
|
||||
|
||||
None means "no usable estimate" and scores what doing nothing scores — a reply that
|
||||
cannot be parsed never raises, and an estimate that is not a sub-distribution is
|
||||
rejected here rather than being quietly renormalised into a valid one, because
|
||||
renormalising would make inflating every rate free.
|
||||
"""
|
||||
blocks = _BLOCK.findall(reply or "")
|
||||
raw = blocks[-1] if blocks else (reply or "")
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
if isinstance(parsed, list):
|
||||
pairs = {}
|
||||
for row in parsed:
|
||||
if isinstance(row, dict):
|
||||
name = row.get("item", row.get("name"))
|
||||
rate = row.get("rate", row.get("probability", row.get("p")))
|
||||
if isinstance(name, str):
|
||||
pairs[name] = rate
|
||||
parsed = pairs
|
||||
if not isinstance(parsed, dict):
|
||||
return None
|
||||
|
||||
lookup = {item.lower(): item for item in items}
|
||||
estimate: dict[str, float] = {}
|
||||
for key, value in parsed.items():
|
||||
item = lookup.get(str(key).strip().lower())
|
||||
if item is None:
|
||||
continue
|
||||
rate = _rate(value)
|
||||
if rate is None or not math.isfinite(rate) or rate < 0.0 or rate > 1.0:
|
||||
return None
|
||||
estimate[item] = rate
|
||||
if not estimate or sum(estimate.values()) > 1.0 + SUM_TOLERANCE:
|
||||
return None
|
||||
return estimate
|
||||
@@ -0,0 +1,252 @@
|
||||
"""The monster, its drop table, and the kills that came out of it.
|
||||
|
||||
A task is one monster. Its table is a set of invented items, each with a rate the agent
|
||||
never sees, and the log is a sample from it: a visible slice the agent reads and a held-out
|
||||
continuation it is graded on. Both come out of one kill stream, so the graded kills are
|
||||
literally the *next* ones the monster would have dropped — not a differently-seeded
|
||||
population that could differ in ways the agent could not have anticipated.
|
||||
|
||||
Four properties of the generator are load-bearing, and none is decoration:
|
||||
|
||||
A LADDER WITH NO REPEATS. Rates are not merely "structured", they are drawn WITHOUT
|
||||
REPLACEMENT from two published sets: commons are distinct multiples of 4/128, and the rest
|
||||
are distinct unit fractions from 1/128 down to 1/24576. Distinctness is the first half of
|
||||
why the tail is learnable. Nothing in a kill log distinguishes one item that dropped zero
|
||||
times from another — any estimator that is a function of (count, kills) alone must give them
|
||||
the same number, and if that number is also the same across TASKS then a memorised constant
|
||||
is a perfect tail estimate and the term is inert. That is exactly what the first version of
|
||||
this environment did: its reference gave every zero-count item (0+0.5)/(kills+0.5(n+1)), a
|
||||
value with four distinct settings across ninety-six tasks, so a reply that never opened the
|
||||
kill log scored 1.000 on the tail. With no repeats, the rungs the OBSERVED items occupy are
|
||||
the rungs the unobserved ones cannot: a table whose counts pin 1/128 and 1/384 leaves its
|
||||
silent item somewhere below, and one that pins nothing above 1/3072 leaves it somewhere much
|
||||
higher.
|
||||
|
||||
Distinctness alone was not enough, and the number that says so is 8%: with the ladder and
|
||||
the elimination but no price, the reference beat the best memorised constant on the tail
|
||||
loss by eight percent, and a frequency copy with a constant floor held 0.86 of the term.
|
||||
Rungs eliminate, but two SILENT items are still interchangeable, and the irreducible spread
|
||||
inside that pair swamped everything the elimination revealed. That is what VALUE_SCALE
|
||||
below is for.
|
||||
|
||||
A PRICE THAT RANKS WHAT THE LOG CANNOT. Every item carries a shop value, lognormal around
|
||||
SCALE * (1/rate) ** EXPONENT — rare drops are worth more, which is a fact about the genre
|
||||
rather than anybody's copy. It is the only channel that separates two silent items, and
|
||||
separating them is what turns the tail from a lottery into an inference: with it the
|
||||
reference beats the best memorised constant on the tail by 4.75x rather than by 8%, and the
|
||||
frequency-copy-with-a-floor attack falls from 0.86 of the term to 0.28. The scatter is
|
||||
deliberate and it is large — sigma 0.55 in log value against 3x rung spacing — so the price
|
||||
ranks the candidates and never decides them, and an agent that reads only the price scores
|
||||
0.46 where one that reads price, counts and leftover rungs together scores 1.000.
|
||||
|
||||
A ZERO-COUNT ITEM, ALWAYS. The kill stream is rejected and redrawn until at least one item
|
||||
on the list dropped zero times in the visible slice. That case is the entire environment:
|
||||
the item list says the item exists, the log says it never appeared, and an estimate of 0 is
|
||||
an infinitely strong claim about something the graded kills will demonstrate is false. A
|
||||
task where every item happened to show up does not pose the question, so it is not used.
|
||||
The rejection is part of the model — `estimate._posterior` divides it back out,
|
||||
because conditioning on "some item was silent" is evidence about the table and an estimator
|
||||
that ignores it is leaving information in the prompt.
|
||||
|
||||
HELD-OUT KILLS UNTIL EVERY ITEM APPEARS. The item list is a promise that all of these drop;
|
||||
the graded slice has to keep it, or an estimate of zero for the rarest item costs nothing.
|
||||
It is also what keeps the counterweight finite: `estimate.restraint` divides by what the
|
||||
continuation produced, and an item the continuation never produced would make it infinite.
|
||||
|
||||
No wiki content: every item, monster and word here is invented. The MECHANICS are the
|
||||
borrowed part, and mechanics are facts.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import bisect
|
||||
import math
|
||||
import random
|
||||
from dataclasses import dataclass
|
||||
|
||||
ADJECTIVES = [
|
||||
"Ashen", "Brackish", "Cindered", "Duskbound", "Ember", "Frostbit", "Gilded", "Hollow",
|
||||
"Ivory", "Jagged", "Kelpish", "Lodestone", "Murkwrought", "Nettled", "Pitchblack",
|
||||
"Quarried", "Saltworn", "Tarnished", "Umbral", "Verdigris",
|
||||
]
|
||||
NOUNS = [
|
||||
"talon", "sigil", "vertebra", "censer", "ingot", "lantern", "chitin", "warhorn",
|
||||
"tessera", "phial", "reliquary", "scale", "cog", "wick", "grimoire", "spur",
|
||||
"anvil-shard", "orb", "tabard", "quill",
|
||||
]
|
||||
MONSTER_PREFIX = ["Marrow", "Gloom", "Rime", "Slagborn", "Fenwater", "Barrow", "Cinder", "Hush"]
|
||||
MONSTER_KIND = ["Warden", "Basilisk", "Revenant", "Colossus", "Hierophant", "Drake", "Herald", "Ossifier"]
|
||||
|
||||
# The common grid, and it is coarse on purpose. Adjacent numerators sit 33-50% apart, so a
|
||||
# few hundred kills can actually tell them apart — a continuous numerator would put the
|
||||
# notches inside the sampling error, the structure would be real but unresolvable, and
|
||||
# "spot the grid" would be a slogan rather than a strategy.
|
||||
COMMON_DENOMINATOR = 128
|
||||
COMMON_NUMERATORS = (8, 12, 16, 24, 32, 40, 48)
|
||||
|
||||
# The rare ladder: unit fractions spaced by roughly a factor of two. No sample of 600 kills
|
||||
# resolves 1/1024 from 1/2048 on its own evidence, and that is the point — down here the
|
||||
# information is in which rungs the OTHER items have already taken.
|
||||
LADDER_DENOMINATORS = (128, 384, 1024, 3072, 8192, 24576)
|
||||
|
||||
# How many of each. Both are published to the agent, because the reference estimator uses
|
||||
# them and a reference the agent cannot compute is an unreachable ceiling wearing a hat.
|
||||
N_COMMON = (3, 4)
|
||||
N_LADDER = (4, 5, 6)
|
||||
|
||||
# The price law, and it is the reason the tail is a skill rather than a lottery.
|
||||
#
|
||||
# Two items that both dropped zero times are EXCHANGEABLE on the kill log: no statistic of
|
||||
# the counts distinguishes them, so every estimator must give them the same number, and
|
||||
# the reference's advantage over a memorised constant collapses to whatever the free-rung
|
||||
# set alone reveals. Measured, that was four to eight percent — the reference beat the best
|
||||
# constant on the tail by less than a tenth, which is not an environment, it is a coin.
|
||||
#
|
||||
# A price breaks the tie, and it is the OSRS-shaped way to break it: rare drops are worth
|
||||
# more. `value` is lognormal around SCALE * (1/rate) ** EXPONENT, so it ranks the silent
|
||||
# items without deciding them — at 3x rung spacing and this sigma, the price alone gets an
|
||||
# adjacent pair the right way round about six times in seven and gets nothing for free two
|
||||
# rungs down. The agent has to combine it with the counts and the leftover rungs, which is
|
||||
# the entire task.
|
||||
VALUE_SCALE = 15.0
|
||||
VALUE_EXPONENT = 0.8
|
||||
VALUE_LOG_SD = 0.55
|
||||
|
||||
# The share of kills that drop something. Bounded away from 1 so "no drop" is always a
|
||||
# substantial category: it is where the mass an agent does not assign to items has to go,
|
||||
# and it is what makes the sum-to-at-most-one constraint bite instead of being cosmetic.
|
||||
MIN_ITEM_MASS = 0.55
|
||||
MAX_ITEM_MASS = 0.90
|
||||
|
||||
# Held-out kills stop when every item has been seen; this caps the walk in case a table
|
||||
# with a 1/24576 tertiary draws a very long silence. Hitting it costs an item's evidence
|
||||
# rather than anything worse, and at twelve times the requested slice it is a 1e-8 event.
|
||||
HELD_OUT_CEILING = 12
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DropTable:
|
||||
"""One monster's table. `rates` is in DISPLAY order, deliberately shuffled: listing
|
||||
items by rarity would hand over the ordering the agent is being asked to recover."""
|
||||
|
||||
monster: str
|
||||
rates: dict[str, float]
|
||||
values: dict[str, int]
|
||||
|
||||
@property
|
||||
def items(self) -> list[str]:
|
||||
return list(self.rates)
|
||||
|
||||
@property
|
||||
def nothing(self) -> float:
|
||||
return 1.0 - sum(self.rates.values())
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Stream:
|
||||
"""One task: the table, and both slices as tallies.
|
||||
|
||||
Tallies rather than kill sequences because the tally is the sufficient statistic — the
|
||||
order kills came in carries no information about the table, so serialising 5,000 lines
|
||||
would cost tokens and teach nothing.
|
||||
"""
|
||||
|
||||
table: DropTable
|
||||
visible: dict[str, int]
|
||||
visible_kills: int
|
||||
held: dict[str, int]
|
||||
held_kills: int
|
||||
|
||||
@property
|
||||
def visible_nothing(self) -> int:
|
||||
return self.visible_kills - sum(self.visible.values())
|
||||
|
||||
@property
|
||||
def held_nothing(self) -> int:
|
||||
return self.held_kills - sum(self.held.values())
|
||||
|
||||
|
||||
def _names(rng: random.Random, count: int) -> list[str]:
|
||||
seen: list[str] = []
|
||||
while len(seen) < count:
|
||||
name = f"{rng.choice(ADJECTIVES)} {rng.choice(NOUNS)}"
|
||||
if name not in seen:
|
||||
seen.append(name)
|
||||
return seen
|
||||
|
||||
|
||||
def build_table(seed: int) -> DropTable:
|
||||
"""The table alone, from the seed. Stable across kill-stream redraws so that a task's
|
||||
identity is its monster, not whichever sample happened to be accepted.
|
||||
|
||||
`sample` rather than `choices` in both tiers: with replacement, two items could share a
|
||||
rung, the leftover-rung deduction would be unsound, and the tail would go back to being
|
||||
a constant nobody has to read the log for.
|
||||
"""
|
||||
rng = random.Random(seed)
|
||||
monster = f"{rng.choice(MONSTER_PREFIX)} {rng.choice(MONSTER_KIND)}"
|
||||
n_common = rng.choice(N_COMMON)
|
||||
n_ladder = rng.choice(N_LADDER)
|
||||
names = _names(rng, n_common + n_ladder)
|
||||
|
||||
# Rejection on total mass, with the tier COUNTS fixed outside the loop: redrawing the
|
||||
# shape as well would make the split's prior depend on how often that shape lands in
|
||||
# the mass window, and the reference's prior — which is P(n_common) * P(n_ladder) —
|
||||
# would then be subtly wrong on every task.
|
||||
while True:
|
||||
numerators = rng.sample(COMMON_NUMERATORS, n_common)
|
||||
denominators = rng.sample(LADDER_DENOMINATORS, n_ladder)
|
||||
rates = [n / COMMON_DENOMINATOR for n in numerators] + [1.0 / d for d in denominators]
|
||||
if MIN_ITEM_MASS <= sum(rates) <= MAX_ITEM_MASS:
|
||||
break
|
||||
order = list(zip(names, rates))
|
||||
rng.shuffle(order)
|
||||
table = dict(order)
|
||||
# Prices drawn AFTER the shuffle so the display order carries nothing, and from the
|
||||
# rate rather than the tier, so a common at 8/128 and a ladder item at 1/128 are priced
|
||||
# identically — the price is evidence about the RATE, never about which tier an item
|
||||
# came from.
|
||||
values = {
|
||||
name: max(1, round(VALUE_SCALE * (1.0 / rate) ** VALUE_EXPONENT
|
||||
* math.exp(VALUE_LOG_SD * rng.gauss(0.0, 1.0))))
|
||||
for name, rate in table.items()
|
||||
}
|
||||
return DropTable(monster=monster, rates=table, values=values)
|
||||
|
||||
|
||||
def _tally(rng: random.Random, table: DropTable, kills: int, until_covered: bool) -> tuple[dict[str, int], int]:
|
||||
"""Roll kills, counting drops. With `until_covered`, keep rolling past `kills` until
|
||||
every item has appeared — see the module docstring for why the graded slice owes the
|
||||
item list that."""
|
||||
items = table.items
|
||||
cumulative, running = [], 0.0
|
||||
for item in items:
|
||||
running += table.rates[item]
|
||||
cumulative.append(running)
|
||||
counts = {item: 0 for item in items}
|
||||
rolled, ceiling = 0, kills * HELD_OUT_CEILING
|
||||
while rolled < kills or (until_covered and not all(counts.values()) and rolled < ceiling):
|
||||
index = bisect.bisect(cumulative, rng.random())
|
||||
if index < len(items):
|
||||
counts[items[index]] += 1
|
||||
rolled += 1
|
||||
return counts, rolled
|
||||
|
||||
|
||||
def build_slices(seed: int, visible_kills: int, held_out_kills: int) -> Stream:
|
||||
"""The two slices of one task, from one seed.
|
||||
|
||||
The kill stream is redrawn — table held fixed — until the visible slice has a
|
||||
zero-count item, because a task where every item showed up does not pose the question
|
||||
this environment exists to ask. Rejection is on the visible slice only; the held-out
|
||||
continuation is whatever the accepted stream produced next.
|
||||
"""
|
||||
table = build_table(seed)
|
||||
for attempt in range(256):
|
||||
rng = random.Random((seed << 9) + attempt)
|
||||
visible, _ = _tally(rng, table, visible_kills, until_covered=False)
|
||||
if all(visible.values()):
|
||||
continue
|
||||
held, held_kills = _tally(rng, table, held_out_kills, until_covered=True)
|
||||
return Stream(table, visible, visible_kills, held, held_kills)
|
||||
raise RuntimeError(f"no stream with an unobserved item for seed {seed}")
|
||||
@@ -0,0 +1,205 @@
|
||||
"""drop-table-inference: recover a drop table from a kill log, graded on the next kills.
|
||||
|
||||
The agent gets a monster's item list — each item with a price and a drop count over a few
|
||||
hundred kills — and never gets the rates. It returns an estimate, and the estimate is scored
|
||||
against the continuation of that same kill stream, thirty times as long and never shown.
|
||||
|
||||
The reward is normalised against a REFERENCE STRATEGY rather than an absolute, and that is
|
||||
not a softening. Sampling error is irreducible: nobody recovers 1/24576 from 1,200 kills, so
|
||||
a reward measured against the true table has a ceiling no amount of reasoning reaches, which
|
||||
is the dead-component failure rule 3 exists to catch.
|
||||
|
||||
WHICH reference is the design, and the first version of this environment got it wrong. It
|
||||
used a closed form — the posterior mean under a Jeffreys prior — and a closed form is a
|
||||
function of (count, kills) alone, so every item that dropped zero times got the same number
|
||||
on every task. A reply that never opened the kill log scored 1.000 on the term built to
|
||||
catch frequency-copying, and frequency-copying plus that one memorised constant reached 90%
|
||||
of the ceiling. The reference here is the posterior over the generator's own hypothesis
|
||||
space: items assigned to distinct rungs of a published ladder, weighted by the multinomial
|
||||
likelihood of the counts AND by each item's price, filtered by the published mass window,
|
||||
and corrected for the fact that the log was selected to contain a silent item. Everything it
|
||||
uses is in the prompt. Nothing it uses is a constant.
|
||||
|
||||
The structure below is published deliberately. A reference the agent cannot compute is an
|
||||
unreachable ceiling wearing a hat, and rule 3 forbids those. What is not published is the
|
||||
answer — which item sits on which rung — and that is a joint inference over every item at
|
||||
once, because no two items share a rung and the leftovers are the only evidence about the
|
||||
ones the log never showed.
|
||||
|
||||
Three terms:
|
||||
|
||||
fit forward divergence ratio against the held-out continuation. Dominated by the
|
||||
common items, because forward KL weights every term by the truth's own mass.
|
||||
rare squared log-ratio on the items that dropped ZERO times, against the true rate,
|
||||
normalised by the reference's own error and multiplied by `restraint`.
|
||||
gate binary: valid, and as good as the reference on all three.
|
||||
|
||||
and the counterweight, `restraint`, which multiplies into `rare` rather than sitting beside
|
||||
it: the nats an estimate spends claiming that a silent item drops more often than the
|
||||
continuation showed. Forward KL rewards covering what you cannot see; restraint prices the
|
||||
covering. Measured, over a uniform hedge mixed into frequency-copying at weight a: fit rises
|
||||
0.136 -> 0.186 -> 0.198 as a goes 0 -> 0.0005 -> 0.002 while restraint falls 1.000 -> 0.996
|
||||
-> 0.443. There is no setting of a where both are at their best, which is what rule 2 asks
|
||||
for and what the first version did not have.
|
||||
|
||||
Probabilities must sum to at most 1; the remainder is the chance of no drop, and it is not
|
||||
small. An estimate that sums past 1 is not a distribution and scores zero.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pydantic import Field
|
||||
|
||||
import verifiers.v1 as vf
|
||||
|
||||
from drop_table_inference.estimate import measure, parse_estimate
|
||||
from drop_table_inference.table import (
|
||||
COMMON_DENOMINATOR,
|
||||
COMMON_NUMERATORS,
|
||||
LADDER_DENOMINATORS,
|
||||
MAX_ITEM_MASS,
|
||||
MIN_ITEM_MASS,
|
||||
N_COMMON,
|
||||
N_LADDER,
|
||||
VALUE_EXPONENT,
|
||||
VALUE_LOG_SD,
|
||||
VALUE_SCALE,
|
||||
build_slices,
|
||||
)
|
||||
|
||||
|
||||
def _english(counts: tuple[int, ...]) -> str:
|
||||
"""The tier counts as a person would write them: 3 or 4, and 4, 5 or 6. Spelled out
|
||||
rather than given as a range because the agent needs the exact set — the reference sums
|
||||
over the splits it allows, and a split it does not know about is one it cannot weigh."""
|
||||
return " or ".join(filter(None, (", ".join(str(n) for n in counts[:-1]), str(counts[-1]))))
|
||||
|
||||
|
||||
_COMMONS = ", ".join(f"{n}/{COMMON_DENOMINATOR}" for n in COMMON_NUMERATORS)
|
||||
_LADDER = ", ".join(f"1/{d}" for d in LADDER_DENOMINATORS)
|
||||
|
||||
SYSTEM = f"""You are estimating a monster's drop table from a kill log.
|
||||
|
||||
You are given every item on the table, its shop value in gp, and how many times it dropped.
|
||||
You are NOT given the rates. Return your estimate as one JSON object in a ```json code
|
||||
block, mapping each item name to its probability per kill:
|
||||
|
||||
```json
|
||||
{{"Ashen talon": 0.125, "Ivory sigil": "1/3072"}}
|
||||
```
|
||||
|
||||
A value may be a decimal or a fraction like "1/3072". Give at least four significant
|
||||
figures — your estimate is compared against a well-reasoned one and rounding throws that
|
||||
comparison away.
|
||||
|
||||
How these tables are built, exactly:
|
||||
|
||||
- {_english(N_COMMON)} COMMON items, each at one of {_COMMONS}, no two the same.
|
||||
- {_english(N_LADDER)} RARE items, each at one of {_LADDER}, no two the same.
|
||||
- The rates sum to between {MIN_ITEM_MASS:.2f} and {MAX_ITEM_MASS:.2f}. The remainder is
|
||||
the chance a kill drops nothing, and it is not small.
|
||||
- An item's value in gp is lognormal around {VALUE_SCALE:g} * (1/rate) ** {VALUE_EXPONENT:g},
|
||||
with standard deviation {VALUE_LOG_SD:g} in natural log. Rarer is worth more, with real
|
||||
scatter: the price ranks the items the counts cannot identify, it does not name them.
|
||||
- This log was chosen because at least one item dropped ZERO times in it. Every item
|
||||
listed does drop. A rate of 0 is the worst answer available for any of them.
|
||||
|
||||
Because no two items share a rate, the rates the observed items take are rates the silent
|
||||
ones cannot. That, the price, and the count are the whole of the evidence.
|
||||
|
||||
You are graded on the next several tens of thousands of kills from this same monster, which
|
||||
you have not seen — on how well your table predicts them, and separately on how close you
|
||||
got for the items this log never showed you."""
|
||||
|
||||
|
||||
class DropTableData(vf.TaskData):
|
||||
seed: int
|
||||
"""Rebuilds the table and both slices exactly; no rate is ever serialized here."""
|
||||
visible_kills: int
|
||||
held_out_kills: int
|
||||
|
||||
|
||||
class DropTableTask(vf.Task[DropTableData]):
|
||||
@vf.stop
|
||||
async def single_turn(self, trace: vf.Trace) -> bool:
|
||||
return trace.num_turns >= 1
|
||||
|
||||
@vf.metric
|
||||
async def scan(self, trace: vf.Trace) -> dict[str, float]:
|
||||
"""Rebuild the stream and score the estimate once; every reward reads this."""
|
||||
stream = build_slices(self.data.seed, self.data.visible_kills, self.data.held_out_kills)
|
||||
estimate = parse_estimate(trace.last_reply, stream.table.items)
|
||||
outcome = measure(stream, estimate)
|
||||
return {
|
||||
"fit": outcome.fit,
|
||||
"rare": outcome.rare,
|
||||
"clean": float(outcome.clean),
|
||||
"valid": float(outcome.valid),
|
||||
"restraint": outcome.restraint,
|
||||
"raw_fit": outcome.raw_fit,
|
||||
"raw_rare": outcome.raw_rare,
|
||||
"kl_estimate": outcome.kl_estimate,
|
||||
"kl_reference": outcome.kl_reference,
|
||||
"tail_estimate": outcome.tail_estimate,
|
||||
"tail_reference": outcome.tail_reference,
|
||||
"rare_items": float(outcome.rare_items),
|
||||
}
|
||||
|
||||
@vf.reward(weight=0.45)
|
||||
async def fit(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("fit", 0.0)
|
||||
|
||||
@vf.reward(weight=0.35)
|
||||
async def rare(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("rare", 0.0)
|
||||
|
||||
@vf.reward(weight=0.20)
|
||||
async def gate(self, trace: vf.Trace) -> float:
|
||||
return trace.metrics.get("clean", 0.0)
|
||||
|
||||
|
||||
class DropTableConfig(vf.TasksetConfig):
|
||||
num_tasks: int = Field(64, ge=1)
|
||||
visible_kills: int = Field(1_200, ge=50)
|
||||
"""Kills the agent reads. Long enough that the common tier is resolvable against the
|
||||
grid, short enough that the deep rungs show up zero times — which is the case the
|
||||
environment exists to pose."""
|
||||
held_out_kills: int = Field(40_000, ge=200)
|
||||
"""Kills it is graded on and never sees. Thirty times the visible slice, so the graded
|
||||
frequencies are the estimate's problem rather than the sample's, and long enough that a
|
||||
1/24576 item is expected to appear."""
|
||||
|
||||
|
||||
class DropTableTaskset(vf.Taskset[DropTableTask, DropTableConfig]):
|
||||
SEED_BASE = 20_000
|
||||
|
||||
def load(self) -> list[DropTableTask]:
|
||||
tasks = []
|
||||
for i in range(self.config.num_tasks):
|
||||
seed = self.SEED_BASE + i
|
||||
stream = build_slices(seed, self.config.visible_kills, self.config.held_out_kills)
|
||||
width = max(len(item) for item in stream.table.items) + 2
|
||||
rows = "\n".join(
|
||||
f" {item:<{width}}{stream.table.values[item]:>9}{stream.visible[item]:>8}"
|
||||
for item in stream.table.items
|
||||
)
|
||||
rows += f"\n {'(no drop)':<{width}}{'':>9}{stream.visible_nothing:>8}"
|
||||
tasks.append(
|
||||
DropTableTask(
|
||||
DropTableData(
|
||||
idx=i,
|
||||
name=f"{stream.table.monster.lower().replace(' ', '-')}-{seed}",
|
||||
prompt=(
|
||||
f"{stream.table.monster} — {stream.visible_kills} kills logged.\n\n"
|
||||
f" {'item':<{width}}{'value':>9}{'drops':>8}\n{rows}\n\n"
|
||||
f"Estimate the drop rate of every item."
|
||||
),
|
||||
system_prompt=SYSTEM,
|
||||
seed=seed,
|
||||
visible_kills=self.config.visible_kills,
|
||||
held_out_kills=self.config.held_out_kills,
|
||||
),
|
||||
self.config.task,
|
||||
)
|
||||
)
|
||||
return tasks
|
||||
@@ -0,0 +1,13 @@
|
||||
[project]
|
||||
name = "drop-table-inference"
|
||||
version = "0.1.0"
|
||||
description = "drop-table-inference — estimate a drop table from a kill log, graded on the kills you never saw."
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["verifiers"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["drop_table_inference"]
|
||||
+3488
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user