lumbridge-public/arena
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Three OSRS environments, and the first real model scores

Browse Source 
grand-exchange, bot-detection and drop-table-inference. All three produce
unbounded or irreducible-error scores, unlike the first four, so all three
normalise against a REFERENCE STRATEGY rather than an absolute -- which is what
makes 1.000 reachable rather than aspirational.

drop-table-inference is the clearest case. Scoring KL against the true drop
table would put the ceiling out of reach, because sampling error is irreducible:
ground truth scores 0.900 against the Bayesian posterior's 1.000, and that
ordering is correct. The best estimate available from 1,200 kills is not the
true table, and an environment that demands it is measuring luck.

grand-exchange needed the market to carry structure a model can actually infer,
or profit is noise and no oracle exists. Measured over 120 baskets: the
reference earns 38,821 gp and is profitable in 120/120, random orders lose 561,
and trading only the random-walk items -- which look like the widest-swinging
lines on the board -- loses 16,860.

bot-detection generates naive bots, cloaked bots that jitter on purpose, and
efficient humans who look bot-like on every naive statistic. Timing features are
drawn BEFORE the generator decides who is a bot, so every latency rule sits at
chance. Reference discriminator F1 1.000, random 0.430.

Each was built by one agent then attacked by two independent reviewers on
exploitability and soundness. They earned their keep: drop-table-inference's
first reward let a memorised constant score 0.901 without opening the kill log,
and its reference was 19% worse than a flat number. probe.py now carries three
named fences asserting those attacks stay dead.

First evaluation also lands. Nemotron 3.5 Lightning, thinking off, 32 rollouts
per environment: 0.570 down to 0.059, every gate at or near zero, nothing
solved and nothing unsolvable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
karti
2026-08-19 03:56:50 -07:00
co-authored by Claude Opus 5
parent 5a99eb86a5
commit 0defaed9c0
23 changed files with 13858 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
environments/grand_exchange/grand_exchange/__init__.py
+3
View File
@@ -0,0 +1,3 @@
from grand_exchange.taskset import ExchangeTaskset
__all__ = ["ExchangeTaskset"]
environments/grand_exchange/grand_exchange/book.py
+525
View File
@@ -0,0 +1,525 @@
"""Executing the agent's orders against the held-out window, and scoring the result.
Realised profit is unbounded above and has no theoretical optimum the data supports, so
scoring it against one would put the ceiling out of reach — which house rule 3 forbids.
Every ratio here is therefore taken against a REFERENCE STRATEGY (`reference_orders`)
that is computed from the visible half and nothing else, and executed through the same
engine as the agent's orders. That is what makes 1.000 reachable by construction, and it
is honest: the model is asked to match a strategy available to anything that can read the
prompt, not to beat hindsight it was never shown.
A reference in the denominator has its own failure mode, and the first cut of this file had
it. If the ceiling is the reference EXACTLY, then above the clip excess profit is worth
nothing while every deviation still costs, so the reward stops being a profit metric and
becomes an imitation score for a strategy whose constants are nowhere in the prompt. It was
measurable: a wider sell band earned sixteen percent more gp and scored 0.812, and one
wasted unit of capital with identical realised profit cost 0.146. Two changes answer it.
The reference's own constants are now the profit-maximising point of its family, swept on
seeds no task is built from. And the ceiling is a BAND — TARGET_SHARE of the reference —
so the whole plateau around it, and everything above it, scores 1.000.
Three numbers come out of a run:
profit_ratio clip(realised / (TARGET_SHARE x reference realised), 0, 1). No orders is
zero, and a round trip that does not clear the tax is negative and also
zero.
efficiency realised profit per coin LOCKED behind an offer, against the reference's
own. This is where the volume limit bites: a fat margin on an item
carrying three thousand gp a tick is an offer that sits there, and a
sitting offer is capital a better item did not get. It also catches the
plan that fills perfectly at prices that were never worth reaching, which
a plain fill rate scores as discipline.
clean cleared the bar on both. Binary, because a trading run either was worth
doing or was not.
`efficiency` is never a reward on its own — one tiny order that traded perfectly would earn
it in full for near-inaction, which is the free-points defect schema-migration shipped with.
It multiplies into profit instead, so it can only ever qualify profit that exists.
"""
from __future__ import annotations
import json
import math
import re
import statistics
from dataclasses import dataclass
from grand_exchange.market import (
DUMP_BASE,
DUMP_CAP,
DUMP_IMPACT,
FILL_SHARE,
TAX,
Item,
Market,
build_market,
)
MAX_ORDERS = 12
BUY_BAND = 0.05
SELL_BAND = 0.05
"""The band the reference buys under its anchor and sells over it.
These two numbers are the reward's denominator, so where they sit is not a taste question.
They are the profit-maximising point of the reference's own family, found by sweeping
(buy band, sell band, crossings threshold, purse cap, size multiplier) over 160 baskets
drawn from SEED 400,000 ONWARD — a range no task is ever built from — and they are within
three percent of the sweep's best cell over the whole plateau.
The first cut used a sell band of 0.01, and that was the defect a reviewer found: a strategy
with a wider sell band earned sixteen percent more gp than the reference and scored 0.812,
because every ratio here divides by the reference and clips at one. When the denominator is
a strategy that leaves money on the table, the reward's argmax is the denominator's
hyperparameters rather than the profit. Tuning the reference to its own family's optimum is
half the fix; TARGET_SHARE below is the other half."""
TARGET_SHARE = 0.90
"""What counts as a full score, as a share of the reference's realised profit.
A ratio that divides by the reference EXACTLY makes the ceiling a single point, and a point
ceiling turns the reward into an imitation score: excess profit is worth nothing above the
clip while any deviation is punished, so the gradient near the top points at replicating a
strategy the prompt does not contain rather than at making money. Measured on the first cut,
one wasted unit of capital with byte-identical realised profit cost 0.146 of total reward,
and a one-gp change to every buy limit that EARNED 463 gp a basket more cost 0.195.
Normalising against nine tenths of the reference makes the ceiling a BAND. Everything from
"ten percent short of the reference" upward scores 1.000, so the whole plateau around the
reference — and everything above it — is the argmax, and a rounding difference costs
nothing. The reference still scores exactly 1.000, so house rule 3 is unchanged."""
MIN_CROSSINGS = 0.25
"""Below this share of ticks crossing the mean, the reference will not trade the item at
all. Over 4,000 baskets a reverting series recrosses its own mean 0.362 of the time and a
random walk 0.113, and the threshold sits in the gap: it lets 4.2% of walks through and
turns away 4.1% of reverters. This one line is the difference between a strategy and a
superstition — without it the reference buys the widest-swinging line in the basket, which
is exactly the line that has no anchor to revert to."""
MAX_ITEM_SHARE = 0.40
"""No more than this share of the purse behind one name. The anchor is an ESTIMATE, and one
bad estimate carrying the whole purse is the only way this strategy loses money over a
window. Spread over the 3.4 names it trades on average it profits in 3,976 of 4,000
baskets, and `viable_market` refuses the other twenty-four."""
VIABILITY_TRIES = 64
@dataclass(frozen=True)
class Order:
item: str
quantity: int
buy: int
sell: int
@dataclass
class Fills:
"""What one plan actually did in the window."""
realised: float = 0.0
"""Closing capital minus starting capital: the only number that pays."""
paper: float = 0.0
"""What the orders CLAIM, filled in full at the limit prices. Not part of any reward —
it exists so `arith_ok` can ask whether the agent's stated expected profit matches the
sum its own orders imply."""
bought: int = 0
sold: int = 0
planned: int = 0
spent: float = 0.0
dumped: int = 0
orders: int = 0
committed: float = 0.0
"""Coins actually locked behind offers. Capital left idle earns nothing, which is the
only reason allocation is a decision."""
offered: int = 0
"""Units the purse could fund, against `planned` units asked for."""
@property
def roc(self) -> float:
"""Realised profit per coin locked behind an offer.
This is the counterweight the profit term is qualified by, and it replaced a plain
fill rate. A fill rate only asks whether an offer was reachable; it says nothing
about whether reaching it was worth doing, so a plan that bought badly at prices it
was always going to reach scored as disciplined as one that bought well. Return on
the coins actually committed asks both questions at once, and it cannot be won by
committing nothing: it multiplies into profit, which needs the coins.
"""
if self.committed <= 0.0:
return 0.0
return self.realised / self.committed
@property
def conversion(self) -> float:
"""Of the coins locked behind offers, the share that actually bought stock.
The first version of this was realised profit over CLAIMED profit, and it was
gameable in one line: set the sell limit a hair over the buy limit and the claim
goes to nearly nothing, so the ratio goes to nearly anything. Coins converted
cannot be talked down — the only way to raise it is to offer a quantity the volume
supports at a price the market reaches, which is the lesson the number is for.
"""
if self.committed <= 0.0:
return 0.0
return min(1.0, self.spent / self.committed)
def paper_profit(orders: list[Order]) -> float:
"""The profit the orders assert, at their own limit prices, net of tax on the sale.
Deliberately the naive multiplication — it is the sum the agent is claiming, and
`arith_ok` checks whether the agent can do it."""
return sum(o.quantity * (o.sell * (1.0 - TAX) - o.buy) for o in orders)
def execute(market: Market, orders: list[Order]) -> Fills:
"""Walk the held-out ticks once, filling what the purse and the book support.
Rules, all quoted to the agent:
- COINS ARE LOCKED WHEN THE OFFER IS PLACED, quantity times buy price, in the order
the orders were listed. This is the exchange's own behaviour and it is what makes
the task an allocation: an offer that never fills has still spent the capital a
better offer needed, so ordering the whole basket at its buy limit is not free.
It is also why a buy limit of a billion buys nothing — nobody can fund that offer.
- a buy fills at the tick's price when it is at or under the limit; a sell fills at
the tick's price when it is at or over. You are never filled at your own limit
when the market is better than it, because that is not how a limit works.
- at most FILL_SHARE of a tick's volume per side, per item, and only on the ticks the
offer was eligible on — flow you were not in the market for is not yours to bank.
- stock bought this tick cannot be sold this tick. Without that, a buy limit above a
sell limit is a free round trip on a single price.
- sale proceeds land in the purse but fund nothing: every offer was placed up front.
- stock still held at the close is forced out at DUMP_BASE under the last price plus
DUMP_IMPACT for every tick's worth of the item's median volume being pushed through.
Depth is taken from the window the stock is actually being sold into, which is the
window the agent estimated from the visible median.
"""
fills = Fills(orders=len(orders), planned=sum(o.quantity for o in orders))
fills.paper = paper_profit(orders)
# Placement: fund each offer in submission order out of one purse.
purse = float(market.capital)
live: list[tuple[Order, Item, int]] = []
for order in orders:
item = market.item(order.item)
if item is None or order.buy <= 0:
continue
qty = min(order.quantity, item.buy_limit, int(purse // order.buy))
if qty <= 0:
continue
purse -= qty * order.buy
live.append((order, item, qty))
fills.committed = market.capital - purse
fills.offered = sum(q for _, _, q in live)
remaining = [q for _, _, q in live]
available = [0] * len(live)
pending = [0] * len(live)
# Capacity carries between eligible ticks instead of being truncated at each one. An
# item that trades two units a tick would otherwise be untradeable rather than thin,
# because a quarter of two is zero every time — and thin is what this is about.
buy_room = [0.0] * len(live)
sell_room = [0.0] * len(live)
proceeds = 0.0
for t in range(market.visible, market.visible + market.held_out):
for i in range(len(live)):
available[i] += pending[i]
pending[i] = 0
for i, (order, item, _) in enumerate(live):
price = item.prices[t]
flow = FILL_SHARE * item.volumes[t]
# Room accrues only on the ticks the order was actually eligible on. Accruing it
# every tick banks the flow of ticks the price never reached, which would let an
# offer fill far past the volume that was ever available to it — and the volume
# limit is the whole reason allocation is a decision here.
if price <= order.buy:
buy_room[i] += flow
if price >= order.sell:
sell_room[i] += flow
if price <= order.buy and remaining[i] > 0:
qty = min(remaining[i], int(buy_room[i]))
if qty > 0:
buy_room[i] -= qty
remaining[i] -= qty
pending[i] += qty
fills.spent += qty * price
fills.bought += qty
if price >= order.sell and available[i] > 0:
qty = min(available[i], int(sell_room[i]))
if qty > 0:
sell_room[i] -= qty
proceeds += qty * price * (1.0 - TAX)
available[i] -= qty
fills.sold += qty
for i, (_, item, _) in enumerate(live):
left = available[i] + pending[i]
if left:
depth = statistics.median(item.volumes[market.visible:])
haircut = min(DUMP_CAP, DUMP_BASE + DUMP_IMPACT * (left / max(1.0, depth)))
fills.dumped += left
proceeds += left * item.prices[-1] * (1.0 - haircut) * (1.0 - TAX)
# Coins locked behind an offer that never filled come back when the window closes, so
# the only thing a wasted offer costs is the profit the capital did not make. That is
# the right price for it: an opportunity cost, not a fine.
fills.realised = proceeds - fills.spent
return fills
def crossings(prices: list[int]) -> float:
"""Share of consecutive ticks that straddle the series' own mean.
The whole discrimination, in one countable number, so it is available to anything that
can read the prompt — no variance ratio, no regression, just how often the line cuts
its own average.
"""
if len(prices) < 2:
return 0.0
anchor = statistics.fmean(prices)
above = [p > anchor for p in prices]
return sum(1 for i in range(1, len(above)) if above[i] != above[i - 1]) / (len(above) - 1)
def reference_orders(market: Market) -> list[Order]:
"""The strategy the reward is normalised against, computed from the visible half only.
Per item: anchor on the mean of the visible prices, buy a band under it, sell a band
over it. Then three judgements, and each of them is a way the reward discriminates:
does it revert `crossings` over MIN_CROSSINGS, or the item is skipped. The decoy
swings widest and is worth nothing.
what it pays the average visible price BELOW the buy limit against the average
ABOVE the sell limit — because a limit fills at the market, not at
the limit, so a wide reverting item pays far more than its band.
Ranking by the band alone ranks every item identically.
what it can hold a quarter of a typical tick's volume, over the ticks that touched
the buy limit. Capital committed beyond that is capital locked
behind an offer that will not fill.
Best return on capital first, until the purse is gone. Nothing here reads a held-out
tick; every input is a column the agent was shown.
"""
plans = []
for item in market.items:
prices = item.visible_prices(market.visible)
volumes = item.visible_volumes(market.visible)
if crossings(prices) < MIN_CROSSINGS:
continue
anchor = statistics.fmean(prices)
buy = max(1, round(anchor * (1.0 - BUY_BAND)))
sell = max(buy + 1, round(anchor * (1.0 + SELL_BAND)))
lows = [p for p in prices if p <= buy]
highs = [p for p in prices if p >= sell]
if not lows or not highs:
continue
entry, exit_ = statistics.fmean(lows), statistics.fmean(highs)
expected = (exit_ * (1.0 - TAX) - entry) / entry
if expected <= 0:
continue
reachable = int(
FILL_SHARE * statistics.median(volumes) * (len(lows) / len(prices)) * market.held_out
)
qty = min(item.buy_limit, reachable, int(MAX_ITEM_SHARE * market.capital // buy))
if qty <= 0:
continue
plans.append((expected, item.name, qty, buy, sell))
plans.sort(key=lambda p: -p[0])
orders, purse = [], float(market.capital)
for _, name, qty, buy, sell in plans:
qty = min(qty, int(purse // buy))
if qty <= 0:
continue
purse -= qty * buy
orders.append(Order(item=name, quantity=qty, buy=buy, sell=sell))
return orders
def viable_market(seed: int, num_items: int, visible: int, held_out: int) -> Market:
"""The next basket from `seed` onward in which the reference strategy makes money.
Every ratio in the reward divides by the reference's realised profit, so a basket where
the reference loses has no reachable ceiling and would quietly break house rule 3 for
that task — the oracle would score below 1.000 and nothing would say why. About one
basket in two thousand is like that, from an anchor the visible half happened to
mis-estimate. Skipping it is a guard, not a crutch, and the seed that was used travels
on the Market so scoring rebuilds exactly the basket that was shown.
"""
for offset in range(VIABILITY_TRIES):
market = build_market(seed + offset, num_items, visible, held_out)
if execute(market, reference_orders(market)).realised > 0:
return market
return build_market(seed, num_items, visible, held_out)
@dataclass
class Outcome:
"""One run, beside the reference run it is normalised against."""
fills: Fills
reference: Fills
stated: float | None
dropped: int = 0
"""Orders that named nothing on the board, or asked for a quantity or price of zero."""
@property
def target(self) -> float:
"""The gp a full score is worth in this basket. See TARGET_SHARE."""
return TARGET_SHARE * self.reference.realised
@property
def profit_ratio(self) -> float:
if self.target <= 0.0:
return 0.0
return min(1.0, max(0.0, self.fills.realised / self.target))
@property
def efficiency(self) -> float:
"""Return on committed capital, against the reference's return on its own.
Not discounted by TARGET_SHARE: it is already a rate rather than a total, so a
strategy that trades well on a small book is not penalised for being small, and
giving it a band as well would hand out the term for free."""
if self.reference.roc <= 0.0:
return 0.0
return min(1.0, max(0.0, self.fills.roc / self.reference.roc))
@property
def conversion_ratio(self) -> float:
"""Of the coins locked, the share that bought stock, against the reference's share.
Recorded, never rewarded — `efficiency` is what qualifies profit now. It stays in
the trace because it separates the two ways `efficiency` falls: offers that never
filled, and offers that filled at prices not worth reaching."""
if self.reference.conversion <= 0.0:
return 0.0
return min(1.0, self.fills.conversion / self.reference.conversion)
@property
def discipline(self) -> float:
"""Profit, qualified by what each committed coin earned. A product, not a term
beside it: a term would pay in full for one tiny order that traded perfectly, which
is inaction with a receipt — the free-points defect schema-migration shipped with."""
return self.profit_ratio * self.efficiency
@property
def clean(self) -> bool:
"""A bar, not a knife-edge.
It used to require matching the reference on both ratios to within 1e-9, which is a
demand for replication rather than for a good run: a reference plan with one extra
one-unit order that never fills has identical realised profit and lost 0.146 of
total reward. The bar is now TARGET_SHARE of the reference on the money AND on the
return that money made, which is a run that was worth doing however it got there.
"""
return (
self.fills.realised >= TARGET_SHARE * self.reference.realised
and self.fills.roc >= TARGET_SHARE * self.reference.roc
)
@property
def arith_ok(self) -> bool:
"""Did the agent's stated expected profit match what its own ACCEPTED orders imply.
Not a reward — a probe into the trace. The target model gets simple arithmetic wrong
with thinking off, and without this line a bad multiplication and a bad strategy are
the same low number and nothing in the trace tells them apart. Measured against the
orders that survived parsing, so claiming profit from an order for an item that is
not on the board reads as the arithmetic error it is.
"""
if self.stated is None:
return False
return abs(self.stated - self.fills.paper) <= max(50.0, 0.02 * abs(self.fills.paper))
@property
def arith_error(self) -> float:
if self.stated is None or self.fills.paper == 0.0:
return 0.0
return abs(self.stated - self.fills.paper) / abs(self.fills.paper)
def measure(market: Market, orders: list[Order], stated: float | None) -> Outcome:
"""Run the agent's plan and the reference plan through the same engine."""
clean, seen, dropped = [], set(), 0
for order in orders[:MAX_ORDERS]:
item = market.item(order.item)
# One order per item: holdings pool per item, so two orders on one name would make
# "which sell limit does this unit belong to" a question the engine has to invent an
# answer to. The first one submitted is the one that counts.
if item is None or order.quantity <= 0 or order.buy <= 0 or order.sell <= 0:
dropped += 1
continue
if item.name in seen:
dropped += 1
continue
seen.add(item.name)
clean.append(Order(item.name, order.quantity, order.buy, order.sell))
return Outcome(
fills=execute(market, clean),
reference=execute(market, reference_orders(market)),
stated=stated,
dropped=dropped,
)
_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL)
def parse_orders(reply: str) -> tuple[list[Order], float | None]:
"""The last JSON block in the reply: an object with "orders" and, optionally,
"expected_profit". A bare array of orders is accepted too — a model that answers the
question and skips the arithmetic has traded, and should be graded on the trade.
A reply that parses to nothing is an empty plan, not an error: it scores what doing
nothing scores. Raising here would turn a formatting slip into a crashed rollout — and
the reward runs inside the metric, so a raise takes the whole rollout with it rather
than scoring zero.
Which is why the except clauses below are wider than they look like they need to be.
Python's `json.loads` is not strict JSON: it accepts the bare literals Infinity,
-Infinity and NaN, and it overflows 1e309 to inf rather than refusing it. RecursionError
is not a ValueError, so twenty thousand nested arrays crashed the decoder; OverflowError
is not a ValueError either, so `int(float("inf"))` crashed the row loop. Both were live
on the first cut and both were reachable from a reply a model can actually emit.
"""
blocks = _BLOCK.findall(reply or "")
raw = blocks[-1] if blocks else (reply or "")
try:
parsed = json.loads(raw.strip())
except (ValueError, RecursionError):
return [], None
stated: float | None = None
if isinstance(parsed, dict):
rows = parsed.get("orders")
value = parsed.get("expected_profit")
# Finite only. `Infinity` and `NaN` parse, and either one propagates through
# `arith_error` into the trace as a non-finite metric, which is a corrupted training
# signal rather than a bad answer. An unusable claim is no claim.
if isinstance(value, (int, float)) and not isinstance(value, bool):
try:
stated = float(value) if math.isfinite(value) else None
except OverflowError:
stated = None
else:
rows = parsed
if not isinstance(rows, list):
return [], stated
orders = []
for row in rows:
if not isinstance(row, dict) or not isinstance(row.get("item"), str):
continue
try:
orders.append(
Order(
item=row["item"],
quantity=int(row.get("quantity", 0)),
buy=int(row.get("buy", 0)),
sell=int(row.get("sell", 0)),
)
)
except (TypeError, ValueError, OverflowError):
continue
return orders, stated
environments/grand_exchange/grand_exchange/market.py
+244
View File
@@ -0,0 +1,244 @@
"""The market generator: price and volume streams with structure a reader can find.
If prices were a random walk this environment would be worthless and would still look
fine — expected profit is zero for every strategy, so the reward is noise, no oracle
exists, and the numbers coming out of it would be plausible and meaningless. So the
structure is put in deliberately and is the whole design:
price_t = fundamental_t * (1 + x_t)
`fundamental_t` drifts by FUND_DRIFT a tick, worth about one percent over a whole stream.
`x_t` is an AR(1) around zero with PHI decay and a stationary spread of NOISE_SD — nine
percent on the deep items, fifteen or sixteen on the thin ones. Noise dominates drift by an
order of magnitude, which is what makes the mean of the visible prices a usable estimate of
the fundamental, and "buy under the estimate, sell over it" a real strategy rather than a
superstition. `book.reference_orders` is that strategy; `probe.py` measures how far it beats
trading at random, and refuses to pass if the margin is thin.
The visible and held-out windows are two halves of ONE stream from ONE seed: the graded
ticks are the next ticks the generator would have produced, not a differently-seeded
population that could have moved somewhere the visible half gave no warning of.
Two things are here to punish reading the price column alone. One or two items per basket do
not revert at all (see DRIFT_STEP and WALKS) and swing widest of everything on screen. And
liquidity — price times volume, gp a tick — is uncorrelated with price, so the fattest
visible margins sit on the items that can absorb the least of the purse.
Item names are invented. Formulas and market mechanics are facts about a kind of game;
item tables are somebody's copyrighted content, and none of it is here.
"""
from __future__ import annotations
import math
import random
from dataclasses import dataclass
# --- the rules of the exchange, quoted to the agent verbatim in the prompt --------------
TAX = 0.01
"""Charged on every sale, the GE's own. It is the counterweight that kills thin flips:
a round trip has to clear it twice over before it is worth doing, so "trade everything"
is a losing strategy rather than a neutral one."""
FILL_SHARE = 0.25
"""Share of a tick's volume one participant can take, and it accrues only on the ticks an
offer was actually eligible on. Without it a fat margin on an item carrying three thousand
gp a tick is worth as much as one carrying a hundred thousand, and the environment stops
measuring allocation."""
DUMP_BASE = 0.03
DUMP_IMPACT = 0.05
DUMP_CAP = 0.40
"""Stock still held when the window closes is forced out at the last price, minus a haircut
of DUMP_BASE plus DUMP_IMPACT for every full tick's worth of the item's median volume that
has to be pushed through, capped at DUMP_CAP.
A FLAT haircut was the first cut of this and it was the wrong shape. Under a flat five
percent, the cost of holding stock at the close was the same whether the leftovers were
twenty units of an item that trades thirty a tick or four thousand units of one that trades
six hundred — so oversizing was only ever punished through the purse, and the purse punishes
it as a cliff: the first over-large offer eats the whole budget and everything after it is
never placed. A cliff is not a gradient. Scaling the haircut by position-over-depth prices
the thing that is actually true — forcing size out costs you in proportion to how much of
the book you are pushing through — and it makes reading the volume column pay smoothly,
which is what the column is here to teach.
It also prices a bad anchor. A buy limit set too high fills fast and leaves the sell limit
out of reach, so the position that a sloppy anchor builds is exactly the position that has
to be dumped, and now it is dumped at a price that scales with its size."""
STARTING_CAPITAL = 250_000
WALKS = (1, 2)
"""How many of the basket's items do not mean-revert, drawn uniformly from this range.
It used to be exactly one, and a fixed count is a free prior: "drop the single widest line
on the board" scores what computing `book.crossings` scores, without computing anything.
It is the same defect `bot_detection` ships an assertion against — a class balance the model
can count on is a class balance it will use instead of the discriminator. With the count
unknown the shape statistic is the only thing that answers the question, and a basket can
punish both over- and under-rejection."""
SEED_BASE = 60_000
"""Where task seeds start. Lives here rather than on the taskset so `probe.py`, which
cannot import the taskset without `verifiers`, grades the baskets a run would actually
serve rather than a different set that happens to share a generator."""
# --- stream parameters -----------------------------------------------------------------
PHI = 0.45
"""AR(1) decay of the mispricing. Half-life under a tick, so a visible window holds many
independent draws around the fundamental — which is what makes the mean of it an estimate
rather than a guess — and a held-out window holds many excursions, so the reward is not one
lucky draw."""
FUND_DRIFT = 0.0008
"""Per-tick drift of the fundamental. Small on purpose: the fundamental has to be
ESTIMABLE from the visible half or there is nothing to learn."""
VOLUME_SD = 0.35
DRIFT_STEP = 0.055
"""Per-tick step of the items that do not mean-revert at all — their price is a pure random
walk, so the fundamental IS wherever it last was.
This is the trap the whole environment is built around, put inside the task instead of
left as a hazard the designer has to avoid. A random walk has no anchor, so buying under
its moving average is not a discount, it is a coin flip that pays the tax and the dump
slippage every time. On screen it is the widest-swinging line in the basket and therefore
the most attractive one, because amplitude is what a careless reader ranks by. The two can
only be told apart by SHAPE: a reverting series crosses its own mean constantly, a walk
wanders on one side of it for a dozen ticks at a time. `book.crossings` is that statistic
and the reference strategy will not trade an item that fails it."""
# (names, base price range, base volume range, buy limit, mispricing spread)
#
# What separates these is LIQUIDITY IN GP PER TICK — base price times base volume — and it
# is deliberately uncorrelated with the price. That is the allocation problem: the purse is
# 250,000 gp and a quarter of the flow over thirty ticks is what any one offer can absorb,
# so a deep tier can take a third of the purse and a thin one can take a twentieth of it no
# matter how good the margin looks. An earlier cut of this file made the expensive items the
# thin ones, which sounds right and is not: eight units a tick of a 46,000 gp item is 368,000
# gp of flow, the deepest thing on the board. Thin means small in coins, not small in units.
#
# The two thin tiers also carry the widest mispricing spread, so they show the fattest margin
# and can absorb the least. That is the trap, and it is the same trap either way an agent
# falls into it: ignore the volume column and either the offers sit unfilled or the purse
# sits idle.
TIERS = [
# deep, ~40k-130k gp a tick: this is where the purse actually goes
(["Thornroot poultice", "Chipped bone charm", "Bogwater draught", "Coarse fletching feather"],
(90, 170), (400, 800), 5000, 0.09),
(["Emberglass shard", "Stormrune tablet", "Marrowsteel nail", "Pale grimoire page"],
(900, 1700), (40, 90), 400, 0.09),
# thin, ~3k-15k gp a tick, and the widest swings on the board
(["Gilded harpoon head", "Cinderweave cloak", "Wyrmbone talisman", "Frostbitten ledger"],
(200, 420), (12, 30), 800, 0.15),
(["Duskforged sigil", "Heart of the sunken cairn", "Voidglass lens", "Tideworn crown"],
(6000, 14000), (1.2, 3.0), 40, 0.16),
]
@dataclass(frozen=True)
class Item:
name: str
reverting: bool
"""Whether this item has an anchor at all. Never shown to the agent — it is here so the
probe can assert that the trap is a trap, and that the reference avoids it for a reason
rather than by luck."""
buy_limit: int
"""Units per item per window, the GE's own limit. With finite capital it is what turns
the task into an allocation problem instead of a single pick."""
prices: list[int]
volumes: list[int]
def visible_prices(self, visible: int) -> list[int]:
return self.prices[:visible]
def visible_volumes(self, visible: int) -> list[int]:
return self.volumes[:visible]
@dataclass(frozen=True)
class Market:
seed: int
"""Carried on the basket so the taskset can store the seed it actually used. Baskets are
skipped when the reference strategy is not profitable in them (see `book.viable_market`),
so the seed a task was built from is not always the one it was asked for."""
items: list[Item]
visible: int
held_out: int
capital: int
def item(self, name: str) -> Item | None:
wanted = name.strip().casefold()
for item in self.items:
if item.name.casefold() == wanted:
return item
return None
def _volumes(rng: random.Random, volume: float, ticks: int) -> list[int]:
return [max(1, round(volume * math.exp(rng.gauss(0.0, VOLUME_SD)))) for _ in range(ticks)]
def _walk(rng: random.Random, base: float, ticks: int) -> list[int]:
"""The decoy: no anchor, no reversion, just a wide random walk. Whatever a moving
average says about where this price belongs is a statement about the past only."""
price = base
out = []
for _ in range(ticks):
price *= 1.0 + rng.gauss(0.0, DRIFT_STEP)
out.append(max(1, round(price)))
return out
def _stream(rng: random.Random, base: float, volume: float, noise_sd: float, ticks: int
) -> tuple[list[int], list[int]]:
"""One item's price and volume history, visible and held-out ticks together."""
# eps is scaled so the AR(1) settles at exactly noise_sd rather than drifting toward it
# over the first few ticks; the visible half would otherwise be quieter than the graded
# half and every anchor estimated from it would be too tight.
eps_sd = noise_sd * math.sqrt(1.0 - PHI * PHI)
x = rng.gauss(0.0, noise_sd)
fundamental = base
prices = []
for _ in range(ticks):
fundamental *= 1.0 + rng.gauss(0.0, FUND_DRIFT)
x = PHI * x + rng.gauss(0.0, eps_sd)
prices.append(max(1, round(fundamental * (1.0 + x))))
return prices, _volumes(rng, volume, ticks)
def build_market(seed: int, num_items: int, visible: int, held_out: int) -> Market:
"""One basket, from one seed.
Every tier is represented before any tier repeats, so a thin item and a deep one are
always both on the table: the allocation choice is the task, and a basket that happened
to be all-deep or all-thin would not pose it.
"""
rng = random.Random(seed)
ticks = visible + held_out
order = list(range(len(TIERS)))
rng.shuffle(order)
picks = [order[i % len(order)] for i in range(num_items)]
# Which slots are walks, and how many, are both drawn here. The tier is drawn
# independently of the walk flag, so a basket where the walk was always the cheap item —
# solvable by reading the price column and never the shape — cannot arise.
decoys = set(rng.sample(range(num_items), rng.randint(*WALKS)))
used: set[str] = set()
items = []
for slot, tier_idx in enumerate(picks):
names, (lo, hi), (vlo, vhi), limit, noise_sd = TIERS[tier_idx]
choices = [n for n in names if n not in used] or names
name = rng.choice(choices)
used.add(name)
base, volume = rng.uniform(lo, hi), rng.uniform(vlo, vhi)
if slot in decoys:
prices, volumes = _walk(rng, base, ticks), _volumes(rng, volume, ticks)
else:
prices, volumes = _stream(rng, base, volume, noise_sd, ticks)
items.append(Item(name=name, reverting=slot not in decoys, buy_limit=limit,
prices=prices, volumes=volumes))
items.sort(key=lambda i: i.name)
return Market(seed=seed, items=items, visible=visible, held_out=held_out,
capital=STARTING_CAPITAL)
environments/grand_exchange/grand_exchange/taskset.py
+230
View File
@@ -0,0 +1,230 @@
"""grand-exchange: place orders that are executed against a window you have not seen.
The agent reads 56 ticks of price and volume for five items and submits limit orders. The
orders are then run against the NEXT 30 ticks of the same streams — the ones the generator
would have produced next — against that window's actual volume, with a 1% sale tax, a
per-item buy limit and one finite purse. Realised profit is what pays.
Four judgements separate a good plan from a plan, and every one of them is available from
the columns on screen:
the anchor the mean of 56 visible ticks estimates the fundamental about four times
more tightly than the last tick does, and a limit set off a bad anchor
fills fast into stock the sell limit never reaches.
the trap one or two items per basket are a random walk, not a reverting series.
They swing widest, which is what a careless reader ranks by, and they have
no anchor to revert to. `book.crossings` is the only thing that tells them
apart, and the COUNT is not fixed, so "drop the widest one" is not a
substitute for computing it.
the size a quarter of a tick's volume, over the ticks that will reach your limit —
offers beyond that are coins locked behind something that cannot fill, and
stock beyond that is stock the close-out haircut charges you for.
the allocation finite purse, funded in submission order, so the best return has to go
first and the rest has to fit.
Realised profit has no ceiling and no optimum the data supports, so scoring it against one
would put 1.000 out of reach. Every ratio is taken against a REFERENCE STRATEGY instead,
which makes those four judgements and nothing else, from the visible half only, and runs
through the same execution engine. Its constants are the profit-maximising point of its own
family, swept on seeds no task is built from — a denominator that leaves money on the table
makes the reward an imitation score rather than a profit metric — and the ceiling is nine
tenths of it, so the plateau around it and everything above it all score 1.000.
`probe.py` measures how far the reference beats trading at random, and refuses to pass if
that margin closes, because a reward normalised against a reference that is no better than
chance is noise wearing a number. It also refuses to pass if a plan that never estimates the
anchor and never applies the trap filter scores as well as one that does.
Three rewards:
profit clip(realised / (0.90 × reference realised), 0, 1). No orders is exactly zero,
and so is any round trip that fails to clear the tax.
discipline profit × realised-per-coin-committed, relative to the reference's. A PRODUCT:
as a term beside profit it would pay in full for one tiny order that traded
perfectly, which is inaction with a receipt.
gate cleared both bars. Binary — a trading run was worth doing or it was not. A
bar, not a match: demanding the reference's exact result made a wasted unit of
capital cost 0.146 with byte-identical profit, which is a gradient pointing at
replication instead of at money.
`arith_ok` is recorded and never rewarded: whether the agent's stated expected profit
matches what its own orders imply. The target model gets simple arithmetic wrong with
thinking off, and without this line in the trace a bad multiplication and a bad strategy
are the same low score.
"""
from __future__ import annotations
from typing import ClassVar
from pydantic import Field
import verifiers.v1 as vf
from grand_exchange.book import MAX_ORDERS, TARGET_SHARE, measure, parse_orders, viable_market
from grand_exchange.market import (
DUMP_BASE,
DUMP_CAP,
DUMP_IMPACT,
FILL_SHARE,
SEED_BASE,
STARTING_CAPITAL,
TAX,
build_market,
)
SYSTEM = f"""You are trading on the Grand Exchange.
You will be shown recent price and volume history for a basket of items. You submit limit
orders. They are executed tick by tick against a FUTURE window you have not seen, drawn
from the same streams.
The rules of the exchange:
- A buy fills at the tick's price when that price is at or below your buy limit. A sell
fills at the tick's price when that price is at or above your sell limit.
- PLACING an offer locks quantity × buy price out of your purse straight away, whether
or not it ever fills. Offers are funded in the order you list them; once the purse is
gone the rest are not placed at all. You start with {STARTING_CAPITAL:,} gp and that is
the entire budget — sale proceeds return to you but fund no further offers.
- You can take at most {FILL_SHARE:.0%} of a tick's traded volume, per item, per side, and only
on ticks where your limit was actually reached.
- Each item has a buy limit: the most units of it you may buy in the whole window.
- {TAX:.0%} tax is charged on every sale.
- Stock bought on a tick cannot be sold on the same tick.
- Stock you still hold when the window closes is forced out at the last price minus {DUMP_BASE:.0%},
plus a further {DUMP_IMPACT:.0%} for every full tick's worth of that item's median volume you have
to push through, capped at {DUMP_CAP:.0%}; then taxed. Buying what you cannot sell is a loss,
not a hold, and it is a bigger loss the more of it there is.
Return ONE ```json code block containing an object with:
"expected_profit": the profit you expect your orders to make, in gp, as a number
"orders": a list of at most {MAX_ORDERS} objects, each with "item" (exactly as named above),
"quantity", "buy" and "sell"
At most one order per item; later duplicates are discarded. Example shape:
```json
{{"expected_profit": 12000,
"orders": [{{"item": "Bogwater draught", "quantity": 400, "buy": 118, "sell": 129}}]}}
```"""
class ExchangeData(vf.TaskData):
seed: int
"""Rebuilds both windows exactly. The held-out ticks are never serialized here — they
are regenerated at scoring time from this integer, so the task data cannot leak them."""
num_items: int
visible: int
held_out: int
class ExchangeTask(vf.Task[ExchangeData]):
@vf.stop
async def single_turn(self, trace: vf.Trace) -> bool:
return trace.num_turns >= 1
@vf.metric
async def scan(self, trace: vf.Trace) -> dict[str, float]:
"""Execute the plan and the reference once; every reward reads this. Recomputing
per reward is how two rewards end up disagreeing about the same episode."""
market = build_market(
self.data.seed, self.data.num_items, self.data.visible, self.data.held_out
)
orders, stated = parse_orders(trace.last_reply)
outcome = measure(market, orders, stated)
return {
"profit_ratio": outcome.profit_ratio,
"efficiency": outcome.efficiency,
"discipline": outcome.discipline,
"clean": float(outcome.clean),
"realised": outcome.fills.realised,
"reference_realised": outcome.reference.realised,
"target_realised": outcome.target,
"roc": outcome.fills.roc,
"reference_roc": outcome.reference.roc,
# Recorded, not rewarded. `efficiency` falls for two different reasons and this
# is what tells them apart in the trace: offers that never filled show here,
# offers that filled at prices not worth reaching do not.
"conversion_ratio": outcome.conversion_ratio,
"paper": outcome.fills.paper,
"filled": float(outcome.fills.bought),
"planned": float(outcome.fills.planned),
"dumped": float(outcome.fills.dumped),
"committed": outcome.fills.committed / max(market.capital, 1),
"conversion": outcome.fills.conversion,
"orders_dropped": float(outcome.dropped),
# Recorded, never rewarded: bad multiplication and bad strategy are different
# failures and score the same without this.
"arith_ok": float(outcome.arith_ok),
"arith_error": outcome.arith_error,
}
@vf.reward(weight=0.45)
async def profit(self, trace: vf.Trace) -> float:
return trace.metrics.get("profit_ratio", 0.0)
@vf.reward(weight=0.30)
async def discipline(self, trace: vf.Trace) -> float:
"""profit_ratio × efficiency, computed once in `scan`. Read from the metric rather
than multiplied here: two rewards that recompute the same product are two rewards
that will eventually disagree about the same episode."""
return trace.metrics.get("discipline", 0.0)
@vf.reward(weight=0.25)
async def gate(self, trace: vf.Trace) -> float:
return trace.metrics.get("clean", 0.0)
class ExchangeConfig(vf.TasksetConfig):
num_tasks: int = Field(48, ge=1)
num_items: int = Field(5, ge=1)
visible: int = Field(56, ge=8)
"""Ticks the agent reads. Enough of them that the mean is an anchor and not a rumour:
shorten this and the reference strategy stops being reliably profitable, which would
take the reward's denominator with it."""
held_out: int = Field(30, ge=1)
"""Ticks it is executed against and never sees."""
class ExchangeTaskset(vf.Taskset[ExchangeTask, ExchangeConfig]):
SEED_BASE: ClassVar[int] = SEED_BASE
def load(self) -> list[ExchangeTask]:
tasks, seed = [], self.SEED_BASE
for i in range(self.config.num_tasks):
# `viable_market` may skip a basket the reference loses money in, so the seed
# stored is the one it landed on and the next scan starts after it — otherwise
# a skip would hand two tasks the same basket.
market = viable_market(
seed, self.config.num_items, self.config.visible, self.config.held_out
)
seed = market.seed + 1
blocks = []
for item in market.items:
prices = " ".join(str(p) for p in item.visible_prices(market.visible))
volumes = " ".join(str(v) for v in item.visible_volumes(market.visible))
blocks.append(
f"{item.name} (buy limit {item.buy_limit} per window)\n"
f" price {prices}\n"
f" volume {volumes}"
)
board = "\n\n".join(blocks)
tasks.append(
ExchangeTask(
ExchangeData(
idx=i,
name=f"basket-{market.seed}",
prompt=(
f"Last {market.visible} ticks, oldest first. Your orders will be "
f"executed against the next {market.held_out}.\n\n{board}\n\n"
f"Purse: {market.capital:,} gp. Place your orders."
),
system_prompt=SYSTEM,
seed=market.seed,
num_items=self.config.num_items,
visible=self.config.visible,
held_out=self.config.held_out,
),
self.config.task,
)
)
return tasks
environments/grand_exchange/pyproject.toml
+13
View File
@@ -0,0 +1,13 @@
[project]
name = "grand-exchange"
version = "0.1.0"
description = "grand-exchange — place orders executed against a market window you never saw."
requires-python = ">=3.11"
dependencies = ["verifiers"]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["grand_exchange"]
environments/grand_exchange/uv.lock
Generated
+3488
View File
File diff suppressed because it is too large Load Diff