karti-ai
a56f097f28
The Python is the source of truth; src/demos/wordle/engine.ts will be a port of it, and CI gates the two against a SHA-256 over all 21.2M (guess, answer) pattern pairs rather than a hand-picked vector file — a vector file only ever catches the cases somebody thought of. The reward is three weighted components, and the third one is the reason this demo is worth building. `solved` and `economy` pull toward winning. `consistency` pulls against them, because a player maximising information deliberately guesses words that cannot win — a word that splits the remaining candidates evenly teaches more than a word that might happen to be right. That is good play, and it costs consistency. The probe ladder proves the tension is real rather than asserted: inaction 0.0000 crude 0.0111 plausible 0.1224 candidate_only 0.8925 exhaustive 0.9031 oracle 0.9458 The two good policies are 0.05 apart and neither dominates — the entropy oracle takes 1.00 economy and 0.73 consistency, the candidate-only player takes 0.75 and 1.00. Which one wins is a decision about what you want, which is the whole argument the site exists to make. probe.py fails CI if either starts dominating. Two traps found by building it. `consistency` is scored over turns SPENT, not guesses accepted: counting only legal guesses hands a free 1.0 to a policy that plays one word and then jams the parser five times — one guess, no contradictions, perfect score. And `economy`'s denominator is the depth the SHIPPED solver reaches, not a depth-optimal search: entropy-greedy is not depth-optimal, so grading it against an exact optimum would make the oracle rung fail its own assertion on some seeds. The word lists are built from Wordnik (MIT) intersected with SCOWL, never from the original game's 2,315 answers. 4,603 answers makes this materially harder than the original, so the published SALET/3.4212 results are cited as belonging to that list and our own reference player's TARES/3.72 is measured here. verifiers is an optional extra. The engine, reward, solver and probe all run — and gate — without an RL stack resolvable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mt6sHQHEnEYrJZvoMCJSB
66 lines
2.4 KiB
Python
66 lines
2.4 KiB
Python
"""Turning model text into a move, and a move into text.
|
|
|
|
The one rule: **never raise**. A malformed reply is a thing the model did, not
|
|
an error in the harness, and it must be scored rather than crash the rollout.
|
|
A parse failure becomes a rejected guess, which costs the model a turn of its
|
|
budget and shows up in `rejected_replies`.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import re
|
|
|
|
from .engine import MAX_GUESSES, WORD_LENGTH
|
|
|
|
SYSTEM_PROMPT = """You are playing a word-guessing game.
|
|
|
|
A secret {length}-letter word has been chosen. You have {turns} attempts.
|
|
After each guess you are told, for every letter:
|
|
G — correct letter, correct position
|
|
Y — the letter is in the word, but somewhere else
|
|
X — the letter is not in the word
|
|
|
|
Think it through, then give your guess as the only square-bracketed token in
|
|
your reply, like [crane]. The game reads the first bracketed token and ignores
|
|
everything else, so do not put any other words in brackets."""
|
|
|
|
_BRACKETED = re.compile(r"\[([A-Za-z]+)\]")
|
|
|
|
|
|
def system_prompt() -> str:
|
|
return SYSTEM_PROMPT.format(length=WORD_LENGTH, turns=MAX_GUESSES)
|
|
|
|
|
|
def parse_guess(reply: str | None) -> str | None:
|
|
"""The first bracketed alphabetic token, lowercased. None if there is none.
|
|
|
|
Deliberately does NOT check length or membership — that is the game's job,
|
|
so that 'the model guessed a six-letter word' and 'the model produced no
|
|
guess at all' stay distinguishable in the metrics.
|
|
"""
|
|
if not reply:
|
|
return None
|
|
match = _BRACKETED.search(reply)
|
|
return match.group(1).lower() if match else None
|
|
|
|
|
|
def render_feedback(guess: str, pattern: str, turns_left: int) -> str:
|
|
"""The feedback block, in the format the system prompt promised.
|
|
|
|
Letters and tiles are space-separated and positionally aligned. Most
|
|
tokenizers give a space-separated character its own token, which is the
|
|
difference between the model reading position 3 and guessing at it.
|
|
"""
|
|
letters = " ".join(guess.upper())
|
|
tiles = " ".join(pattern)
|
|
plural = "" if turns_left == 1 else "es"
|
|
return f"{letters}\n{tiles}\n\nYou have {turns_left} guess{plural} left."
|
|
|
|
|
|
def render_rejection(reason: str, turns_left: int) -> str:
|
|
plural = "" if turns_left == 1 else "es"
|
|
return (
|
|
f"That guess was not accepted: {reason}.\n"
|
|
f"It did not use up a turn. You have {turns_left} guess{plural} left."
|
|
)
|