lumbridge-public/arena
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Harden redaction-pressure reward and provenance
arena-environments / validate (3.11) (push) Successful in 1m8s
Details
arena-environments / validate (3.12) (push) Successful in 31s
Details

Browse Source
This commit is contained in:
karti
2026-08-19 01:26:21 -07:00
parent 5a99eb86a5
commit d691acf131
16 changed files with 897 additions and 163 deletions
Download Patch File Download Diff File Expand all files Collapse all files
probe.py
+15 -13
View File
@@ -3,9 +3,9 @@
Run this before believing any reward in this repository. Every environment here was wrong
the first time and this is what caught it:
redaction-pressure scored SURVIVORS — secrets caught, decoys kept, prose kept. An empty
ruleset destroys nothing so it keeps everything, and it tied the best
real attempt at 0.500. Counting removals instead put inaction at zero.
redaction-pressure originally scored SURVIVORS, so an empty ruleset tied a real attempt.
Its v0.2 scanner additionally tracks source offsets: changing one byte
of a secret earns zero recall, and malformed/unsafe rules earn zero.
canary-trap let the paraphrase restate the corpus verbatim for facts that had no
distinct rewording, so GUID probes caught it by accident and the
environment taught nothing. Every fact now has a `core` token that is
@@ -50,23 +50,25 @@ def redaction() -> dict[str, float]:
def run(rules: list[dict]) -> float:
total = 0.0
for i in range(TASKS):
_, held = build_slices(90_000 + i, 4, 12)
_, held = build_slices(200_000 + i, 4, 12)
o = measure(held, parse_rules("```json\n" + json.dumps(rules) + "\n```"))
total += 0.35 * o.recall + 0.35 * o.precision + 0.30 * float(o.clean)
total += o.validity * (
0.35 * o.recall + 0.35 * o.precision + 0.30 * float(o.clean)
)
return total / TASKS
oracle = [
{"pattern": r"sk-live-\w+", "replacement": "[KEY]"},
{"pattern": r"[\w.]+@(?!lumbridgecorp\.com)[\w.]+\.com", "replacement": "[EMAIL]"},
{"pattern": r"\+1-\d{3}-\d{3}-\d{4}", "replacement": "[PHONE]"},
{"pattern": r"(?<!ORD-)\b\d{4}-\d{4}-\d{4}-\d{4}\b", "replacement": "[CARD]"},
{"pattern": r"\b[\w-]+\.internal\.lumbridge\b", "replacement": "[HOST]"},
{"pattern": r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b", "replacement": "[GUID]"},
{"pattern": r"\b(Dana|Priya|Marcus|Ines|Tomas|Rui|Nadia|Owen) (Okafor|Lindqvist|Baptiste|Moreau|Sato|Ahmed|Vance|Duarte)\b", "replacement": "[PERSON]"},
{"pattern": r"(?<=Customer pasted their key )sk-live-[a-z0-9]{24}", "replacement": "[REDACTED]"},
{"pattern": r"(?<=Reply-to was )[a-z]+[.][a-z]+@[a-z]{6}[.]com", "replacement": "[REDACTED]"},
{"pattern": r"(?<=Callback number )[+]1-[0-9]{3}-[0-9]{3}-[0-9]{4}", "replacement": "[REDACTED]"},
{"pattern": r"(?<=They read the card )[0-9]{4}(?:-[0-9]{4}){3}", "replacement": "[REDACTED]"},
{"pattern": r"(?<=Trace pointed at )[a-z]{5}-db-[1-9][.]internal[.]lumbridge", "replacement": "[REDACTED]"},
{"pattern": r"(?<=Session )[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}", "replacement": "[REDACTED]"},
{"pattern": r"(?<=Escalated by )(?:Dana|Priya|Marcus|Ines|Tomas|Rui|Nadia|Owen) (?:Okafor|Lindqvist|Baptiste|Moreau|Sato|Ahmed|Vance|Duarte)", "replacement": "[REDACTED]"},
]
return {
"inaction": run([]),
"crude": run([{"pattern": r"\S+", "replacement": "[X]"}]),
"crude": run([{"pattern": r"\S+", "replacement": "[REDACTED]"}]),
"plausible": run(oracle[:4]),
"oracle": run(oracle),
}