probe: discover environments from manifests, not a hardcoded list

The environment list was written out four times — probe.py's shim tuple and its
results dict, tests/test_probe.py's ENVIRONMENTS, and ci.yml's wheel loop — so
adding an environment meant four edits and five workstreams collided on three
files. Discovery is now a tomllib scan of environments/*/pyproject.toml plus a
@probes(name) registry, with an optional [tool.arena] tasksets key for one
package that ships several (tera_spatial declares four).

An environment discovered without a registered probe WARNS rather than exits 1,
because interactive environments land their engine before their probe. That is
temporary; it becomes a hard failure with the last registration.

The warning is not a licence to stop being gated. test_probe.py pins the
ungated set to an explicit PENDING_PROBES allowlist that only ever shrinks —
without it, removing a @probes decorator moved the name from the table into a
warning line that already carries five, and every remaining assertion stayed
vacuously true while the environment silently stopped being gated.

Also untracks three environments/*/uv.lock files, which had been failing the
lock-policy step on main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.gitea/workflows/ci.yml

@@ -23,7 +23,16 @@ jobs:
           python-version: ${{ matrix.python }}
       - name: Enforce library lock policy
         run: test -z "$(find environments -name uv.lock -print -quit)"
-      - name: Run four-environment floor and ceiling probe
+      # An environment is `environments/*/pyproject.toml` and nothing else — the same
+      # denominator probe.py's discover() uses. A half-created scaffold directory with no
+      # manifest is invisible to both, so the two can never disagree about what exists.
+      - name: Refuse an empty environment scan
+        run: |
+          set -euo pipefail
+          count=$(find environments -mindepth 2 -maxdepth 2 -name pyproject.toml | wc -l)
+          echo "discovered $count environment manifests"
+          test "$count" -gt 0
+      - name: Run the floor and ceiling probe for every discovered environment
         run: uv run --with regex python probe.py
       - name: Run root probe integration test
         run: uv run --with regex python -m unittest discover -s tests -v
@@ -35,7 +44,9 @@ jobs:
           python -m unittest discover -s environments/redaction_pressure/tests -v
       - name: Build all environment distributions
         run: |
-          for environment in redaction_pressure canary_trap fault_localisation schema_migration bot_detection grand_exchange drop_table_inference; do
+          set -euo pipefail
+          for manifest in environments/*/pyproject.toml; do
+            environment=$(basename "$(dirname "$manifest")")
             uv build "environments/$environment" --out-dir "dist/$environment"
           done
       - name: Verify documented Redaction Pressure entrypoint

probe.py

@@ -62,7 +62,10 @@ the first time and this is what caught it:
                       the first block was the shortcut's luckiest.
 
 Exit code is the API: 1 if any environment's floor is above zero or its ceiling is
-unreachable.
+unreachable. Nothing here is listed by hand — the environments are `environments/*/pyproject.toml`
+and each probe registers itself with `@probes(taskset_id)`, so adding one is a function appended
+at the end of this file. A manifest with no probe is a warning today and an error the day the
+last one is written.
 
     uv run --with regex python probe.py
 """
@@ -76,25 +79,80 @@ import random
 import re
 import statistics
 import sys
+import tomllib
 import types
+from collections.abc import Callable
 from pathlib import Path
 
 ENVS = Path(__file__).parent / "environments"
 
+
+def discover() -> dict[str, list[Path]]:
+    """Taskset id -> the package directories that ship it, read off the manifests.
+
+    The seven names this file used to carry as a literal tuple are `environments/*/pyproject.toml`
+    now: an environment is gated by existing, not by being remembered here. One directory may
+    declare more than one taskset — the Tera bridge ships four spatial tasksets out of a single
+    package — through an optional `[tool.arena] tasksets = [...]`. Absent that key the taskset id
+    is the project name, which is what all seven of the originals do.
+
+    Other sessions create directories under `environments/` while this runs, so a manifest that
+    disappears between the glob and the read is skipped rather than fatal. A manifest that is
+    present and unparseable is still fatal — that is a broken commit, not a race.
+    """
+    found: dict[str, list[Path]] = {}
+    for manifest in sorted(ENVS.glob("*/pyproject.toml")):
+        try:
+            data = tomllib.loads(manifest.read_text(encoding="utf-8"))
+        except FileNotFoundError:
+            continue
+        wheel = (data.get("tool", {}).get("hatch", {}).get("build", {})
+                 .get("targets", {}).get("wheel", {}))
+        packages = wheel.get("packages") or [manifest.parent.name]
+        names = data.get("tool", {}).get("arena", {}).get("tasksets")
+        if not names:
+            name = data.get("project", {}).get("name")
+            if not name:
+                print(f"warning: {manifest} declares no project.name — skipped", file=sys.stderr)
+                continue
+            names = [name]
+        for name in names:
+            found[name] = [manifest.parent / package for package in packages]
+    return found
+
+
 # Each package's __init__ imports its taskset, which imports verifiers — a heavy dependency
 # this file does not need and should not require. A self-check that only runs once the
 # training stack is installed is a self-check nobody runs. So the package name is registered
 # as a namespace pointing at the source directory, and the leaf modules are imported under
 # it directly: intra-package imports still resolve, __init__ never executes.
-for _pkg in ("redaction_pressure", "canary_trap", "fault_localisation", "schema_migration",
+for _paths in discover().values():
-             "bot_detection", "grand_exchange", "drop_table_inference"):
+    for _path in _paths:
-    _shim = types.ModuleType(_pkg)
+        _shim = types.ModuleType(_path.name)
-    _shim.__path__ = [str(ENVS / _pkg / _pkg)]
+        _shim.__path__ = [str(_path)]
-    sys.modules[_pkg] = _shim
+        sys.modules[_path.name] = _shim
+
+
+# Every floor-and-ceiling function registers itself here under the taskset id `discover()`
+# reads out of the manifest, so adding an environment is one appended function at the end of
+# this file and no edit anywhere else in the repository.
+_PROBES: dict[str, Callable[[], dict[str, float]]] = {}
+
+
+def probes(name: str):
+    def register(fn: Callable[[], dict[str, float]]) -> Callable[[], dict[str, float]]:
+        if name in _PROBES:
+            raise RuntimeError(f"two probes are registered for {name}")
+        _PROBES[name] = fn
+        return fn
+
+    return register
+
 
 TASKS = 24
 
 
+@probes("redaction-pressure")
 def redaction() -> dict[str, float]:
     from redaction_pressure.corpus import build_slices
     from redaction_pressure.scan import measure, parse_rules
@@ -126,6 +184,7 @@ def redaction() -> dict[str, float]:
     }
 
 
+@probes("canary-trap")
 def canary() -> dict[str, float]:
     from canary_trap.corpus import SUBJECTS, build
     from canary_trap.scan import measure, parse_probes
@@ -149,6 +208,7 @@ def canary() -> dict[str, float]:
     }
 
 
+@probes("fault-localisation")
 def fault() -> dict[str, float]:
     from fault_localisation.incident import build, loudest
 
@@ -174,6 +234,7 @@ def fault() -> dict[str, float]:
     }
 
 
+@probes("schema-migration")
 def migration() -> dict[str, float]:
     from schema_migration.run import measure
 
@@ -206,6 +267,7 @@ def migration() -> dict[str, float]:
     }
 
 
+@probes("bot-detection")
 def bots() -> dict[str, float]:
     from bot_detection.accounts import build_slices
     from bot_detection.scan import (
@@ -327,6 +389,7 @@ def bots() -> dict[str, float]:
     }
 
 
+@probes("grand-exchange")
 def exchange() -> dict[str, float]:
     from grand_exchange.book import (
         BUY_BAND, FILL_SHARE, MAX_ITEM_SHARE, MAX_ORDERS, MIN_CROSSINGS, SELL_BAND, TAX,
@@ -605,6 +668,7 @@ def exchange() -> dict[str, float]:
     }
 
 
+@probes("drop-table-inference")
 def drops() -> dict[str, float]:
     from drop_table_inference.estimate import (
         _tail_loss, measure, parse_estimate, reference_estimate,
@@ -749,15 +813,29 @@ def drops() -> dict[str, float]:
 
 
 def main() -> int:
-    results = {
+    discovered = discover()
-        "redaction-pressure": redaction(),
+    if not discovered:
-        "canary-trap": canary(),
+        print("no environments under environments/*/pyproject.toml — the scan is empty, which is"
-        "fault-localisation": fault(),
+              " a broken checkout and not a clean run", file=sys.stderr)
-        "schema-migration": migration(),
+        return 1
-        "bot-detection": bots(),
+
-        "grand-exchange": exchange(),
+    orphans = sorted(set(_PROBES) - set(discovered))
-        "drop-table-inference": drops(),
+    if orphans:
-    }
+        print(f"probe registered for {', '.join(orphans)}, which no manifest declares — a"
+              " renamed or deleted environment", file=sys.stderr)
+        return 1
+
+    # A WARNING and not exit 1, deliberately. A new environment's `pyproject.toml` lands before
+    # its probe does, several sessions share this repository, and a hard gate here turns CI red
+    # for all of them the moment someone scaffolds a directory. This becomes `return 1` in the
+    # same commit that lands `exchange_live()` — the last environment that has a manifest and no
+    # probe. Until then an ungated environment is loud but not fatal.
+    missing = sorted(set(discovered) - set(_PROBES))
+    if missing:
+        print(f"warning: no probe is registered for {', '.join(missing)} — NOT GATED",
+              file=sys.stderr)
+
+    results = {name: _PROBES[name]() for name in sorted(discovered) if name in _PROBES}
     print(f"{'environment':22}{'inaction':>10}{'crude':>9}{'plausible':>11}{'oracle':>9}   verdict")
     failed = False
     for name, r in results.items():

tests/test_probe.py

@@ -1,24 +1,47 @@
 from __future__ import annotations
 
+import re
 import subprocess
 import sys
+import tomllib
 import unittest
 from pathlib import Path
 
 
 ROOT = Path(__file__).resolve().parents[1]
-# Every environment probe.py gates. A new environment that is not added here still has to
+ROW = re.compile(r"^(?P<name>\S(?:.*\S)?)\s{2,}(?:-?\d+\.\d+\s*){4}\s+(?P<verdict>.+?)$")
-# print "ok", because the count assertion below counts them — so this list going stale fails
+
-# the build rather than quietly narrowing what the test covers.
+# The environments that are allowed to carry no probe yet, because their taskset layer is not
-ENVIRONMENTS = (
+# written. Everything discovered and NOT in here MUST be gated.
-    "redaction-pressure",
+#
-    "canary-trap",
+# This list is the floor, and it is why it exists rather than a bare count: without it, both
-    "fault-localisation",
+# loops below are vacuous when nothing is gated, and CI goes green while gating zero
-    "schema-migration",
+# environments. Commenting out a single `@probes(...)` decorator used to pass — the degated
-    "bot-detection",
+# name simply moved into the warning line, which already legitimately carries five names.
-    "grand-exchange",
+#
-    "drop-table-inference",
+# It only ever shrinks. Delete a name here in the same commit that lands its probe.
-)
+PENDING_PROBES = frozenset({
+    "grand-exchange-live",
+    "tera-crow-nav",
+    "tera-drive-101",
+    "tera-office-nav",
+    "tera-california-flight",
+})
+
+
+def discovered_tasksets() -> set[str]:
+    """The same denominator `probe.py:discover()` uses, derived independently of it.
+
+    Deriving it a second time here rather than importing `probe` is the point: if discovery
+    and the manifests ever disagree, this test is what says so. A directory with no
+    `pyproject.toml` is not an environment to either of us.
+    """
+    names: set[str] = set()
+    for manifest in sorted((ROOT / "environments").glob("*/pyproject.toml")):
+        data = tomllib.loads(manifest.read_text(encoding="utf-8"))
+        declared = data.get("tool", {}).get("arena", {}).get("tasksets")
+        names.update(declared or [data["project"]["name"]])
+    return names
 
 
 class RootProbeIntegrationTests(unittest.TestCase):
@@ -28,18 +51,47 @@ class RootProbeIntegrationTests(unittest.TestCase):
             cwd=ROOT,
             text=True,
             capture_output=True,
-            timeout=30,
+            # The probe sweeps constants and runs multi-block ladders, and the interactive
+            # environments will make it dearer still. A tight timeout here only ever fails a
+            # slow machine, never a wrong reward.
+            timeout=180,
             check=False,
         )
-        self.assertEqual(
+        detail = f"\nstdout:\n{completed.stdout}\nstderr:\n{completed.stderr}"
-            completed.returncode,
+        self.assertEqual(completed.returncode, 0, msg=f"root probe failed:{detail}")
-            0,
+
-            msg=f"root probe failed:\nstdout:\n{completed.stdout}\nstderr:\n{completed.stderr}",
+        discovered = discovered_tasksets()
+        self.assertTrue(discovered, msg="no environment manifests found — the scan is empty")
+
+        rows = {}
+        for line in completed.stdout.splitlines()[1:]:
+            match = ROW.match(line.rstrip())
+            if match:
+                rows[match["name"]] = match["verdict"].strip()
+
+        for name, verdict in rows.items():
+            with self.subTest(environment=name):
+                self.assertIn(name, discovered, msg=f"probe gated {name}, no manifest declares it")
+                self.assertEqual(verdict, "ok", msg=f"{name} is not ok:{detail}")
+
+        # Anything discovered and not gated has to be named in the warning. That warning is
+        # temporary — it becomes a hard exit 1 in probe.py once the last unregistered
+        # environment lands its probe — and this assertion holds either way.
+        ungated = discovered - set(rows)
+        for name in sorted(ungated):
+            with self.subTest(environment=name):
+                self.assertIn(name, completed.stderr,
+                              msg=f"{name} is neither gated nor reported as ungated:{detail}")
+
+        # The floor. Being named in the warning is not a licence to stop being gated: an
+        # environment may only be ungated if it is on the pending list. Without this, removing
+        # a `@probes(...)` decorator moves the name from the table into the warning and every
+        # assertion above stays vacuously true.
+        self.assertLessEqual(
+            ungated, PENDING_PROBES,
+            msg=f"these were gated and no longer are: {sorted(ungated - PENDING_PROBES)}{detail}",
         )
-        for environment in ENVIRONMENTS:
+        self.assertTrue(rows, msg=f"the probe gated nothing at all:{detail}")
-            with self.subTest(environment=environment):
-                self.assertIn(environment, completed.stdout)
-        self.assertEqual(completed.stdout.count("   ok"), len(ENVIRONMENTS))
 
 
 if __name__ == "__main__":