lumbridge-code/docs/TESTING.md

Testing strategy

Lumbridge controls shells, repositories, credentials, agents, and long-lived processes. A test plan built only from UI clicks would be slow, flaky, and blind to the failures that matter. The product is designed with programmatic testing surfaces from the start.

The central seam

The desktop UI and the test driver use the same versioned local IPC commands and events exposed by the session runtime. UI tests can therefore:

1. start an isolated runtime with a temporary data directory;
2. issue real commands such as create workspace, split pane, launch harness, resize PTY, approve tool, cancel turn, and restore session;
3. wait on semantic events rather than sleeps;
4. query a redacted state snapshot; and
5. separately confirm the visible/accessibility representation where needed.

The test API is not a back door. It uses the same validation and state machine as the UI, binds locally, requires a per-run capability token, and is absent from release builds unless explicitly enabled.

Test layers

Pure and property tests

Layouts, terminal grid transitions, usage windows, forecasts, event reduction, path rules, and capability negotiation are deterministic functions. Test them exhaustively without starting a window. Property tests should target split-tree invariants, event replay, Unicode width, and usage aggregation.

Contract tests

Every platform adapter implements the same suites for PTYs, process groups, signals, resize, secret handles, notifications, paths, and atomic replacement. Every harness adapter implements discovery, launch, resume, cancel, status, and usage contracts. Unsupported capability is a valid explicit result.

Fake harnesses

Small Rust fixture binaries behave like coding agents without network access or subscriptions. Scripts cover partial frames, noisy stderr, permission prompts, child processes, crashes, hangs, huge output, invalid bytes, and cancellation. Tests assert both user-visible state and cleanup of the entire process tree.

ACP replay

Synthetic ACP transcripts are replayed in both directions against the official Rust SDK. Golden fixtures cover protocol versions and capability combinations. Fuzzing targets frame decoding and state transitions; stdout must remain protocol pure and diagnostics must stay on stderr.

Terminal conformance

Byte-stream fixtures assert screen cells, styles, cursor, alternate screen, scrollback, selection, hyperlinks, bracketed paste, keyboard modes, and Unicode. PTY integration tests exercise real shells on each supported OS. Human typing and agent automation are stressed concurrently to catch input-order corruption.

Persistence and recovery

The runtime is interrupted after each durable event and external mutation, then restarted. Tests prove accepted commands are not lost, incomplete work is reconciled, migrations are forward-only, and corrupt/truncated tails fail safely.

UI and rendering

Component tests drive semantic actions and accessibility identifiers. Golden images cover a small set of high-value layouts at fixed fonts, scale, theme, and GPU/software renderer. Snapshot updates require review; snapshots never replace state assertions.

Performance gates

Criterion and end-to-end probes record input-to-present latency, output ingest, scrollback search, six-pane frame time, idle CPU, memory per pane, startup, and restore. CI uses generous regression ceilings; dedicated metal/macOS runs retain the detailed distributions.

Packaging smoke tests

Fresh macOS, Ubuntu, and Omarchy/Arch environments install, launch, open a shell, upgrade, roll back, and uninstall. Artifacts are checked for signatures, checksums, SBOM, license notices, forbidden secrets, and source revision.

Developer loop

# Continuous cargo check. Press t for nextest, c for strict Clippy, v for all.
bacon

# Direct fast test run.
cargo xtest

# Exactly what must pass before a commit.
./scripts/ci.sh

cargo-watch is no longer the project default because upstream archived it in 2025. scripts/watch.sh uses Bacon and retains a cargo-watch fallback for people who already have the final cargo-watch release installed.

Implemented vertical-slice coverage

• The framework-neutral six-surface model has deterministic tests for focus, direct selection, needs-input transitions, command-palette lifecycle, identical GPUI/Floem action replay, bounded output, and workload counters.
• lumbridge-pty runs synthetic /bin/sh tests for raw output, non-zero exits, input, resize, one-chunk backpressure, hung-process termination, invalid dimensions, and exclusion of provider credentials from the child environment.
• lumbridge-runtime tests ordered actor output, serialized input and resize, event polling, invalid configuration, and bounded-time cleanup of a hung PTY.
• The GPUI slice tests byte line-framing across CRLF, chunk boundaries, split UTF-8, and invalid bytes. It renders one real actor-owned PTY and keeps five surfaces deterministic; line framing is explicitly not VT emulation.
• The current-GPUI probe compile-checks real AccessKit element wiring and real platform input-handler installation. Unit tests cover its semantic tree and UTF-16/UTF-8 composed-text mutations. OS screen readers, IME candidate windows, and macOS behavior remain platform tests.

CI matrix as the project grows

Every change runs format, Clippy, unit, contract, integration, and doctests on Linux. Pull requests that touch platform/UI code also run macOS. Nightly jobs add MSRV, coverage, dependency/license audit, fuzz smoke, renderer snapshots, soak, and performance tracking. Release candidates must pass native packaging smoke on macOS, Ubuntu, and Arch/Omarchy rather than treating cross-compilation as proof.