This commit is contained in:
+110
@@ -0,0 +1,110 @@
|
||||
# Testing strategy
|
||||
|
||||
Lumbridge controls shells, repositories, credentials, agents, and long-lived
|
||||
processes. A test plan built only from UI clicks would be slow, flaky, and blind
|
||||
to the failures that matter. The product is designed with programmatic testing
|
||||
surfaces from the start.
|
||||
|
||||
## The central seam
|
||||
|
||||
The desktop UI and the test driver use the same versioned local IPC commands and
|
||||
events exposed by the session runtime. UI tests can therefore:
|
||||
|
||||
1. start an isolated runtime with a temporary data directory;
|
||||
2. issue real commands such as create workspace, split pane, launch harness,
|
||||
resize PTY, approve tool, cancel turn, and restore session;
|
||||
3. wait on semantic events rather than sleeps;
|
||||
4. query a redacted state snapshot; and
|
||||
5. separately confirm the visible/accessibility representation where needed.
|
||||
|
||||
The test API is not a back door. It uses the same validation and state machine as
|
||||
the UI, binds locally, requires a per-run capability token, and is absent from
|
||||
release builds unless explicitly enabled.
|
||||
|
||||
## Test layers
|
||||
|
||||
### Pure and property tests
|
||||
|
||||
Layouts, terminal grid transitions, usage windows, forecasts, event reduction,
|
||||
path rules, and capability negotiation are deterministic functions. Test them
|
||||
exhaustively without starting a window. Property tests should target split-tree
|
||||
invariants, event replay, Unicode width, and usage aggregation.
|
||||
|
||||
### Contract tests
|
||||
|
||||
Every platform adapter implements the same suites for PTYs, process groups,
|
||||
signals, resize, secret handles, notifications, paths, and atomic replacement.
|
||||
Every harness adapter implements discovery, launch, resume, cancel, status, and
|
||||
usage contracts. Unsupported capability is a valid explicit result.
|
||||
|
||||
### Fake harnesses
|
||||
|
||||
Small Rust fixture binaries behave like coding agents without network access or
|
||||
subscriptions. Scripts cover partial frames, noisy stderr, permission prompts,
|
||||
child processes, crashes, hangs, huge output, invalid bytes, and cancellation.
|
||||
Tests assert both user-visible state and cleanup of the entire process tree.
|
||||
|
||||
### ACP replay
|
||||
|
||||
Synthetic ACP transcripts are replayed in both directions against the official
|
||||
Rust SDK. Golden fixtures cover protocol versions and capability combinations.
|
||||
Fuzzing targets frame decoding and state transitions; stdout must remain protocol
|
||||
pure and diagnostics must stay on stderr.
|
||||
|
||||
### Terminal conformance
|
||||
|
||||
Byte-stream fixtures assert screen cells, styles, cursor, alternate screen,
|
||||
scrollback, selection, hyperlinks, bracketed paste, keyboard modes, and Unicode.
|
||||
PTY integration tests exercise real shells on each supported OS. Human typing
|
||||
and agent automation are stressed concurrently to catch input-order corruption.
|
||||
|
||||
### Persistence and recovery
|
||||
|
||||
The runtime is interrupted after each durable event and external mutation, then
|
||||
restarted. Tests prove accepted commands are not lost, incomplete work is
|
||||
reconciled, migrations are forward-only, and corrupt/truncated tails fail safely.
|
||||
|
||||
### UI and rendering
|
||||
|
||||
Component tests drive semantic actions and accessibility identifiers. Golden
|
||||
images cover a small set of high-value layouts at fixed fonts, scale, theme, and
|
||||
GPU/software renderer. Snapshot updates require review; snapshots never replace
|
||||
state assertions.
|
||||
|
||||
### Performance gates
|
||||
|
||||
Criterion and end-to-end probes record input-to-present latency, output ingest,
|
||||
scrollback search, six-pane frame time, idle CPU, memory per pane, startup, and
|
||||
restore. CI uses generous regression ceilings; dedicated metal/macOS runs retain
|
||||
the detailed distributions.
|
||||
|
||||
### Packaging smoke tests
|
||||
|
||||
Fresh macOS, Ubuntu, and Omarchy/Arch environments install, launch, open a shell,
|
||||
upgrade, roll back, and uninstall. Artifacts are checked for signatures,
|
||||
checksums, SBOM, license notices, forbidden secrets, and source revision.
|
||||
|
||||
## Developer loop
|
||||
|
||||
```bash
|
||||
# Continuous cargo check. Press t for nextest, c for strict Clippy, v for all.
|
||||
bacon
|
||||
|
||||
# Direct fast test run.
|
||||
cargo xtest
|
||||
|
||||
# Exactly what must pass before a commit.
|
||||
./scripts/ci.sh
|
||||
```
|
||||
|
||||
`cargo-watch` is no longer the project default because upstream archived it in
|
||||
2025. `scripts/watch.sh` uses Bacon and retains a cargo-watch fallback for people
|
||||
who already have the final cargo-watch release installed.
|
||||
|
||||
## CI matrix as the project grows
|
||||
|
||||
Every change runs format, Clippy, unit, contract, integration, and doctests on
|
||||
Linux. Pull requests that touch platform/UI code also run macOS. Nightly jobs add
|
||||
MSRV, coverage, dependency/license audit, fuzz smoke, renderer snapshots, soak,
|
||||
and performance tracking. Release candidates must pass native packaging smoke on
|
||||
macOS, Ubuntu, and Arch/Omarchy rather than treating cross-compilation as proof.
|
||||
Reference in New Issue
Block a user