podman/docs/PLAN.md

PodMan — Implementation Plan (12 hours)

Architecture locked. See docs/idea.md for concept, integration specs for details. Tasks ordered by dependency and demo criticality. Never cut items above the cut line.

---

What we are building

Engineers join a LiveKit room with earbuds. The PodMan agent (backend/src/agent.ts) subscribes to each engineer's screen-share track, samples frames at ~1fps, calls Gemini Vision, detects collisions, and publishes interventions via LiveKit data channel + voice. MongoDB Atlas stores observations, collisions, interventions, and outcomes — persisting across sessions for continual learning.

Actual architecture (differs from original spec): Screen frames flow through the LiveKit room (agent subscribes to screen tracks), not via HTTP POST /ingest from the PWA. The agent runs as a separate process (pnpm dev:agent). The HTTP server handles token minting, pods CRUD, memory stats, and outcome recording.

Demo: Alice builds auth endpoint. Carol is blocked. PodMan notices, warns Carol. Alice's server starts. PodMan tells Carol and Bob they're clear to integrate. No Slack. No asking.

---

Status legend

• ✅ Done and deployed on server
• ⚠️ Partial — code exists, needs work
• ❌ Not started

---

Critical path — must ship for demo

1. Environment + health check

Status: ✅ Done

• GET /health returns { ok: true } — live at http://165.22.129.249:8787/health
• MongoDB connection established in backend/src/memory/db.ts — collections() export
• Pods CRUD: GET/POST /api/pods, GET/DELETE /api/pods/:id, POST/DELETE /api/pods/:id/members
• Backend running in tmux on DO droplet (pnpm dev via tsx watch src/server.ts)
• Root .env not present — backend/.env has creds, sufficient for current setup

Actual files: backend/src/memory/db.ts, backend/src/server.ts, backend/src/env.ts

---

2. PWA frame capture

Status: ✅ Done (different approach than originally planned)

Original plan: PWA captures JPEG every 30s → POST /ingest. Actual: PWA publishes screen as LiveKit video track → agent subscribes and samples at ~1fps using livekit-client-node + sharp.

• frontend/src/lib/pod.ts — joinPod() connects to LiveKit, publishes screen + mic
• backend/src/agent.ts — subscribes to SOURCE_SCREENSHARE tracks, throttles at SAMPLE_INTERVAL_MS
• Resize to max 1280px wide, JPEG quality 70 via sharp
• Dev mock-join fallback when LiveKit unconfigured
• frontend/src/livekit/useScreenPublish.ts — React hook for screen track

Actual files: backend/src/agent.ts, frontend/src/lib/pod.ts, frontend/src/livekit/useScreenPublish.ts

---

2b. Local git watcher script

Status: ❌ Not started

• scripts/podman-agent.mjs — CLI script polling git every 15s, writing to MongoDB
• Args: --name alice --pod demo-pod
• Every 15s: git status --short, git diff --stat HEAD, git log --oneline -1, git branch --show-current
• Upsert into observations collection — update only git fields, leave vision fields untouched
• Graceful exit on Ctrl+C

Usage:

node scripts/podman-agent.mjs --name alice --pod demo-pod

Files: scripts/podman-agent.mjs (new)

---

3. MongoDB state layer

Status: ✅ Done (different collection names than originally planned)

• observations — recordObservation(ctx: EngineerContext)
• collisions — recordCollision(collision: Collision)
• interventions — recordIntervention(intervention: Intervention)
• outcomes — recordOutcome(outcome: InterventionOutcome)
• memoryStats() — count per collection, exposed via GET /api/memory/stats
• Vector recall: recallSimilar(collision) in backend/src/memory/vectors.ts
• Policy gate: shouldIntervene(collision, prior) in backend/src/memory/policy.ts

Actual files: backend/src/memory/db.ts, backend/src/memory/store.ts, backend/src/memory/policy.ts, backend/src/memory/vectors.ts

---

4. Gemini Vision pipeline

Status: ✅ Done

• analyzeFrame(engineerId, podId, jpeg) calls gemini-2.0-flash with structured JSON schema
• Extracts: currentFile, currentSymbol, activity, hasUnpushedChanges, confidence
• Low media resolution for speed, zero thinking budget
• Called from agent.ts → podman.onScreenFrame() → analyzeFrame() → recordObservation()
• Collision detection runs after every frame: detectCollisions([...contexts], github)

Actual files: backend/src/vision/gemini.ts, backend/src/agent/podman.ts

---

5. Event detector + nudge generator

Status: ✅ Done (collision-based, not Gemini-event-detection-based)

• detectCollisions() — groups engineers by normalized file path, flags when 2+ editing same file with unpushed changes
• shouldIntervene() + preferredAction() policy gate (prevents spam)
• recallSimilar() — vector recall elevates severity if prior collision on same file
• Generates intervention message string (inline, not via Gemini call)
• Publishes { type: 'COLLISION', collision, intervention } via data channel
• Calls speak(room, message) (currently stub — just logs)
• Cooldown per pod not yet implemented (policy gate provides some protection)

Actual files: backend/src/collision/detector.ts, backend/src/agent/podman.ts, backend/src/memory/policy.ts

---

6. Voice via LiveKit

Status: ⚠️ Partial — agent structure done, voice is a stub

• backend/src/agent.ts — full LiveKit agent: joins room, subscribes to screen tracks, 1fps sampling
• backend/src/voice/live.ts — speak(room, message) function wired into pipeline
• room.localParticipant.publishData() — data channel publish working
• speak() is a stub — logs message but does NOT produce audio
• Gemini Live 2.5 audio streaming not implemented
• Agent not started on server — start-podman.sh has it commented out; needs pnpm dev:agent + real LiveKit creds

To unblock: Implement real TTS in speak() (Gemini TTS → WAV → publish audio track), then uncomment agent window in start-podman.sh.

Actual files: backend/src/voice/live.ts, backend/src/agent.ts, backend/src/agent/podman.ts

---

7. PWA active session UI

Status: ❌ Not started

• frontend/src/components/SessionView.tsx — active session screen post-join
• Teammate status cards (name, inferred file, inferred task)
• RoomEvent.DataReceived listener for collision/intervention messages from agent
• Nudge feed: last 5 interventions, timestamped
• "PodMan is watching" indicator

Depends on: Task 6 (agent producing data channel messages — working even without voice)

Files: frontend/src/components/SessionView.tsx (new), frontend/src/App.tsx (wire post-join)

---

Cut line — below here only if tasks 1–7 done before hour 10

8. Ownership warm-start demo

Status: ⚠️ Partial — memory persists, no warm-start logging

• On agent startup: log "Loading session memory for pod X — N collisions known"
• Demo: session 1 cold, session 2 faster — visible in logs

---

9. GitHub-enriched collision detection

Status: ⚠️ Exists but may not be needed for demo

• backend/src/github/client.ts exists — fetches PR state as GithubStateSnapshot
• Not required for the demo since screen-based hasUnpushedChanges is sufficient

---

10. Backend state endpoint

Status: ✅ Done (as pods API)

• GET /api/pods — returns all pods
• GET /api/pods/:id — returns pod with members
• No per-engineer state endpoint yet (frontend uses data channel instead)

---

What needs to happen next (priority order)

1. Voice (Task 6) — implement real speak() with Gemini TTS → WAV → LiveKit audio track
2. Session UI (Task 7) — SessionView.tsx with data channel listener + nudge feed
3. Start agent on server — uncomment agent window in start-podman.sh, ensure LiveKit creds in backend/.env
4. Git watcher (Task 2b) — scripts/podman-agent.mjs for demo terminals

---

Server state (165.22.129.249)

• Backend: ✅ running on :8787 (tmux podman:backend)
• Frontend: ✅ running on :81 (tmux podman:prod, port 80 taken)
• Agent: ❌ NOT running — needs LiveKit creds + pnpm dev:agent
• To start agent: tmux new-window -t podman -n agent && tmux send-keys -t podman:agent 'cd /root/podman && pnpm --filter @podman/backend dev:agent' C-m
• To attach: ssh root@165.22.129.249 → tmux attach -t podman

---

Cut immediately

• Mic transcription or voice input from engineers
• Manual task input fields on PWA
• Multilingual voice
• Slack / Linear / Jira integrations
• Webcam tracks
• Always-on raw screen surveillance (1fps sampling is by design)
• Full task management features
• User auth / accounts

---

Risk table

Risk	Mitigation
Gemini Vision accuracy on screens	Large font (18pt+), single editor window, file tab fully visible.
Gemini Live TTS not implemented	Fallback: @google/genai TTS → WAV buffer → publish as audio track manually.
Agent needs HTTPS for screen capture	Demo from localhost or ngrok. PWA has dev mock-join fallback.
Event detection false positives	Policy gate in shouldIntervene(). Demo is pre-staged so collisions fire cleanly.
DO deploy: frontend on :81 not :80	Port 80 taken — either kill the process or update DNS/proxy to :81.
Multiple collisions fire at once	Policy gate prevents spam; same-collision dedup via file+engineers key.

---

Demo script (3 min)

(0:00) Three laptops visible. Alice, Bob, Carol join demo-pod via PWA. Agent logs [agent] PodMan joined room demo-pod.

(0:20) Alice opens auth/middleware.ts (18pt font, clearly visible). First frame processed. [vision] currentFile: auth/middleware.ts.

(0:45) Bob opens auth/middleware.ts too. Carol runs curl http://localhost:3001/auth → connection refused.

(1:20) MONEY MOMENT 1 — COLLISION_DETECTED: PodMan data channel message fires: "alice and bob are both editing auth/middleware.ts and one has unpushed changes." Voice says it aloud in the room.

(1:50) Alice pushes. Bob pulls. Carol's integration unblocks.

(2:20) Show GET /api/memory/stats → { observations: 47, collisions: 1, interventions: 1, outcomes: 0 }. Session 2 warm-start: collision detected in 12s vs 3min cold.

(2:45) Close: "PodMan — the teammate that sees what Slack can't."

---

Pre-demo checklist (day of)

See docs/demo-setup.md for full laptop setup. Key items:

• All laptops: font 18pt+, single editor window, file tab visible
• demo-pod created in LiveKit Cloud, creds in backend/.env
• Agent running: tmux attach -t podman → check agent window
• GET /health returns OK on deployed URL
• Earbuds tested — voice audible through browser
• Backup video recorded and on separate device
• Demo rehearsed 3×