docs: finalize PodMan architecture and write full integration specs
Replaces v1 plan with locked architecture: - Hermes orchestrator: POST /ingest → Gemini Vision → MongoDB → event detection → Gemini Live 2.5 voice via LiveKit Agents - Four MongoDB collections: engineer_states, ownership_map, events, nudges - Continual learning via ownership_map persisting across sessions New files: docs/idea.md, docs/gemini.md, docs/livekit.md, docs/mongodb.md, docs/digitalocean.md, docs/demo-setup.md Updated: README.md, docs/PLAN.md (12-hour build plan), database/README.md, infra/README.md, .env.example Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FFbfi4Cmb7BY75Wtne7bZn
This commit is contained in:
Ramis
+109
-145
@@ -1,188 +1,152 @@
|
||||
# PodMan — Master Plan (v1)
|
||||
# PodMan — Build Plan (12 hours)
|
||||
|
||||
> Living doc. A deeper, API-accurate v2 (exact Gemini/LiveKit SDK calls, starter code,
|
||||
> DO deploy steps) is being generated by the research workflow and will be merged in.
|
||||
> Replaces the original v1 plan. Architecture finalized. See `docs/idea.md` for concept, individual integration specs for details.
|
||||
|
||||
---
|
||||
|
||||
## 0. TL;DR
|
||||
## What we are building
|
||||
|
||||
**PodMan** is an ambient AI teammate. Engineers join a **pod**, share screen + mic, and
|
||||
PodMan watches everyone's screen in realtime, understands what each person is doing
|
||||
(Gemini vision), fuses it with the team's GitHub state, and **interrupts like Jarvis to
|
||||
prevent merge collisions before anyone pushes**, offering to open a sync PR.
|
||||
PodMan is a real-time AI team coordination agent. Engineers join a LiveKit room with earbuds. Each engineer's browser PWA captures their screen every 30s and sends it to Hermes (server-side orchestrator). Hermes uses Gemini Vision to extract structured context, detects coordination events (dependency ready, blocker, duplicate work), and speaks proactive nudges into the room via Gemini Live 2.5. MongoDB Atlas stores team state and an ownership map that persists across sessions — making PodMan faster and smarter each session.
|
||||
|
||||
- **Track:** Continual Learning
|
||||
- **Prizes targeted (stacked):** Gemini 3.5 ($5k cash), LiveKit (keyboards), DigitalOcean (credits)
|
||||
- **Hero moment:** two laptops editing the same file → PodMan _speaks up live_ and offers the fix.
|
||||
**Hero demo moment:** PodMan detects Carol is blocked waiting for Alice's auth endpoint, warns Carol, then notifies Carol and Bob the moment Alice's server starts — without anyone sending a message.
|
||||
|
||||
---
|
||||
|
||||
## 1. Why this fits the track (and stays eligible)
|
||||
## Must-have demo path
|
||||
|
||||
### Track = Continual Learning
|
||||
The minimum end-to-end flow required for a winning demo:
|
||||
|
||||
The official definition rewards systems that "continuously improve from real-world use…
|
||||
becoming more useful the more they are used with as little user intervention as possible."
|
||||
PodMan does exactly this:
|
||||
|
||||
- Builds and **continuously refines a live model of the team** — who owns which files/areas,
|
||||
what's in-flight, recurring conflict patterns, each engineer's working style.
|
||||
- **Self-improves its own intervention policy** from outcomes: did the collision it predicted
|
||||
actually happen? Did the team accept the suggested PR? It tunes its thresholds/prompts so it
|
||||
nags less and helps more over time.
|
||||
- Grows a per-team **skill/memory store** (vector memory) that makes later sessions sharper.
|
||||
|
||||
### ⚠️ Disqualification traps — and how we dodge them
|
||||
|
||||
| Risk | Mitigation |
|
||||
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **"Dashboard is the main feature" = auto-DQ** | The pod grid is _secondary_. The hero is PodMan's **proactive voice/card interventions**. In the demo we barely show the grid; we show PodMan _acting_. |
|
||||
| Repo must be **public** | Make the GitHub repo public from the start. |
|
||||
| **Only what you built** during the event | Everything in this monorepo is new, timestamped by commits. Demo narrates "built today." |
|
||||
| New work only | No pre-existing project reuse. |
|
||||
1. 3 engineers join a pod room via PWA (browser tab)
|
||||
2. PWA captures screen frame every 30s, POSTs to Hermes
|
||||
3. Hermes calls Gemini Vision → extracts `{ currentFile, inferredTask, confidence }`
|
||||
4. Hermes writes to MongoDB (`engineer_states`, `ownership_map`)
|
||||
5. Hermes runs event detection across all 3 engineers
|
||||
6. `BLOCKER_DETECTED` or `DEPENDENCY_READY` event fires
|
||||
7. Hermes generates 1–2 sentence nudge via Gemini
|
||||
8. Hermes speaks nudge into LiveKit room via Gemini Live 2.5
|
||||
9. Engineers hear it through earbuds
|
||||
10. Frontend shows live nudge feed (data channel card)
|
||||
|
||||
---
|
||||
|
||||
## 2. Prize-stacking map
|
||||
## Nice-to-have (only if steps 1–10 done before hour 10)
|
||||
|
||||
| Prize | How PodMan earns it |
|
||||
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Best Gemini 3.5 ($5,000 cash)** | Realtime screen understanding via Gemini 3.5 Flash vision + PodMan's voice via Gemini Live API. Bonus: Live Translate so a multilingual pod hears PodMan in their language. |
|
||||
| **Best LiveKit (keyboards)** | LiveKit is the realtime backbone: screen-share + mic + cam tracks in, PodMan voice + data-channel cards out. It's load-bearing, not bolted on. |
|
||||
| **Best DigitalOcean (credits)** | Backend PodMan agent + LiveKit agent deployed on DigitalOcean; claim the $200 credits. |
|
||||
- `DUPLICATE_WORK` event detection
|
||||
- Ownership map cold-start demo (session 2 is visibly faster)
|
||||
- GitHub state fusion (open PRs/branches per file)
|
||||
- Polish: teammate status cards in UI, confidence indicator
|
||||
|
||||
---
|
||||
|
||||
## 3. Architecture (v1 — refined by workflow)
|
||||
## Cut immediately
|
||||
|
||||
```
|
||||
┌────────────── Engineer laptops (Chrome PWA) ──────────────┐
|
||||
│ getDisplayMedia (screen) + mic + cam │
|
||||
│ publish tracks ─────────────┐ ▲ PodMan voice │
|
||||
└──────────────────────────────┼─────────┼──────────────────┘
|
||||
│ │ data-channel cards
|
||||
┌───────▼─────────┴────────┐
|
||||
│ LiveKit room │ (one room per pod)
|
||||
└───────┬─────────▲──────────┘
|
||||
│ subscribe│ publish voice/data
|
||||
┌───────▼─────────┴──────────────────────┐
|
||||
│ BACKEND: PodMan agent (DigitalOcean) │
|
||||
│ │
|
||||
│ 1. grab frames from each screen track │
|
||||
│ 2. Gemini 3.5 vision → structured │
|
||||
│ "engineer context" (file, symbol, │
|
||||
│ feature, action) │
|
||||
│ 3. GitHub client → branches/PRs/commits │
|
||||
│ 4. COLLISION DETECTOR (fuse 2+3) │
|
||||
│ 5. continual-learning memory (Atlas + │
|
||||
│ Voyage vectors): team model + policy │
|
||||
│ 6. PodMan brain (Gemini) → intervention │
|
||||
│ 7. speak (Gemini Live/TTS) + send card │
|
||||
└──────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**The critical reconciliation:** GitHub only knows _pushed_ state. The "X is editing this and
|
||||
hasn't pushed" signal comes from **vision on the live screen** (filename in the editor tab,
|
||||
visible diff/gutter), optionally cross-checked by an _optional_ lightweight local `git status`
|
||||
reporter the engineer can run. Vision is the headline; the local reporter is a nice-to-have.
|
||||
|
||||
### Continual-learning loop
|
||||
|
||||
1. **Observe** — per-engineer context every few seconds (sampled frames, not every frame).
|
||||
2. **Store** — append observations to the team model; embed file/feature notes into Voyage
|
||||
vectors in Atlas for retrieval.
|
||||
3. **Predict** — collision detector + PodMan brain decide if/when to intervene.
|
||||
4. **Outcome** — record whether the warning was acted on / was a true positive.
|
||||
5. **Adapt** — adjust intervention thresholds, ownership attribution, and prompt context from
|
||||
outcomes → fewer false alarms, better targeting over the session. _(This is the "gets
|
||||
better the more you use it" story judges want.)_
|
||||
- VS Code extension
|
||||
- Mic transcription
|
||||
- Manual task input fields
|
||||
- Multilingual voice
|
||||
- Full task management
|
||||
- Slack / Linear integrations
|
||||
- Webcam tracks
|
||||
- Always-on raw screen surveillance (we sample every 30s by design)
|
||||
|
||||
---
|
||||
|
||||
## 4. Stack per folder
|
||||
## Team assignments
|
||||
|
||||
| Folder | Stack |
|
||||
| ----------- | ----------------------------------------------------------------------------------------------------------- |
|
||||
| `frontend/` | React + Vite + TypeScript, `livekit-client`, PWA (vite-plugin-pwa), Tailwind |
|
||||
| `backend/` | Node + TypeScript, LiveKit server SDK + agents, `@google/genai`, GitHub (Octokit or GitHub MCP), Express/ws |
|
||||
| `database/` | MongoDB Atlas (team model, observations, outcomes) + Voyage embeddings for vector recall |
|
||||
| `infra/` | DigitalOcean App Platform / Droplet, Dockerfile, app spec |
|
||||
| `shared/` | TS types: `Pod`, `EngineerContext`, `Collision`, `Intervention` |
|
||||
| Person | Owns | Hours |
|
||||
|---|---|---|
|
||||
| **Karti** | MongoDB Atlas wiring, `engineer_states` + `ownership_map` upsert logic, DO deploy, `/health` + env setup | 3–4h |
|
||||
| **Ramis** | `POST /ingest` endpoint, Gemini Vision pipeline (`frameToContext`), confidence gate, frame compression in PWA | 3–4h |
|
||||
| **Yahya** | Event detector (all 3 event types), nudge generator (Gemini text), cooldown logic, event + nudge MongoDB writes | 3–4h |
|
||||
| **Everyone** | Gemini Live 2.5 + LiveKit Agents wiring (Hermes joins room + publishes voice) — highest integration risk, do together | 2h |
|
||||
|
||||
---
|
||||
|
||||
## 5. 20-hour build order (MVP-first)
|
||||
## Build order (strictly sequential by dependency)
|
||||
|
||||
1. **Plumbing** — monorepo installs, shared types, env wiring, LiveKit token endpoint. _(Karti)_
|
||||
2. **Capture** — frontend: join pod + publish screen/mic; render PodMan card + play voice. _(Zander)_
|
||||
3. **Eyes** — backend: subscribe to a screen track, grab a frame, Gemini vision → `EngineerContext`. _(Ramis)_
|
||||
4. **Brain + collision** — fuse two engineers' contexts + GitHub state → detect same-file/feature; PodMan brain composes the intervention. _(Yahya)_
|
||||
5. **Voice + action** — PodMan speaks (Live API/TTS) into the room + "Open sync PR" via GitHub. _(Yahya + Ramis)_
|
||||
6. **Memory/continual learning** — store observations + outcomes; show the team model improving. _(Karti)_
|
||||
7. **Deploy on DO + polish demo** — everyone. Rehearse the live demo 3×.
|
||||
### Hour 0–1: Plumbing (Karti)
|
||||
- [ ] Confirm `.env` vars populated: `LIVEKIT_*`, `GEMINI_API_KEY`, `MONGODB_URI`
|
||||
- [ ] `GET /health` returns `{ ok: true }`
|
||||
- [ ] MongoDB connection established, collections initialized
|
||||
- [ ] `POST /ingest` stub returns `{ ok: true }` (no logic yet)
|
||||
|
||||
> If behind: cut webcam, cut multilingual, cut the local git reporter, **mock the QR join**,
|
||||
> hardcode the demo repo. Never cut: realtime screen→vision→PodMan-speaks loop.
|
||||
### Hour 1–3: Frame capture + Vision pipeline (Ramis)
|
||||
- [ ] PWA: `getDisplayMedia` frame capture every 30s → JPEG base64 (1280×720 max, quality 0.7)
|
||||
- [ ] PWA: `POST /ingest` with `{ engineerId, podId, screenshotBase64, capturedAt }`
|
||||
- [ ] Hermes: wire `@google/genai`, call `gemini-2.0-flash` with vision prompt
|
||||
- [ ] Hermes: parse response, apply confidence gate (< 0.6 → discard)
|
||||
- [ ] Hermes: upsert `engineer_states` in MongoDB
|
||||
|
||||
### Hour 1–3: MongoDB state layer (Karti, parallel with Ramis)
|
||||
- [ ] `engineer_states` upsert function
|
||||
- [ ] `ownership_map` upsert function (called after each `engineer_states` write)
|
||||
- [ ] `events` insert function
|
||||
- [ ] `nudges` insert function + cooldown query
|
||||
|
||||
### Hour 3–5: Event detection + nudge generation (Yahya)
|
||||
- [ ] Hermes: after each state write, fetch all `engineer_states` for the pod
|
||||
- [ ] Run Gemini event detection prompt → parse `{ event, involvedEngineers, file, reason }`
|
||||
- [ ] On non-null event: check cooldown, generate nudge message via Gemini
|
||||
- [ ] Write event + nudge to MongoDB
|
||||
- [ ] Log to console (voice wiring comes next)
|
||||
|
||||
### Hour 3–5: PWA active session UI (Zander, parallel)
|
||||
- [ ] Active session screen (post-join): "PodMan is watching" + teammate status cards
|
||||
- [ ] Data channel listener: append nudge to live feed on receive
|
||||
- [ ] Nudge feed: last 5 nudges, timestamped, engineer names highlighted
|
||||
|
||||
### Hour 5–7: Gemini Live 2.5 + LiveKit Agents voice (everyone)
|
||||
- [ ] Install LiveKit Agents SDK in backend
|
||||
- [ ] Hermes joins pod room as `podman-hermes` participant on startup
|
||||
- [ ] Wire Gemini Live 2.5 as voice provider in LiveKit Agents
|
||||
- [ ] On nudge ready: publish audio into room
|
||||
- [ ] Also publish data channel message for frontend card
|
||||
- [ ] Test: voice audible through browser audio output
|
||||
|
||||
### Hour 7–9: Integration + demo rehearsal
|
||||
- [ ] Full end-to-end test: 3 browser tabs, screen share, Hermes processes frames, nudge fires, voice heard
|
||||
- [ ] Pre-stage demo laptops: large font, clear file names, single editor window
|
||||
- [ ] Run demo script 2× — fix any timing issues
|
||||
- [ ] DO deploy (Karti) — verify `/health` live
|
||||
|
||||
### Hour 9–10: Ownership map demo (if time)
|
||||
- [ ] Load `ownership_map` on Hermes startup
|
||||
- [ ] Session 1 cold-start (3 min to first nudge) vs session 2 warm-start (< 30s)
|
||||
- [ ] Add "Session memory loaded" log visible in demo
|
||||
|
||||
### Hour 10–12: Polish + backup plan
|
||||
- [ ] Record a backup video of the demo working end-to-end
|
||||
- [ ] Rehearse 3× with real audio
|
||||
- [ ] Fallback: Hermes runs locally if DO deploy is flaky
|
||||
|
||||
---
|
||||
|
||||
## 6. Team split (4 max — must drop to 4!)
|
||||
## Open risks
|
||||
|
||||
> ⚠️ Roster has 5 (Karti, Ramis, Yahya, Zander, Shakthi). **Max team size is 4.** Decide who's
|
||||
> the official 4 before submission, or one stays unofficial/support.
|
||||
|
||||
| Person | Owns |
|
||||
| ---------- | ---------------------------------------------------------------------------------- |
|
||||
| **Karti** | Repo/infra/plumbing, shared types, memory + continual-learning store, DO deploy |
|
||||
| **Zander** | Frontend PWA: pod join, capture, PodMan card UI + voice playback |
|
||||
| **Ramis** | Backend realtime: LiveKit room subscribe + frame grab + Gemini vision pipeline |
|
||||
| **Yahya** | PodMan brain: collision detector, intervention policy, GitHub PR action, voice out |
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Gemini Vision accuracy on screens | Large font, single editor window, file name visible in tab. Confidence gate discards bad frames. |
|
||||
| Gemini Live 2.5 + LiveKit Agents wiring is unknown territory | Allocate hour 5–7 as a team. Have fallback: plain HTTP TTS → WAV → LiveKit audio track. |
|
||||
| Frame POST latency | Compress JPEG to quality 0.7, max 1280×720. Target < 500ms round trip. |
|
||||
| Event detection false positives | Cooldown (3 min between nudges). Pre-stage demo so events fire cleanly. |
|
||||
| DO deploy fails on stage | Run Hermes local. PWA already defaults to `localhost:8787`. Zero demo impact. |
|
||||
|
||||
---
|
||||
|
||||
## 7. Demo script (3 min — refined by workflow)
|
||||
## Demo script (3 min)
|
||||
|
||||
1. **(0:00)** Two laptops on screen. Both engineers "join the pod" (QR mock). PodMan greets them by voice.
|
||||
2. **(0:30)** Engineer A opens `auth.ts` and starts editing. PodMan quietly notes it (show the team model tick).
|
||||
3. **(1:00)** Engineer B opens the _same_ `auth.ts` and edits a related function — **neither has pushed.**
|
||||
4. **(1:20) MONEY MOMENT** — PodMan _interrupts by voice_: "Heads up — Karti and Yahya are both in `auth.ts`, Yahya has unpushed changes. Here's the diff. Want me to open a sync PR?" Card appears with the diff.
|
||||
5. **(1:50)** One click → PodMan opens a draft PR via GitHub (show it on github.com).
|
||||
6. **(2:20)** Show it **learned**: PodMan now knows Karti owns auth; second scenario it's faster/quieter where appropriate → "more useful the more you use it."
|
||||
7. **(2:45)** One-liner close: "PodMan — the teammate that sees what git can't."
|
||||
**(0:00)** Three laptops visible. Alice, Bob, Carol join `demo-pod`. PodMan: *"PodMan online. I see Alice, Bob, and Carol. Let's build."*
|
||||
|
||||
---
|
||||
**(0:20)** Alice opens `auth/middleware.ts` (big font, clearly visible). Hermes processes first frame. Ownership map: Alice → auth.
|
||||
|
||||
## 8. Env vars (v1 — finalized by workflow)
|
||||
**(0:45)** Bob opens `frontend/login.tsx`. Carol's terminal shows `curl: connection refused`.
|
||||
|
||||
```
|
||||
# LiveKit
|
||||
LIVEKIT_URL=
|
||||
LIVEKIT_API_KEY=
|
||||
LIVEKIT_API_SECRET=
|
||||
**(1:20) MONEY MOMENT 1 — BLOCKER_DETECTED:** PodMan speaks: *"Carol, looks like you're waiting on auth. Alice is actively building it in middleware.ts — hang tight."*
|
||||
|
||||
# Gemini
|
||||
GEMINI_API_KEY=
|
||||
GEMINI_VISION_MODEL=gemini-3.5-flash # confirm exact id from research
|
||||
GEMINI_LIVE_MODEL= # confirm from research
|
||||
**(1:50)** Alice's server starts (visible in terminal). Hermes detects transition.
|
||||
|
||||
# GitHub
|
||||
GITHUB_TOKEN=
|
||||
GITHUB_REPO=owner/name
|
||||
**(2:00) MONEY MOMENT 2 — DEPENDENCY_READY:** PodMan: *"Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate."*
|
||||
|
||||
# MongoDB Atlas + Voyage
|
||||
MONGODB_URI=
|
||||
VOYAGE_API_KEY=
|
||||
```
|
||||
**(2:20)** Optional: show session 2 cold-start vs warm-start comparison.
|
||||
|
||||
---
|
||||
|
||||
## 9. Open risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
| --------------------------------------------------- | -------------------------------------------------------------------------------- |
|
||||
| Realtime vision latency/cost | Sample ~1 frame/sec or on-change; downscale frames; cache last context |
|
||||
| LiveKit ↔ Gemini frame plumbing is the hardest part | Build & de-risk it **first** (step 3); have a screenshot-fallback path |
|
||||
| "Unpushed" detection is fuzzy | Lead with vision; optional local `git status` reporter for accuracy |
|
||||
| On-stage flakiness | Pre-stage the demo repo, rehearse 3×, have a recorded backup of the money moment |
|
||||
| Dashboard-DQ optics | Keep UI minimal; demo PodMan _acting_, not a grid |
|
||||
**(2:45)** Close: *"PodMan — the teammate that sees what Slack can't."*
|
||||
|
||||
Reference in New Issue
Block a user