docs: finalize PodMan architecture and write full integration specs
Replaces v1 plan with locked architecture: - Hermes orchestrator: POST /ingest → Gemini Vision → MongoDB → event detection → Gemini Live 2.5 voice via LiveKit Agents - Four MongoDB collections: engineer_states, ownership_map, events, nudges - Continual learning via ownership_map persisting across sessions New files: docs/idea.md, docs/gemini.md, docs/livekit.md, docs/mongodb.md, docs/digitalocean.md, docs/demo-setup.md Updated: README.md, docs/PLAN.md (12-hour build plan), database/README.md, infra/README.md, .env.example Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FFbfi4Cmb7BY75Wtne7bZn
This commit is contained in:
Ramis
+109
-145
@@ -1,188 +1,152 @@
|
||||
# PodMan — Master Plan (v1)
|
||||
# PodMan — Build Plan (12 hours)
|
||||
|
||||
> Living doc. A deeper, API-accurate v2 (exact Gemini/LiveKit SDK calls, starter code,
|
||||
> DO deploy steps) is being generated by the research workflow and will be merged in.
|
||||
> Replaces the original v1 plan. Architecture finalized. See `docs/idea.md` for concept, individual integration specs for details.
|
||||
|
||||
---
|
||||
|
||||
## 0. TL;DR
|
||||
## What we are building
|
||||
|
||||
**PodMan** is an ambient AI teammate. Engineers join a **pod**, share screen + mic, and
|
||||
PodMan watches everyone's screen in realtime, understands what each person is doing
|
||||
(Gemini vision), fuses it with the team's GitHub state, and **interrupts like Jarvis to
|
||||
prevent merge collisions before anyone pushes**, offering to open a sync PR.
|
||||
PodMan is a real-time AI team coordination agent. Engineers join a LiveKit room with earbuds. Each engineer's browser PWA captures their screen every 30s and sends it to Hermes (server-side orchestrator). Hermes uses Gemini Vision to extract structured context, detects coordination events (dependency ready, blocker, duplicate work), and speaks proactive nudges into the room via Gemini Live 2.5. MongoDB Atlas stores team state and an ownership map that persists across sessions — making PodMan faster and smarter each session.
|
||||
|
||||
- **Track:** Continual Learning
|
||||
- **Prizes targeted (stacked):** Gemini 3.5 ($5k cash), LiveKit (keyboards), DigitalOcean (credits)
|
||||
- **Hero moment:** two laptops editing the same file → PodMan _speaks up live_ and offers the fix.
|
||||
**Hero demo moment:** PodMan detects Carol is blocked waiting for Alice's auth endpoint, warns Carol, then notifies Carol and Bob the moment Alice's server starts — without anyone sending a message.
|
||||
|
||||
---
|
||||
|
||||
## 1. Why this fits the track (and stays eligible)
|
||||
## Must-have demo path
|
||||
|
||||
### Track = Continual Learning
|
||||
The minimum end-to-end flow required for a winning demo:
|
||||
|
||||
The official definition rewards systems that "continuously improve from real-world use…
|
||||
becoming more useful the more they are used with as little user intervention as possible."
|
||||
PodMan does exactly this:
|
||||
|
||||
- Builds and **continuously refines a live model of the team** — who owns which files/areas,
|
||||
what's in-flight, recurring conflict patterns, each engineer's working style.
|
||||
- **Self-improves its own intervention policy** from outcomes: did the collision it predicted
|
||||
actually happen? Did the team accept the suggested PR? It tunes its thresholds/prompts so it
|
||||
nags less and helps more over time.
|
||||
- Grows a per-team **skill/memory store** (vector memory) that makes later sessions sharper.
|
||||
|
||||
### ⚠️ Disqualification traps — and how we dodge them
|
||||
|
||||
| Risk | Mitigation |
|
||||
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **"Dashboard is the main feature" = auto-DQ** | The pod grid is _secondary_. The hero is PodMan's **proactive voice/card interventions**. In the demo we barely show the grid; we show PodMan _acting_. |
|
||||
| Repo must be **public** | Make the GitHub repo public from the start. |
|
||||
| **Only what you built** during the event | Everything in this monorepo is new, timestamped by commits. Demo narrates "built today." |
|
||||
| New work only | No pre-existing project reuse. |
|
||||
1. 3 engineers join a pod room via PWA (browser tab)
|
||||
2. PWA captures screen frame every 30s, POSTs to Hermes
|
||||
3. Hermes calls Gemini Vision → extracts `{ currentFile, inferredTask, confidence }`
|
||||
4. Hermes writes to MongoDB (`engineer_states`, `ownership_map`)
|
||||
5. Hermes runs event detection across all 3 engineers
|
||||
6. `BLOCKER_DETECTED` or `DEPENDENCY_READY` event fires
|
||||
7. Hermes generates 1–2 sentence nudge via Gemini
|
||||
8. Hermes speaks nudge into LiveKit room via Gemini Live 2.5
|
||||
9. Engineers hear it through earbuds
|
||||
10. Frontend shows live nudge feed (data channel card)
|
||||
|
||||
---
|
||||
|
||||
## 2. Prize-stacking map
|
||||
## Nice-to-have (only if steps 1–10 done before hour 10)
|
||||
|
||||
| Prize | How PodMan earns it |
|
||||
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Best Gemini 3.5 ($5,000 cash)** | Realtime screen understanding via Gemini 3.5 Flash vision + PodMan's voice via Gemini Live API. Bonus: Live Translate so a multilingual pod hears PodMan in their language. |
|
||||
| **Best LiveKit (keyboards)** | LiveKit is the realtime backbone: screen-share + mic + cam tracks in, PodMan voice + data-channel cards out. It's load-bearing, not bolted on. |
|
||||
| **Best DigitalOcean (credits)** | Backend PodMan agent + LiveKit agent deployed on DigitalOcean; claim the $200 credits. |
|
||||
- `DUPLICATE_WORK` event detection
|
||||
- Ownership map cold-start demo (session 2 is visibly faster)
|
||||
- GitHub state fusion (open PRs/branches per file)
|
||||
- Polish: teammate status cards in UI, confidence indicator
|
||||
|
||||
---
|
||||
|
||||
## 3. Architecture (v1 — refined by workflow)
|
||||
## Cut immediately
|
||||
|
||||
```
|
||||
┌────────────── Engineer laptops (Chrome PWA) ──────────────┐
|
||||
│ getDisplayMedia (screen) + mic + cam │
|
||||
│ publish tracks ─────────────┐ ▲ PodMan voice │
|
||||
└──────────────────────────────┼─────────┼──────────────────┘
|
||||
│ │ data-channel cards
|
||||
┌───────▼─────────┴────────┐
|
||||
│ LiveKit room │ (one room per pod)
|
||||
└───────┬─────────▲──────────┘
|
||||
│ subscribe│ publish voice/data
|
||||
┌───────▼─────────┴──────────────────────┐
|
||||
│ BACKEND: PodMan agent (DigitalOcean) │
|
||||
│ │
|
||||
│ 1. grab frames from each screen track │
|
||||
│ 2. Gemini 3.5 vision → structured │
|
||||
│ "engineer context" (file, symbol, │
|
||||
│ feature, action) │
|
||||
│ 3. GitHub client → branches/PRs/commits │
|
||||
│ 4. COLLISION DETECTOR (fuse 2+3) │
|
||||
│ 5. continual-learning memory (Atlas + │
|
||||
│ Voyage vectors): team model + policy │
|
||||
│ 6. PodMan brain (Gemini) → intervention │
|
||||
│ 7. speak (Gemini Live/TTS) + send card │
|
||||
└──────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**The critical reconciliation:** GitHub only knows _pushed_ state. The "X is editing this and
|
||||
hasn't pushed" signal comes from **vision on the live screen** (filename in the editor tab,
|
||||
visible diff/gutter), optionally cross-checked by an _optional_ lightweight local `git status`
|
||||
reporter the engineer can run. Vision is the headline; the local reporter is a nice-to-have.
|
||||
|
||||
### Continual-learning loop
|
||||
|
||||
1. **Observe** — per-engineer context every few seconds (sampled frames, not every frame).
|
||||
2. **Store** — append observations to the team model; embed file/feature notes into Voyage
|
||||
vectors in Atlas for retrieval.
|
||||
3. **Predict** — collision detector + PodMan brain decide if/when to intervene.
|
||||
4. **Outcome** — record whether the warning was acted on / was a true positive.
|
||||
5. **Adapt** — adjust intervention thresholds, ownership attribution, and prompt context from
|
||||
outcomes → fewer false alarms, better targeting over the session. _(This is the "gets
|
||||
better the more you use it" story judges want.)_
|
||||
- VS Code extension
|
||||
- Mic transcription
|
||||
- Manual task input fields
|
||||
- Multilingual voice
|
||||
- Full task management
|
||||
- Slack / Linear integrations
|
||||
- Webcam tracks
|
||||
- Always-on raw screen surveillance (we sample every 30s by design)
|
||||
|
||||
---
|
||||
|
||||
## 4. Stack per folder
|
||||
## Team assignments
|
||||
|
||||
| Folder | Stack |
|
||||
| ----------- | ----------------------------------------------------------------------------------------------------------- |
|
||||
| `frontend/` | React + Vite + TypeScript, `livekit-client`, PWA (vite-plugin-pwa), Tailwind |
|
||||
| `backend/` | Node + TypeScript, LiveKit server SDK + agents, `@google/genai`, GitHub (Octokit or GitHub MCP), Express/ws |
|
||||
| `database/` | MongoDB Atlas (team model, observations, outcomes) + Voyage embeddings for vector recall |
|
||||
| `infra/` | DigitalOcean App Platform / Droplet, Dockerfile, app spec |
|
||||
| `shared/` | TS types: `Pod`, `EngineerContext`, `Collision`, `Intervention` |
|
||||
| Person | Owns | Hours |
|
||||
|---|---|---|
|
||||
| **Karti** | MongoDB Atlas wiring, `engineer_states` + `ownership_map` upsert logic, DO deploy, `/health` + env setup | 3–4h |
|
||||
| **Ramis** | `POST /ingest` endpoint, Gemini Vision pipeline (`frameToContext`), confidence gate, frame compression in PWA | 3–4h |
|
||||
| **Yahya** | Event detector (all 3 event types), nudge generator (Gemini text), cooldown logic, event + nudge MongoDB writes | 3–4h |
|
||||
| **Everyone** | Gemini Live 2.5 + LiveKit Agents wiring (Hermes joins room + publishes voice) — highest integration risk, do together | 2h |
|
||||
|
||||
---
|
||||
|
||||
## 5. 20-hour build order (MVP-first)
|
||||
## Build order (strictly sequential by dependency)
|
||||
|
||||
1. **Plumbing** — monorepo installs, shared types, env wiring, LiveKit token endpoint. _(Karti)_
|
||||
2. **Capture** — frontend: join pod + publish screen/mic; render PodMan card + play voice. _(Zander)_
|
||||
3. **Eyes** — backend: subscribe to a screen track, grab a frame, Gemini vision → `EngineerContext`. _(Ramis)_
|
||||
4. **Brain + collision** — fuse two engineers' contexts + GitHub state → detect same-file/feature; PodMan brain composes the intervention. _(Yahya)_
|
||||
5. **Voice + action** — PodMan speaks (Live API/TTS) into the room + "Open sync PR" via GitHub. _(Yahya + Ramis)_
|
||||
6. **Memory/continual learning** — store observations + outcomes; show the team model improving. _(Karti)_
|
||||
7. **Deploy on DO + polish demo** — everyone. Rehearse the live demo 3×.
|
||||
### Hour 0–1: Plumbing (Karti)
|
||||
- [ ] Confirm `.env` vars populated: `LIVEKIT_*`, `GEMINI_API_KEY`, `MONGODB_URI`
|
||||
- [ ] `GET /health` returns `{ ok: true }`
|
||||
- [ ] MongoDB connection established, collections initialized
|
||||
- [ ] `POST /ingest` stub returns `{ ok: true }` (no logic yet)
|
||||
|
||||
> If behind: cut webcam, cut multilingual, cut the local git reporter, **mock the QR join**,
|
||||
> hardcode the demo repo. Never cut: realtime screen→vision→PodMan-speaks loop.
|
||||
### Hour 1–3: Frame capture + Vision pipeline (Ramis)
|
||||
- [ ] PWA: `getDisplayMedia` frame capture every 30s → JPEG base64 (1280×720 max, quality 0.7)
|
||||
- [ ] PWA: `POST /ingest` with `{ engineerId, podId, screenshotBase64, capturedAt }`
|
||||
- [ ] Hermes: wire `@google/genai`, call `gemini-2.0-flash` with vision prompt
|
||||
- [ ] Hermes: parse response, apply confidence gate (< 0.6 → discard)
|
||||
- [ ] Hermes: upsert `engineer_states` in MongoDB
|
||||
|
||||
### Hour 1–3: MongoDB state layer (Karti, parallel with Ramis)
|
||||
- [ ] `engineer_states` upsert function
|
||||
- [ ] `ownership_map` upsert function (called after each `engineer_states` write)
|
||||
- [ ] `events` insert function
|
||||
- [ ] `nudges` insert function + cooldown query
|
||||
|
||||
### Hour 3–5: Event detection + nudge generation (Yahya)
|
||||
- [ ] Hermes: after each state write, fetch all `engineer_states` for the pod
|
||||
- [ ] Run Gemini event detection prompt → parse `{ event, involvedEngineers, file, reason }`
|
||||
- [ ] On non-null event: check cooldown, generate nudge message via Gemini
|
||||
- [ ] Write event + nudge to MongoDB
|
||||
- [ ] Log to console (voice wiring comes next)
|
||||
|
||||
### Hour 3–5: PWA active session UI (Zander, parallel)
|
||||
- [ ] Active session screen (post-join): "PodMan is watching" + teammate status cards
|
||||
- [ ] Data channel listener: append nudge to live feed on receive
|
||||
- [ ] Nudge feed: last 5 nudges, timestamped, engineer names highlighted
|
||||
|
||||
### Hour 5–7: Gemini Live 2.5 + LiveKit Agents voice (everyone)
|
||||
- [ ] Install LiveKit Agents SDK in backend
|
||||
- [ ] Hermes joins pod room as `podman-hermes` participant on startup
|
||||
- [ ] Wire Gemini Live 2.5 as voice provider in LiveKit Agents
|
||||
- [ ] On nudge ready: publish audio into room
|
||||
- [ ] Also publish data channel message for frontend card
|
||||
- [ ] Test: voice audible through browser audio output
|
||||
|
||||
### Hour 7–9: Integration + demo rehearsal
|
||||
- [ ] Full end-to-end test: 3 browser tabs, screen share, Hermes processes frames, nudge fires, voice heard
|
||||
- [ ] Pre-stage demo laptops: large font, clear file names, single editor window
|
||||
- [ ] Run demo script 2× — fix any timing issues
|
||||
- [ ] DO deploy (Karti) — verify `/health` live
|
||||
|
||||
### Hour 9–10: Ownership map demo (if time)
|
||||
- [ ] Load `ownership_map` on Hermes startup
|
||||
- [ ] Session 1 cold-start (3 min to first nudge) vs session 2 warm-start (< 30s)
|
||||
- [ ] Add "Session memory loaded" log visible in demo
|
||||
|
||||
### Hour 10–12: Polish + backup plan
|
||||
- [ ] Record a backup video of the demo working end-to-end
|
||||
- [ ] Rehearse 3× with real audio
|
||||
- [ ] Fallback: Hermes runs locally if DO deploy is flaky
|
||||
|
||||
---
|
||||
|
||||
## 6. Team split (4 max — must drop to 4!)
|
||||
## Open risks
|
||||
|
||||
> ⚠️ Roster has 5 (Karti, Ramis, Yahya, Zander, Shakthi). **Max team size is 4.** Decide who's
|
||||
> the official 4 before submission, or one stays unofficial/support.
|
||||
|
||||
| Person | Owns |
|
||||
| ---------- | ---------------------------------------------------------------------------------- |
|
||||
| **Karti** | Repo/infra/plumbing, shared types, memory + continual-learning store, DO deploy |
|
||||
| **Zander** | Frontend PWA: pod join, capture, PodMan card UI + voice playback |
|
||||
| **Ramis** | Backend realtime: LiveKit room subscribe + frame grab + Gemini vision pipeline |
|
||||
| **Yahya** | PodMan brain: collision detector, intervention policy, GitHub PR action, voice out |
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Gemini Vision accuracy on screens | Large font, single editor window, file name visible in tab. Confidence gate discards bad frames. |
|
||||
| Gemini Live 2.5 + LiveKit Agents wiring is unknown territory | Allocate hour 5–7 as a team. Have fallback: plain HTTP TTS → WAV → LiveKit audio track. |
|
||||
| Frame POST latency | Compress JPEG to quality 0.7, max 1280×720. Target < 500ms round trip. |
|
||||
| Event detection false positives | Cooldown (3 min between nudges). Pre-stage demo so events fire cleanly. |
|
||||
| DO deploy fails on stage | Run Hermes local. PWA already defaults to `localhost:8787`. Zero demo impact. |
|
||||
|
||||
---
|
||||
|
||||
## 7. Demo script (3 min — refined by workflow)
|
||||
## Demo script (3 min)
|
||||
|
||||
1. **(0:00)** Two laptops on screen. Both engineers "join the pod" (QR mock). PodMan greets them by voice.
|
||||
2. **(0:30)** Engineer A opens `auth.ts` and starts editing. PodMan quietly notes it (show the team model tick).
|
||||
3. **(1:00)** Engineer B opens the _same_ `auth.ts` and edits a related function — **neither has pushed.**
|
||||
4. **(1:20) MONEY MOMENT** — PodMan _interrupts by voice_: "Heads up — Karti and Yahya are both in `auth.ts`, Yahya has unpushed changes. Here's the diff. Want me to open a sync PR?" Card appears with the diff.
|
||||
5. **(1:50)** One click → PodMan opens a draft PR via GitHub (show it on github.com).
|
||||
6. **(2:20)** Show it **learned**: PodMan now knows Karti owns auth; second scenario it's faster/quieter where appropriate → "more useful the more you use it."
|
||||
7. **(2:45)** One-liner close: "PodMan — the teammate that sees what git can't."
|
||||
**(0:00)** Three laptops visible. Alice, Bob, Carol join `demo-pod`. PodMan: *"PodMan online. I see Alice, Bob, and Carol. Let's build."*
|
||||
|
||||
---
|
||||
**(0:20)** Alice opens `auth/middleware.ts` (big font, clearly visible). Hermes processes first frame. Ownership map: Alice → auth.
|
||||
|
||||
## 8. Env vars (v1 — finalized by workflow)
|
||||
**(0:45)** Bob opens `frontend/login.tsx`. Carol's terminal shows `curl: connection refused`.
|
||||
|
||||
```
|
||||
# LiveKit
|
||||
LIVEKIT_URL=
|
||||
LIVEKIT_API_KEY=
|
||||
LIVEKIT_API_SECRET=
|
||||
**(1:20) MONEY MOMENT 1 — BLOCKER_DETECTED:** PodMan speaks: *"Carol, looks like you're waiting on auth. Alice is actively building it in middleware.ts — hang tight."*
|
||||
|
||||
# Gemini
|
||||
GEMINI_API_KEY=
|
||||
GEMINI_VISION_MODEL=gemini-3.5-flash # confirm exact id from research
|
||||
GEMINI_LIVE_MODEL= # confirm from research
|
||||
**(1:50)** Alice's server starts (visible in terminal). Hermes detects transition.
|
||||
|
||||
# GitHub
|
||||
GITHUB_TOKEN=
|
||||
GITHUB_REPO=owner/name
|
||||
**(2:00) MONEY MOMENT 2 — DEPENDENCY_READY:** PodMan: *"Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate."*
|
||||
|
||||
# MongoDB Atlas + Voyage
|
||||
MONGODB_URI=
|
||||
VOYAGE_API_KEY=
|
||||
```
|
||||
**(2:20)** Optional: show session 2 cold-start vs warm-start comparison.
|
||||
|
||||
---
|
||||
|
||||
## 9. Open risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
| --------------------------------------------------- | -------------------------------------------------------------------------------- |
|
||||
| Realtime vision latency/cost | Sample ~1 frame/sec or on-change; downscale frames; cache last context |
|
||||
| LiveKit ↔ Gemini frame plumbing is the hardest part | Build & de-risk it **first** (step 3); have a screenshot-fallback path |
|
||||
| "Unpushed" detection is fuzzy | Lead with vision; optional local `git status` reporter for accuracy |
|
||||
| On-stage flakiness | Pre-stage the demo repo, rehearse 3×, have a recorded backup of the money moment |
|
||||
| Dashboard-DQ optics | Keep UI minimal; demo PodMan _acting_, not a grid |
|
||||
**(2:45)** Close: *"PodMan — the teammate that sees what Slack can't."*
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
# Demo Setup
|
||||
|
||||
Pre-stage checklist for the 3-minute live demo. Do this on all 3 laptops before walking on stage.
|
||||
|
||||
---
|
||||
|
||||
## Before demo day
|
||||
|
||||
- [ ] `demo-pod` room created in LiveKit Cloud dashboard
|
||||
- [ ] Hermes deployed on DO (or confirmed running locally as fallback)
|
||||
- [ ] MongoDB Atlas cluster running, `MONGODB_URI` set in Hermes env
|
||||
- [ ] All `.env` vars populated and verified via `GET /health` returning `{ ok: true }`
|
||||
- [ ] Record a backup video of the full demo working end-to-end
|
||||
- [ ] Rehearse the demo script 3× with real audio
|
||||
|
||||
---
|
||||
|
||||
## Laptop setup (all 3 machines)
|
||||
|
||||
### Editor settings
|
||||
- Font size: **18pt or larger** — Gemini Vision must read file names and code
|
||||
- Single editor window — no split panes, no overlapping terminals
|
||||
- File tab visible with full file name shown (not truncated)
|
||||
- Light or dark theme is fine — avoid low-contrast themes
|
||||
|
||||
### Browser
|
||||
- Chrome (best `getDisplayMedia` support)
|
||||
- PWA tab open and joined to `demo-pod`
|
||||
- Earbuds / headphones plugged in and tested
|
||||
- Volume: medium — PodMan voice should be clearly audible but not startle
|
||||
|
||||
### Screen layout
|
||||
- Editor takes 2/3 of screen
|
||||
- Terminal takes bottom 1/3 (always visible)
|
||||
- No other windows on top
|
||||
|
||||
---
|
||||
|
||||
## Demo file setup
|
||||
|
||||
Pre-create these files in the demo repo before the demo:
|
||||
|
||||
**Alice's machine:**
|
||||
- Open `auth/middleware.ts` — has visible function stubs
|
||||
- Terminal shows nothing running initially, then `Server running on :3001` at the right moment
|
||||
|
||||
**Bob's machine:**
|
||||
- Open `frontend/login.tsx` — has visible form component code
|
||||
- Terminal idle
|
||||
|
||||
**Carol's machine:**
|
||||
- Open `frontend/integration.ts` or similar
|
||||
- Terminal shows: `curl http://localhost:3001/auth` → `curl: (7) Failed to connect`
|
||||
|
||||
---
|
||||
|
||||
## Demo script timing
|
||||
|
||||
| Time | Action | Who |
|
||||
|---|---|---|
|
||||
| 0:00 | All three join `demo-pod` | All |
|
||||
| 0:05 | PodMan greets by voice | Hermes auto |
|
||||
| 0:20 | Alice opens `auth/middleware.ts`, starts typing | Alice |
|
||||
| 0:45 | Bob opens `frontend/login.tsx` | Bob |
|
||||
| 0:50 | Carol runs `curl` command, sees error | Carol |
|
||||
| ~1:20 | BLOCKER_DETECTED nudge fires | Hermes auto |
|
||||
| 1:50 | Alice starts her server (`node server.js`) | Alice |
|
||||
| ~2:00 | DEPENDENCY_READY nudge fires | Hermes auto |
|
||||
| 2:20 | Optional: show session 2 ownership warm-start | Presenter |
|
||||
| 2:45 | Close | Presenter |
|
||||
|
||||
---
|
||||
|
||||
## Gemini Vision reliability tips
|
||||
|
||||
- Keep font at 18pt+ throughout the demo — do not zoom out
|
||||
- Avoid opening file picker dialogs or overlapping modals during the demo
|
||||
- File names in editor tabs must be fully visible (not `auth/middle...`)
|
||||
- If Hermes logs show `confidence < 0.6` frames: bump font size, ensure file tab is clear
|
||||
- Terminal output must be on a single line — avoid long stack traces during demo
|
||||
|
||||
---
|
||||
|
||||
## Cooldown note
|
||||
|
||||
Hermes has a 3-minute cooldown between nudges per pod. For the demo, if you need to trigger a second event quickly:
|
||||
|
||||
Option 1: restart Hermes between the two demo scenarios (resets cooldown state)
|
||||
Option 2: set `NUDGE_COOLDOWN_MS=0` via env var during demo (add this override to Hermes)
|
||||
|
||||
---
|
||||
|
||||
## Fallback plan
|
||||
|
||||
If any system fails on stage:
|
||||
|
||||
1. **Hermes unreachable:** switch to local (`pnpm --filter backend dev`) — PWA auto-falls back to `localhost:8787`
|
||||
2. **Gemini Vision low confidence:** presenter narrates what PodMan "saw" while playing the backup video
|
||||
3. **LiveKit audio not working:** play backup video — show the nudge text cards on screen instead
|
||||
4. **Full system failure:** play the backup recording, narrate the demo live
|
||||
|
||||
Always have the backup video on a separate device, not the same laptop running Hermes.
|
||||
@@ -0,0 +1,121 @@
|
||||
# DigitalOcean Deployment Spec
|
||||
|
||||
PodMan backend (Hermes) runs on DigitalOcean. Frontend is served as a static site. Both are deployed from the same monorepo.
|
||||
|
||||
---
|
||||
|
||||
## Services
|
||||
|
||||
### 1. Hermes — Backend API + LiveKit Agent
|
||||
|
||||
**Type:** DigitalOcean App Platform — Web Service (or Droplet if App Platform has issues)
|
||||
|
||||
**Runtime:** Node.js 20
|
||||
|
||||
**Build command:** `pnpm --filter backend build`
|
||||
|
||||
**Run command:** `node dist/index.js`
|
||||
|
||||
**Port:** `8787` (set via `PORT` env var)
|
||||
|
||||
**Resources:** Basic ($12/mo) — 1 vCPU, 1GB RAM. Sufficient for hackathon load.
|
||||
|
||||
---
|
||||
|
||||
### 2. Frontend PWA — Static Site
|
||||
|
||||
**Type:** DigitalOcean App Platform — Static Site
|
||||
|
||||
**Build command:** `pnpm --filter frontend build`
|
||||
|
||||
**Output directory:** `frontend/dist`
|
||||
|
||||
**Routes:** SPA — all routes → `index.html`
|
||||
|
||||
---
|
||||
|
||||
## Environment variables (set in App Platform dashboard)
|
||||
|
||||
```
|
||||
# LiveKit
|
||||
LIVEKIT_URL=wss://your-livekit-server.livekit.cloud
|
||||
LIVEKIT_API_KEY=
|
||||
LIVEKIT_API_SECRET=
|
||||
|
||||
# Gemini
|
||||
GEMINI_API_KEY=
|
||||
GEMINI_VISION_MODEL=gemini-2.0-flash
|
||||
GEMINI_LIVE_MODEL=gemini-live-2.5-flash
|
||||
|
||||
# MongoDB Atlas
|
||||
MONGODB_URI=mongodb+srv://...
|
||||
|
||||
# Server
|
||||
PORT=8787
|
||||
|
||||
# Frontend (Vite — set in App Platform as static site env vars)
|
||||
VITE_BACKEND_URL=https://your-hermes-app.ondigitalocean.app
|
||||
VITE_LIVEKIT_URL=wss://your-livekit-server.livekit.cloud
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dockerfile (backend)
|
||||
|
||||
Located at `infra/Dockerfile`. Already scaffolded. Ensure it:
|
||||
|
||||
1. Uses `node:20-slim`
|
||||
2. Installs `pnpm`
|
||||
3. Copies workspace root + backend package
|
||||
4. Runs `pnpm install --frozen-lockfile`
|
||||
5. Runs `pnpm --filter backend build`
|
||||
6. `CMD ["node", "backend/dist/index.js"]`
|
||||
|
||||
---
|
||||
|
||||
## App Platform spec (`infra/app.yaml`)
|
||||
|
||||
Already scaffolded. Key fields to confirm before deploy:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
- name: hermes
|
||||
source_dir: /
|
||||
dockerfile_path: infra/Dockerfile
|
||||
http_port: 8787
|
||||
instance_size_slug: basic-xxs
|
||||
envs:
|
||||
- key: LIVEKIT_URL
|
||||
scope: RUN_TIME
|
||||
value: ${LIVEKIT_URL}
|
||||
# ... other vars
|
||||
|
||||
static_sites:
|
||||
- name: frontend
|
||||
source_dir: frontend
|
||||
build_command: pnpm build
|
||||
output_dir: dist
|
||||
index_document: index.html
|
||||
error_document: index.html
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deploy checklist
|
||||
|
||||
- [ ] MongoDB Atlas IP allowlist: add DigitalOcean outbound IPs (or allow all: `0.0.0.0/0` for hackathon)
|
||||
- [ ] LiveKit Cloud: confirm `LIVEKIT_URL` points to your LiveKit Cloud project
|
||||
- [ ] Gemini API key has quota for `gemini-2.0-flash` + `gemini-live-2.5-flash`
|
||||
- [ ] `VITE_BACKEND_URL` set to the deployed Hermes URL (not localhost)
|
||||
- [ ] Test `GET /health` returns `{ ok: true }` after deploy
|
||||
|
||||
---
|
||||
|
||||
## Fallback plan (if App Platform deploy fails on stage)
|
||||
|
||||
Run Hermes locally:
|
||||
```bash
|
||||
cd backend && pnpm dev
|
||||
```
|
||||
|
||||
Frontend already points to `http://localhost:8787` by default via `VITE_BACKEND_URL` fallback. Demo works fully local — no DigitalOcean dependency for the live demo itself.
|
||||
+134
@@ -0,0 +1,134 @@
|
||||
# Gemini Integration Spec
|
||||
|
||||
PodMan uses Gemini for two distinct jobs: **vision** (understanding screens) and **voice** (speaking nudges).
|
||||
|
||||
---
|
||||
|
||||
## 1. Vision — Screen Understanding
|
||||
|
||||
**Model:** `gemini-2.0-flash` (fast, cheap, strong multimodal)
|
||||
|
||||
**Trigger:** every 30s per active engineer, when Hermes receives a `POST /ingest` frame
|
||||
|
||||
**Input:** base64-encoded JPEG, max 1280×720, ~50–80KB after compression
|
||||
|
||||
**Prompt:**
|
||||
|
||||
```
|
||||
You are analyzing a software engineer's screen during a coding session.
|
||||
Extract the following JSON. If you cannot determine a field with confidence above 0.7, set it to null.
|
||||
|
||||
{
|
||||
"currentFile": "string | null", // active file visible in editor tab or title bar
|
||||
"inferredTask": "string | null", // 1 sentence: what the engineer appears to be doing
|
||||
"terminalVisible": true | false, // is a terminal or CLI panel visible
|
||||
"recentTerminalOutput": "string | null", // last meaningful line of terminal output if visible
|
||||
"confidence": 0.0–1.0 // your overall confidence in this extraction
|
||||
}
|
||||
|
||||
Respond with valid JSON only. No explanation. No markdown.
|
||||
```
|
||||
|
||||
**Confidence gate:** if `confidence < 0.6`, Hermes discards the frame — no state update, no event detection triggered.
|
||||
|
||||
**Rate limit:** 1 call per engineer per 30s. With 3 engineers = 6 calls/min ≈ $0.002/min at Flash pricing.
|
||||
|
||||
**Demo setup requirement:** editors must have large font (18pt+), single window, file name clearly visible in tab. This is the primary reliability lever.
|
||||
|
||||
---
|
||||
|
||||
## 2. Event Detection — Coordination Awareness
|
||||
|
||||
**Model:** `gemini-2.0-flash` (text only, fast)
|
||||
|
||||
**Trigger:** after every successful state write to MongoDB, Hermes runs event detection over all active engineer contexts.
|
||||
|
||||
**Input:** JSON snapshot of all engineers' current states + ownership map
|
||||
|
||||
**Prompt:**
|
||||
|
||||
```
|
||||
You are a team coordination agent. Below is the current state of each engineer on the team.
|
||||
|
||||
Engineer states:
|
||||
{{engineerStates}}
|
||||
|
||||
Ownership map (who owns which files):
|
||||
{{ownershipMap}}
|
||||
|
||||
Detect if any of these coordination events are occurring:
|
||||
- DEPENDENCY_READY: an engineer who was blocked or waiting now has what they need because another engineer completed relevant work
|
||||
- BLOCKER_DETECTED: an engineer appears stuck (same file, error in terminal, no progress) and another teammate could help
|
||||
- DUPLICATE_WORK: two or more engineers are working on the same file simultaneously
|
||||
|
||||
If an event is detected, respond with:
|
||||
{
|
||||
"event": "DEPENDENCY_READY" | "BLOCKER_DETECTED" | "DUPLICATE_WORK" | null,
|
||||
"involvedEngineers": ["engineerId", ...],
|
||||
"file": "string | null",
|
||||
"reason": "1 sentence explanation"
|
||||
}
|
||||
|
||||
If no event, respond with { "event": null }.
|
||||
Respond with valid JSON only.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Nudge Generation — Voice Message
|
||||
|
||||
**Model:** `gemini-2.0-flash` (text only)
|
||||
|
||||
**Trigger:** when event detection returns a non-null event
|
||||
|
||||
**Input:** event type + engineer names + file + reason
|
||||
|
||||
**Prompt:**
|
||||
|
||||
```
|
||||
You are PodMan, a friendly AI teammate. Generate a short spoken message (1–2 sentences max) to notify the team about this coordination event.
|
||||
|
||||
Event: {{eventType}}
|
||||
Engineers involved: {{engineerNames}}
|
||||
File: {{file}}
|
||||
Context: {{reason}}
|
||||
|
||||
Rules:
|
||||
- Use first names only
|
||||
- Be direct and specific
|
||||
- Do not use filler words
|
||||
- Sound natural when spoken aloud
|
||||
- Do not start with "Hey" or "Attention"
|
||||
|
||||
Respond with the message text only.
|
||||
```
|
||||
|
||||
**Example output:**
|
||||
> "Carol — Alice just got the auth endpoint running. You're clear to integrate."
|
||||
|
||||
---
|
||||
|
||||
## 4. Voice Output — Gemini Live 2.5 via LiveKit
|
||||
|
||||
**Model:** `gemini-live-2.5-flash` (confirm exact ID from LiveKit Agents docs)
|
||||
|
||||
**Integration:** LiveKit Agents framework — Hermes runs as a LiveKit Agent with Gemini Live 2.5 as the voice provider
|
||||
|
||||
**Flow:**
|
||||
1. Nudge message text generated (step 3)
|
||||
2. Hermes passes text to Gemini Live via LiveKit Agents
|
||||
3. Gemini Live streams audio back in real-time
|
||||
4. LiveKit publishes audio into the pod room
|
||||
5. All participants hear it through their audio output
|
||||
|
||||
**Why Gemini Live (not plain TTS):**
|
||||
- Streams audio directly — no intermediate WAV file conversion
|
||||
- Latency ~300–500ms from text to first audio packet
|
||||
- Natural-sounding voice
|
||||
- Strong prize story: Gemini Live 2.5 is the headline model
|
||||
|
||||
---
|
||||
|
||||
## Cooldown
|
||||
|
||||
Per-pod cooldown of **3 minutes** between nudges. Prevents spam if multiple events fire simultaneously. Implemented in Hermes, not in Gemini.
|
||||
@@ -0,0 +1,93 @@
|
||||
# PodMan — Idea
|
||||
|
||||
## One-line value prop
|
||||
|
||||
PodMan is a real-time AI team coordination agent that watches consented work signals, maintains live project memory, and proactively notifies collaborators when dependencies, blockers, or handoffs emerge — before anyone has to ask.
|
||||
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
Teams working on the same project lose time because progress is fragmented across people, editors, terminals, and half-finished messages. Coordination gaps — a completed endpoint, a resolved blocker, two engineers duplicating work — are discovered too late, causing idle time, broken handoffs, and missed dependencies.
|
||||
|
||||
Slack doesn't help. Stand-ups are too slow. GitHub only knows pushed state.
|
||||
|
||||
---
|
||||
|
||||
## Solution
|
||||
|
||||
PodMan is an ambient AI agent that:
|
||||
|
||||
1. Watches each engineer's screen via periodic snapshots (consented, browser-native)
|
||||
2. Extracts structured context using Gemini Vision — current file, inferred task, terminal state
|
||||
3. Maintains a shared live model of the team in MongoDB Atlas — who is doing what, who owns which files
|
||||
4. Detects coordination events: dependency ready, blocker detected, duplicate work
|
||||
5. Speaks proactively into the team's LiveKit room — engineers hear PodMan through their earbuds without leaving their editor
|
||||
|
||||
**The AI's job is not to chat. It is to notice what teammates miss and say so, exactly when it matters.**
|
||||
|
||||
---
|
||||
|
||||
## Target user
|
||||
|
||||
Small software teams: hackathon squads, startup engineering teams, student dev teams collaborating in real time on a shared codebase.
|
||||
|
||||
---
|
||||
|
||||
## Core AI job
|
||||
|
||||
- Maintain per-person live context (file, task, terminal)
|
||||
- Infer shared project state (who owns what, what's blocked, what's ready)
|
||||
- Detect 3 coordination event types:
|
||||
- `DEPENDENCY_READY` — engineer A was waiting on work engineer B just completed
|
||||
- `BLOCKER_DETECTED` — engineer appears stuck; another teammate can unblock
|
||||
- `DUPLICATE_WORK` — 2+ engineers working on the same file simultaneously
|
||||
- Generate a 1–2 sentence proactive voice nudge
|
||||
- Deliver it into the LiveKit room via Gemini Live 2.5
|
||||
|
||||
---
|
||||
|
||||
## How it fits the Continual Learning track
|
||||
|
||||
PodMan builds an **ownership map** in MongoDB that persists across sessions:
|
||||
|
||||
- Session 1: PodMan needs 3–5 minutes of screen observations to infer who owns what
|
||||
- Session 2+: PodMan already knows. First nudge fires in under 30 seconds.
|
||||
|
||||
The system gets demonstrably more useful the more it is used, with no user configuration required. That is the track definition met exactly.
|
||||
|
||||
---
|
||||
|
||||
## Architecture (one paragraph)
|
||||
|
||||
Each engineer opens a browser PWA on their laptop. The PWA captures a screen frame every 30 seconds via `getDisplayMedia` and POSTs it to Hermes, the server-side orchestrator running on DigitalOcean. Hermes calls Gemini Vision to extract structured context, writes it to MongoDB Atlas, updates the ownership map, and runs event detection across all active engineers. When a coordination event fires, Hermes generates a short spoken message and publishes it as audio into the team's LiveKit room via Gemini Live 2.5. Engineers hear PodMan through their earbuds. No Slack. No tab switching. No interruption to the editor flow.
|
||||
|
||||
---
|
||||
|
||||
## Demo wow moment
|
||||
|
||||
> Alice is building the auth endpoint. Carol is visibly blocked — her terminal shows `connection refused`. PodMan detects the blocker and says aloud: "Carol, looks like you're waiting on auth. Alice is actively building it — hang tight."
|
||||
>
|
||||
> Two minutes later, Alice's server starts. PodMan says: "Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate."
|
||||
>
|
||||
> Nobody asked. Nobody pinged anyone on Slack. PodMan just knew.
|
||||
|
||||
---
|
||||
|
||||
## What PodMan is NOT
|
||||
|
||||
- Not a chat interface
|
||||
- Not a dashboard product
|
||||
- Not raw surveillance — engineers consent by joining the room and sharing their screen
|
||||
- Not a task manager
|
||||
- Not a GitHub integration (v1)
|
||||
|
||||
---
|
||||
|
||||
## Prize alignment
|
||||
|
||||
| Prize | How PodMan earns it |
|
||||
|---|---|
|
||||
| Best Gemini 3.5 / 2.5 | Gemini Vision for screen understanding + Gemini Live 2.5 for voice output |
|
||||
| Best LiveKit | LiveKit is the real-time backbone for room presence and voice delivery — load-bearing, not decorative |
|
||||
| Best DigitalOcean | Hermes deployed on DigitalOcean App Platform; MongoDB Atlas on DO-adjacent infrastructure |
|
||||
@@ -0,0 +1,98 @@
|
||||
# LiveKit Integration Spec
|
||||
|
||||
LiveKit is the real-time backbone for PodMan. It handles room presence and voice delivery. It is load-bearing — not decorative.
|
||||
|
||||
---
|
||||
|
||||
## Room structure
|
||||
|
||||
- One LiveKit room per project pod: `room = podId`
|
||||
- Engineers join as named participants (e.g. `alice`, `bob`)
|
||||
- Hermes joins as `podman-hermes`
|
||||
- All participants stay connected for the duration of the session
|
||||
|
||||
---
|
||||
|
||||
## Engineer side (PWA)
|
||||
|
||||
**Joining:**
|
||||
1. PWA calls `POST /pods/:podId/token` → receives `{ token, url }`
|
||||
2. LiveKit client connects to the room with the token
|
||||
3. PWA publishes screen track via `getDisplayMedia` (used client-side for frame capture — Hermes does NOT subscribe to this track)
|
||||
4. PWA sets mic enabled for ambient presence
|
||||
|
||||
**Receiving:**
|
||||
- LiveKit client automatically receives Hermes audio track
|
||||
- No special subscription needed — LiveKit delivers audio to all participants
|
||||
- PWA also listens for data channel messages from Hermes for UI card updates
|
||||
|
||||
**Data channel listener (PWA):**
|
||||
```ts
|
||||
room.on(RoomEvent.DataReceived, (payload, participant) => {
|
||||
if (participant?.identity !== 'podman-hermes') return;
|
||||
const nudge = JSON.parse(new TextDecoder().decode(payload));
|
||||
// nudge: { type, message, involvedEngineers, file, sentAt }
|
||||
appendNudgeToFeed(nudge);
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Hermes side (LiveKit Agent)
|
||||
|
||||
**Framework:** LiveKit Agents (Node.js)
|
||||
|
||||
**Startup:**
|
||||
1. Hermes mints its own token via the same `createPodToken` function with `identity: 'podman-hermes'`
|
||||
2. Connects to the room on pod creation / first engineer joining
|
||||
3. Registers as a LiveKit Agent with Gemini Live 2.5 as voice provider
|
||||
|
||||
**Voice delivery:**
|
||||
1. Nudge message text is ready (from Gemini text generation)
|
||||
2. Hermes passes text to Gemini Live 2.5 via LiveKit Agents voice pipeline
|
||||
3. Audio streams into the room in real-time
|
||||
4. All participants hear it
|
||||
|
||||
**Data channel message (sent alongside audio):**
|
||||
```ts
|
||||
const nudge = {
|
||||
type: 'DEPENDENCY_READY' | 'BLOCKER_DETECTED' | 'DUPLICATE_WORK',
|
||||
message: string, // the spoken text
|
||||
involvedEngineers: string[],
|
||||
file: string | null,
|
||||
sentAt: string, // ISO timestamp
|
||||
};
|
||||
room.localParticipant.publishData(
|
||||
new TextEncoder().encode(JSON.stringify(nudge)),
|
||||
{ reliable: true }
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Token endpoint
|
||||
|
||||
Already implemented at `POST /pods/:podId/token`.
|
||||
|
||||
Hermes uses the same endpoint. Grants:
|
||||
- `roomJoin: true`
|
||||
- `canPublish: true` (for audio track)
|
||||
- `canPublishData: true` (for data channel)
|
||||
- `canSubscribe: true`
|
||||
|
||||
---
|
||||
|
||||
## Gemini Live 2.5 model
|
||||
|
||||
- Model ID: `gemini-live-2.5-flash` — confirm exact ID from LiveKit Agents + Gemini docs at build time
|
||||
- LiveKit Agents has native Gemini Live integration — no manual audio encoding needed
|
||||
- Hermes passes text string → Agents handles streaming audio publication
|
||||
|
||||
---
|
||||
|
||||
## What LiveKit does NOT do in PodMan
|
||||
|
||||
- Hermes does NOT subscribe to engineer screen tracks (frame capture happens client-side)
|
||||
- No video tracks from Hermes
|
||||
- No mic transcription (not needed for v1)
|
||||
- No SFU mixing — standard room behavior is sufficient
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
# MongoDB Atlas Integration Spec
|
||||
|
||||
MongoDB Atlas is PodMan's shared memory. It stores live engineer state, the ownership map that enables continual learning, coordination events, and nudge history.
|
||||
|
||||
---
|
||||
|
||||
## Collections
|
||||
|
||||
### `engineer_states`
|
||||
|
||||
Latest context per engineer. Upserted on every successful `/ingest` call.
|
||||
|
||||
```ts
|
||||
{
|
||||
_id: string, // engineerId (stable across sessions)
|
||||
podId: string,
|
||||
name: string, // display name
|
||||
currentFile: string | null, // from Gemini Vision
|
||||
inferredTask: string | null, // from Gemini Vision
|
||||
terminalVisible: boolean,
|
||||
recentTerminalOutput: string | null,
|
||||
confidence: number, // last frame confidence (0–1)
|
||||
updatedAt: Date
|
||||
}
|
||||
```
|
||||
|
||||
**Index:** `{ podId: 1, updatedAt: -1 }`
|
||||
|
||||
**Usage:** Hermes reads all documents for a given `podId` after each update to run event detection across the full team.
|
||||
|
||||
---
|
||||
|
||||
### `ownership_map`
|
||||
|
||||
Tracks who works on which files. Built up over the session. **Persists across sessions** — this is the continual learning artifact.
|
||||
|
||||
```ts
|
||||
{
|
||||
_id: string, // `${podId}:${file}`
|
||||
podId: string,
|
||||
file: string,
|
||||
primaryOwner: string, // engineerId with most recent activity on this file
|
||||
contributors: string[], // all engineerIds observed on this file
|
||||
observationCount: number, // total frames where this file was seen
|
||||
lastSeenAt: Date
|
||||
}
|
||||
```
|
||||
|
||||
**Index:** `{ podId: 1, file: 1 }` (unique)
|
||||
|
||||
**Upsert logic:**
|
||||
- On each context update where `currentFile` is non-null:
|
||||
- Increment `observationCount`
|
||||
- Update `primaryOwner` to the engineer with the most recent `lastSeenAt` on this file
|
||||
- Add engineerId to `contributors` if not present
|
||||
- Update `lastSeenAt`
|
||||
|
||||
**Continual learning:** Hermes loads this collection on startup for the pod. If history exists, it pre-populates the in-memory ownership cache before the first frame arrives.
|
||||
|
||||
---
|
||||
|
||||
### `events`
|
||||
|
||||
Every coordination event detected by Hermes.
|
||||
|
||||
```ts
|
||||
{
|
||||
_id: ObjectId,
|
||||
podId: string,
|
||||
type: 'DEPENDENCY_READY' | 'BLOCKER_DETECTED' | 'DUPLICATE_WORK',
|
||||
involvedEngineers: string[],
|
||||
file: string | null,
|
||||
reason: string, // 1-sentence explanation from Gemini
|
||||
nudgeSent: boolean, // false if suppressed by cooldown
|
||||
detectedAt: Date
|
||||
}
|
||||
```
|
||||
|
||||
**Index:** `{ podId: 1, detectedAt: -1 }`
|
||||
|
||||
---
|
||||
|
||||
### `nudges`
|
||||
|
||||
Every voice nudge sent to the room.
|
||||
|
||||
```ts
|
||||
{
|
||||
_id: ObjectId,
|
||||
podId: string,
|
||||
eventId: ObjectId, // ref to events collection
|
||||
targetEngineers: string[],
|
||||
message: string, // the spoken text
|
||||
sentAt: Date
|
||||
}
|
||||
```
|
||||
|
||||
**Index:** `{ podId: 1, sentAt: -1 }`
|
||||
|
||||
**Cooldown check:** before sending a nudge, Hermes queries this collection for any nudge in the last 3 minutes for the same `podId`. If found, suppresses the new nudge and marks the event as `nudgeSent: false`.
|
||||
|
||||
---
|
||||
|
||||
## Hermes startup sequence
|
||||
|
||||
```
|
||||
1. Connect to Atlas using MONGODB_URI
|
||||
2. Load ownership_map for this podId
|
||||
3. Build in-memory cache: Map<file, { primaryOwner, contributors }>
|
||||
4. Begin accepting /ingest requests
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Atlas configuration
|
||||
|
||||
- **Cluster tier:** M0 (free) is sufficient for hackathon scale
|
||||
- **Region:** same as DigitalOcean deployment (e.g. NYC1)
|
||||
- **Auth:** connection string in `MONGODB_URI` env var
|
||||
- **Collections created automatically** on first write (no schema migration needed)
|
||||
|
||||
---
|
||||
|
||||
## What MongoDB does NOT store
|
||||
|
||||
- Raw screenshot frames (too large — frames are processed in-memory by Hermes and discarded)
|
||||
- Full Gemini response objects (only extracted fields are stored)
|
||||
- Session recordings
|
||||
@@ -0,0 +1,129 @@
|
||||
---
|
||||
name: podman-design
|
||||
description: Full system design for PodMan — real-time AI team coordination agent using Gemini Vision, Gemini Live 2.5, LiveKit, and MongoDB Atlas
|
||||
metadata:
|
||||
type: project
|
||||
---
|
||||
|
||||
# PodMan — System Design
|
||||
|
||||
## Concept
|
||||
|
||||
PodMan is a real-time AI team coordination agent for software teams. Engineers join a LiveKit room with earbuds. Each engineer's browser PWA captures their screen every 30s and sends it to Hermes (server-side orchestrator on DigitalOcean). Hermes uses Gemini Vision to extract structured context per engineer, detects coordination events, and speaks proactive nudges into the room via Gemini Live 2.5 through LiveKit. MongoDB Atlas stores team state and an ownership map that persists across sessions.
|
||||
|
||||
**Track:** Continual Learning — the ownership map makes PodMan faster and smarter each session with no user configuration.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────── Engineer laptop (Browser PWA) ──────────────────┐
|
||||
│ getDisplayMedia → frame every 30s │
|
||||
│ HTTP POST /ingest → { screenshot, engineerId, podId } │
|
||||
│ LiveKit room joined → receives voice audio from Hermes │
|
||||
│ Earbuds: hears PodMan proactive nudges │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│ POST /ingest
|
||||
▼
|
||||
┌────────────────── HERMES (DigitalOcean) ─────────────────────────┐
|
||||
│ 1. Receive frame → Gemini Vision → EngineerContext │
|
||||
│ 2. Write context to MongoDB (per-user state) │
|
||||
│ 3. Update ownership map (file → engineer) │
|
||||
│ 4. Run event detector over all active contexts │
|
||||
│ 5. If event detected → Gemini generates voice message │
|
||||
│ 6. Push audio into LiveKit room via Gemini Live 2.5 │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│ read/write
|
||||
▼
|
||||
MongoDB Atlas
|
||||
(engineer_states, ownership_map,
|
||||
events, nudges)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Components
|
||||
|
||||
### PWA (local agent)
|
||||
- Joins LiveKit room via existing `joinPod` flow
|
||||
- Captures frame every 30s via `getDisplayMedia`, compresses to JPEG (1280×720, quality 0.7)
|
||||
- POSTs `{ engineerId, podId, screenshotBase64, capturedAt }` to `POST /ingest`
|
||||
- Receives Hermes audio track (automatic via LiveKit)
|
||||
- Listens for data channel messages → renders nudge feed
|
||||
- Two screens: join screen (built), active session screen (to build)
|
||||
|
||||
### Hermes (orchestrator)
|
||||
- Express server + LiveKit Agent on DigitalOcean
|
||||
- `POST /ingest`: receives frame, queues for vision
|
||||
- Vision pipeline: Gemini 2.0 Flash → `EngineerContext`
|
||||
- Confidence gate: discard frames with confidence < 0.6
|
||||
- State writer: upsert `engineer_states` + `ownership_map` in MongoDB
|
||||
- Event detector: Gemini text prompt over all active states
|
||||
- Nudge generator: Gemini text → 1–2 sentence spoken message
|
||||
- Voice publisher: Gemini Live 2.5 via LiveKit Agents → audio into room
|
||||
- Data channel: sends structured nudge payload alongside audio
|
||||
- Cooldown: 3 min between nudges per pod
|
||||
|
||||
### Gemini usage
|
||||
- **Vision:** `gemini-2.0-flash` — screen → `{ currentFile, inferredTask, terminalVisible, recentTerminalOutput, confidence }`
|
||||
- **Event detection:** `gemini-2.0-flash` — all engineer states → `{ event, involvedEngineers, file, reason }`
|
||||
- **Nudge generation:** `gemini-2.0-flash` — event → spoken message text
|
||||
- **Voice:** `gemini-live-2.5-flash` via LiveKit Agents — text → streaming audio
|
||||
|
||||
### MongoDB Atlas (4 collections)
|
||||
- `engineer_states`: latest context per engineer, upserted each ingest
|
||||
- `ownership_map`: file → primaryOwner + contributors, persists across sessions (continual learning)
|
||||
- `events`: all detected coordination events
|
||||
- `nudges`: all voice nudges sent + cooldown history
|
||||
|
||||
### LiveKit
|
||||
- One room per pod
|
||||
- Engineers publish screen track (used client-side for capture — Hermes does not subscribe)
|
||||
- Hermes joins as `podman-hermes`, publishes audio + data channel messages
|
||||
- Engineers receive audio automatically
|
||||
|
||||
---
|
||||
|
||||
## Event types
|
||||
|
||||
| Event | Trigger | Example nudge |
|
||||
|---|---|---|
|
||||
| `BLOCKER_DETECTED` | Engineer stuck (error in terminal, same file N frames) + teammate can help | "Carol, looks like you're waiting on auth. Alice is actively building it — hang tight." |
|
||||
| `DEPENDENCY_READY` | Engineer A completes work that Engineer B was waiting on | "Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate." |
|
||||
| `DUPLICATE_WORK` | 2+ engineers on same file simultaneously | "Alice and Bob — you're both in login.tsx. Coordinate before pushing." |
|
||||
|
||||
---
|
||||
|
||||
## Continual learning story
|
||||
|
||||
The `ownership_map` collection persists across sessions. On Hermes startup:
|
||||
1. Load ownership map for this pod from Atlas
|
||||
2. Build in-memory cache: `Map<file, { primaryOwner, contributors }>`
|
||||
3. Event detection uses priors immediately — no ramp-up phase
|
||||
|
||||
**Demo:** Session 1 takes 3 min to first nudge. Session 2 fires in < 30 seconds. That is the learning, visible on stage.
|
||||
|
||||
---
|
||||
|
||||
## Demo flow (3 min)
|
||||
|
||||
1. **(0:00)** Three engineers join pod. PodMan greets by voice.
|
||||
2. **(0:20)** Alice opens `auth/middleware.ts`. Hermes infers ownership.
|
||||
3. **(0:45)** Bob opens `frontend/login.tsx`. Carol's terminal shows connection refused.
|
||||
4. **(1:20) BLOCKER_DETECTED:** "Carol, looks like you're waiting on auth. Alice is actively building it — hang tight."
|
||||
5. **(2:00) DEPENDENCY_READY:** "Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate."
|
||||
6. **(2:20)** Optional: session 2 warm-start comparison.
|
||||
7. **(2:45)** Close: "PodMan — the teammate that sees what Slack can't."
|
||||
|
||||
---
|
||||
|
||||
## Key risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Gemini Vision accuracy | Large font, single editor window, confidence gate |
|
||||
| Gemini Live 2.5 + LiveKit Agents integration | Build together hour 5–7, have TTS fallback |
|
||||
| Frame POST latency | JPEG compression, target < 500ms |
|
||||
| Event false positives | 3-min cooldown, pre-staged demo |
|
||||
| DO deploy failure | Hermes runs local, PWA defaults to localhost:8787 |
|
||||
Reference in New Issue
Block a user