docs: finalize PodMan architecture and write full integration specs

Replaces v1 plan with locked architecture: - Hermes orchestrator: POST /ingest → Gemini Vision → MongoDB → event detection → Gemini Live 2.5 voice via LiveKit Agents - Four MongoDB collections: engineer_states, ownership_map, events, nudges - Continual learning via ownership_map persisting across sessions New files: docs/idea.md, docs/gemini.md, docs/livekit.md, docs/mongodb.md, docs/digitalocean.md, docs/demo-setup.md Updated: README.md, docs/PLAN.md (12-hour build plan), database/README.md, infra/README.md, .env.example Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FFbfi4Cmb7BY75Wtne7bZn
2026-06-27 15:23:36 -07:00
parent e26961d526
commit 923ab1cf58
13 changed files with 1313 additions and 210 deletions
@@ -1,32 +1,72 @@
-# 🛰️ PodMan — Jarvis for engineering teams
+# PodMan — Real-time AI Team Coordination Agent

-Ambient AI teammate for the **2026 AI Engineer World's Fair Hackathon** (track: **Continual Learning**).
+**2026 AI Engineer World's Fair Hackathon** — Track: **Continual Learning**

-Engineers join a **pod** and share their screen + mic. PodMan watches every screen in
-**realtime**, understands what each person is working on, fuses that with the team's
-GitHub state, and **proactively prevents collisions before code is even pushed** —
-speaking up like Jarvis: _"Karti and Yahya are both editing `auth.ts`, Yahya has unpushed
-changes — here's the diff, want a sync PR?"_
+PodMan is an ambient AI teammate. Engineers join a pod room with earbuds in. Each engineer's browser PWA captures their screen every 30 seconds. PodMan watches, understands what each person is working on, detects coordination gaps — blockers, dependencies, duplicate work — and speaks up proactively before anyone has to ask.

-> **The moat:** unpushed local code is invisible to the GitHub API. The realtime
-> screen-vision layer is the _only_ way to catch a collision before the push. That is the
-> whole product — and the reason this is an ambient agent, not a dashboard.
+> "Carol, looks like you're waiting on auth. Alice is actively building it — hang tight."
+> "Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate."
+
+Nobody sent a message. Nobody pinged on Slack. PodMan just knew.
+
+---
+
+## How it works
+
+1. Engineers open the PWA in their browser and join a pod room
+2. PWA captures a screen frame every 30s via `getDisplayMedia`, POSTs it to Hermes
+3. **Hermes** (server-side orchestrator) calls Gemini Vision to extract structured context per engineer — current file, inferred task, terminal state
+4. Hermes writes context to MongoDB Atlas, updates the ownership map
+5. Hermes runs event detection across all engineers — dependency ready, blocker, duplicate work
+6. When an event fires, Hermes generates a spoken nudge and publishes it into the LiveKit room via Gemini Live 2.5
+7. Engineers hear PodMan through their earbuds
+
+The ownership map persists across sessions — making PodMan faster and smarter each time the team works together.
+
+---

 ## Monorepo layout

-| Folder      | What                                                                                               |
-| ----------- | -------------------------------------------------------------------------------------------------- |
-| `frontend/` | React + Vite PWA — join pod, screen/mic/cam capture, PodMan voice + intervention cards             |
-| `backend/`  | PodMan agent: LiveKit room subscriber, Gemini vision, GitHub fusion, collision detector, voice out |
-| `infra/`    | DigitalOcean deploy (App Platform / Droplet), IaC, Dockerfiles                                     |
-| `database/` | MongoDB Atlas schema + Voyage vector memory for continual learning                                 |
-| `shared/`   | Shared TypeScript types (pod, engineer context, intervention)                                      |
-| `docs/`     | `PLAN.md` (the north star), demo script                                                            |
+| Folder | What |
+|---|---|
+| `frontend/` | React + Vite PWA — join pod, screen capture, nudge feed, live teammate status |
+| `backend/` | Hermes orchestrator: `/ingest` endpoint, Gemini vision pipeline, event detector, LiveKit agent |
+| `database/` | MongoDB Atlas schema — engineer states, ownership map, events, nudges |
+| `infra/` | DigitalOcean App Platform deploy spec + Dockerfile |
+| `shared/` | Shared TypeScript types |
+| `docs/` | Full specs — read these first |

-## Prizes we're stacking
+---

- 🏆 **Best Gemini 3.5** — $5,000 cash (realtime vision + Live API voice)
- 🏆 **Best LiveKit** — Keychron keyboards (realtime transport is the core)
- 🏆 **Best DigitalOcean** — credits (deploy target)
+## Docs

-👉 **Read [`docs/PLAN.md`](docs/PLAN.md) first.**
+| File | What |
+|---|---|
+| [`docs/idea.md`](docs/idea.md) | Full concept, value prop, demo moment |
+| [`docs/PLAN.md`](docs/PLAN.md) | 12-hour build plan, team assignments, build order |
+| [`docs/gemini.md`](docs/gemini.md) | Gemini Vision + event detection + Gemini Live 2.5 voice |
+| [`docs/livekit.md`](docs/livekit.md) | LiveKit room structure, Hermes agent, voice delivery |
+| [`docs/mongodb.md`](docs/mongodb.md) | MongoDB collections, schemas, continual learning hook |
+| [`docs/digitalocean.md`](docs/digitalocean.md) | DO deploy config, env vars, fallback plan |
+| [`docs/demo-setup.md`](docs/demo-setup.md) | Demo laptop setup, pre-staging checklist |
+
+---
+
+## Prizes targeted
+
+- **Best Gemini** — Gemini Vision (screen understanding) + Gemini Live 2.5 (voice output via LiveKit Agents)
+- **Best LiveKit** — LiveKit is the real-time backbone for room presence and voice delivery
+- **Best DigitalOcean** — Hermes deployed on DigitalOcean App Platform
+
+---
+
+## Quick start
+
+```bash
+cp .env.example .env
+# fill in LIVEKIT_*, GEMINI_API_KEY, MONGODB_URI
+
+pnpm install
+pnpm --filter backend dev   # Hermes on :8787
+pnpm --filter frontend dev  # PWA on :5173
+```