# PodMan — Master Plan (v1) > Living doc. A deeper, API-accurate v2 (exact Gemini/LiveKit SDK calls, starter code, > DO deploy steps) is being generated by the research workflow and will be merged in. --- ## 0. TL;DR **PodMan** is an ambient AI teammate. Engineers join a **pod**, share screen + mic, and PodMan watches everyone's screen in realtime, understands what each person is doing (Gemini vision), fuses it with the team's GitHub state, and **interrupts like Jarvis to prevent merge collisions before anyone pushes**, offering to open a sync PR. - **Track:** Continual Learning - **Prizes targeted (stacked):** Gemini 3.5 ($5k cash), LiveKit (keyboards), DigitalOcean (credits) - **Hero moment:** two laptops editing the same file → PodMan _speaks up live_ and offers the fix. --- ## 1. Why this fits the track (and stays eligible) ### Track = Continual Learning The official definition rewards systems that "continuously improve from real-world use… becoming more useful the more they are used with as little user intervention as possible." PodMan does exactly this: - Builds and **continuously refines a live model of the team** — who owns which files/areas, what's in-flight, recurring conflict patterns, each engineer's working style. - **Self-improves its own intervention policy** from outcomes: did the collision it predicted actually happen? Did the team accept the suggested PR? It tunes its thresholds/prompts so it nags less and helps more over time. - Grows a per-team **skill/memory store** (vector memory) that makes later sessions sharper. ### ⚠️ Disqualification traps — and how we dodge them | Risk | Mitigation | | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | **"Dashboard is the main feature" = auto-DQ** | The pod grid is _secondary_. The hero is PodMan's **proactive voice/card interventions**. In the demo we barely show the grid; we show PodMan _acting_. | | Repo must be **public** | Make the GitHub repo public from the start. | | **Only what you built** during the event | Everything in this monorepo is new, timestamped by commits. Demo narrates "built today." | | New work only | No pre-existing project reuse. | --- ## 2. Prize-stacking map | Prize | How PodMan earns it | | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Best Gemini 3.5 ($5,000 cash)** | Realtime screen understanding via Gemini 3.5 Flash vision + PodMan's voice via Gemini Live API. Bonus: Live Translate so a multilingual pod hears PodMan in their language. | | **Best LiveKit (keyboards)** | LiveKit is the realtime backbone: screen-share + mic + cam tracks in, PodMan voice + data-channel cards out. It's load-bearing, not bolted on. | | **Best DigitalOcean (credits)** | Backend PodMan agent + LiveKit agent deployed on DigitalOcean; claim the $200 credits. | --- ## 3. Architecture (v1 — refined by workflow) ``` ┌────────────── Engineer laptops (Chrome PWA) ──────────────┐ │ getDisplayMedia (screen) + mic + cam │ │ publish tracks ─────────────┐ ▲ PodMan voice │ └──────────────────────────────┼─────────┼──────────────────┘ │ │ data-channel cards ┌───────▼─────────┴────────┐ │ LiveKit room │ (one room per pod) └───────┬─────────▲──────────┘ │ subscribe│ publish voice/data ┌───────▼─────────┴──────────────────────┐ │ BACKEND: PodMan agent (DigitalOcean) │ │ │ │ 1. grab frames from each screen track │ │ 2. Gemini 3.5 vision → structured │ │ "engineer context" (file, symbol, │ │ feature, action) │ │ 3. GitHub client → branches/PRs/commits │ │ 4. COLLISION DETECTOR (fuse 2+3) │ │ 5. continual-learning memory (Atlas + │ │ Voyage vectors): team model + policy │ │ 6. PodMan brain (Gemini) → intervention │ │ 7. speak (Gemini Live/TTS) + send card │ └──────────────────────────────────────────┘ ``` **The critical reconciliation:** GitHub only knows _pushed_ state. The "X is editing this and hasn't pushed" signal comes from **vision on the live screen** (filename in the editor tab, visible diff/gutter), optionally cross-checked by an _optional_ lightweight local `git status` reporter the engineer can run. Vision is the headline; the local reporter is a nice-to-have. ### Continual-learning loop 1. **Observe** — per-engineer context every few seconds (sampled frames, not every frame). 2. **Store** — append observations to the team model; embed file/feature notes into Voyage vectors in Atlas for retrieval. 3. **Predict** — collision detector + PodMan brain decide if/when to intervene. 4. **Outcome** — record whether the warning was acted on / was a true positive. 5. **Adapt** — adjust intervention thresholds, ownership attribution, and prompt context from outcomes → fewer false alarms, better targeting over the session. _(This is the "gets better the more you use it" story judges want.)_ --- ## 4. Stack per folder | Folder | Stack | | ----------- | ----------------------------------------------------------------------------------------------------------- | | `frontend/` | React + Vite + TypeScript, `livekit-client`, PWA (vite-plugin-pwa), Tailwind | | `backend/` | Node + TypeScript, LiveKit server SDK + agents, `@google/genai`, GitHub (Octokit or GitHub MCP), Express/ws | | `database/` | MongoDB Atlas (team model, observations, outcomes) + Voyage embeddings for vector recall | | `infra/` | DigitalOcean App Platform / Droplet, Dockerfile, app spec | | `shared/` | TS types: `Pod`, `EngineerContext`, `Collision`, `Intervention` | --- ## 5. 20-hour build order (MVP-first) 1. **Plumbing** — monorepo installs, shared types, env wiring, LiveKit token endpoint. _(Karti)_ 2. **Capture** — frontend: join pod + publish screen/mic; render PodMan card + play voice. _(Zander)_ 3. **Eyes** — backend: subscribe to a screen track, grab a frame, Gemini vision → `EngineerContext`. _(Ramis)_ 4. **Brain + collision** — fuse two engineers' contexts + GitHub state → detect same-file/feature; PodMan brain composes the intervention. _(Yahya)_ 5. **Voice + action** — PodMan speaks (Live API/TTS) into the room + "Open sync PR" via GitHub. _(Yahya + Ramis)_ 6. **Memory/continual learning** — store observations + outcomes; show the team model improving. _(Karti)_ 7. **Deploy on DO + polish demo** — everyone. Rehearse the live demo 3×. > If behind: cut webcam, cut multilingual, cut the local git reporter, **mock the QR join**, > hardcode the demo repo. Never cut: realtime screen→vision→PodMan-speaks loop. --- ## 6. Team split (4 max — must drop to 4!) > ⚠️ Roster has 5 (Karti, Ramis, Yahya, Zander, Shakthi). **Max team size is 4.** Decide who's > the official 4 before submission, or one stays unofficial/support. | Person | Owns | | ---------- | ---------------------------------------------------------------------------------- | | **Karti** | Repo/infra/plumbing, shared types, memory + continual-learning store, DO deploy | | **Zander** | Frontend PWA: pod join, capture, PodMan card UI + voice playback | | **Ramis** | Backend realtime: LiveKit room subscribe + frame grab + Gemini vision pipeline | | **Yahya** | PodMan brain: collision detector, intervention policy, GitHub PR action, voice out | --- ## 7. Demo script (3 min — refined by workflow) 1. **(0:00)** Two laptops on screen. Both engineers "join the pod" (QR mock). PodMan greets them by voice. 2. **(0:30)** Engineer A opens `auth.ts` and starts editing. PodMan quietly notes it (show the team model tick). 3. **(1:00)** Engineer B opens the _same_ `auth.ts` and edits a related function — **neither has pushed.** 4. **(1:20) MONEY MOMENT** — PodMan _interrupts by voice_: "Heads up — Karti and Yahya are both in `auth.ts`, Yahya has unpushed changes. Here's the diff. Want me to open a sync PR?" Card appears with the diff. 5. **(1:50)** One click → PodMan opens a draft PR via GitHub (show it on github.com). 6. **(2:20)** Show it **learned**: PodMan now knows Karti owns auth; second scenario it's faster/quieter where appropriate → "more useful the more you use it." 7. **(2:45)** One-liner close: "PodMan — the teammate that sees what git can't." --- ## 8. Env vars (v1 — finalized by workflow) ``` # LiveKit LIVEKIT_URL= LIVEKIT_API_KEY= LIVEKIT_API_SECRET= # Gemini GEMINI_API_KEY= GEMINI_VISION_MODEL=gemini-3.5-flash # confirm exact id from research GEMINI_LIVE_MODEL= # confirm from research # GitHub GITHUB_TOKEN= GITHUB_REPO=owner/name # MongoDB Atlas + Voyage MONGODB_URI= VOYAGE_API_KEY= ``` --- ## 9. Open risks | Risk | Mitigation | | --------------------------------------------------- | -------------------------------------------------------------------------------- | | Realtime vision latency/cost | Sample ~1 frame/sec or on-change; downscale frames; cache last context | | LiveKit ↔ Gemini frame plumbing is the hardest part | Build & de-risk it **first** (step 3); have a screenshot-fallback path | | "Unpushed" detection is fuzzy | Lead with vision; optional local `git status` reporter for accuracy | | On-stage flakiness | Pre-stage the demo repo, rehearse 3×, have a recorded backup of the money moment | | Dashboard-DQ optics | Keep UI minimal; demo PodMan _acting_, not a grid |