From 3223e331f4fb3c99aec3493eab2c41d8cd41ba2d Mon Sep 17 00:00:00 2001 From: Kartikeya <176560021+karti-ai@users.noreply.github.com> Date: Sat, 27 Jun 2026 14:04:54 -0700 Subject: [PATCH] docs: add PodMan vision README and master plan Define the product (ambient AI teammate that prevents merge collisions before push), architecture, stack per folder, build order, and demo script. Co-Authored-By: Claude Opus 4.8 --- README.md | 40 +++++++---- docs/PLAN.md | 184 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 211 insertions(+), 13 deletions(-) create mode 100644 docs/PLAN.md diff --git a/README.md b/README.md index ff40103..644b5c5 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,32 @@ -# Podman +# πŸ›°οΈ PodMan β€” Jarvis for engineering teams -Monorepo for the Podman project. +Ambient AI teammate for the **2026 AI Engineer World's Fair Hackathon** (track: **Continual Learning**). -## Structure +Engineers join a **pod** and share their screen + mic. PodMan watches every screen in +**realtime**, understands what each person is working on, fuses that with the team's +GitHub state, and **proactively prevents collisions before code is even pushed** β€” +speaking up like Jarvis: _"Karti and Yahya are both editing `auth.ts`, Yahya has unpushed +changes β€” here's the diff, want a sync PR?"_ -| Directory | Purpose | -| ----------- | ------------------------------------------------ | -| `backend/` | Server-side application code and APIs | -| `frontend/` | Client-side application / UI | -| `database/` | Schema, migrations, and seed data | -| `infra/` | Infrastructure, deployment, and IaC | -| `shared/` | Code shared across packages (types, utils, etc.)| -| `docs/` | Project documentation | +> **The moat:** unpushed local code is invisible to the GitHub API. The realtime +> screen-vision layer is the *only* way to catch a collision before the push. That is the +> whole product β€” and the reason this is an ambient agent, not a dashboard. -## Getting started +## Monorepo layout -This is the initial scaffold. Each package will be filled in as development progresses. +| Folder | What | +|---|---| +| `frontend/` | React + Vite PWA β€” join pod, screen/mic/cam capture, PodMan voice + intervention cards | +| `backend/` | PodMan agent: LiveKit room subscriber, Gemini vision, GitHub fusion, collision detector, voice out | +| `infra/` | DigitalOcean deploy (App Platform / Droplet), IaC, Dockerfiles | +| `database/` | MongoDB Atlas schema + Voyage vector memory for continual learning | +| `shared/` | Shared TypeScript types (pod, engineer context, intervention) | +| `docs/` | `PLAN.md` (the north star), demo script | + +## Prizes we're stacking + +- πŸ† **Best Gemini 3.5** β€” $5,000 cash (realtime vision + Live API voice) +- πŸ† **Best LiveKit** β€” Keychron keyboards (realtime transport is the core) +- πŸ† **Best DigitalOcean** β€” credits (deploy target) + +πŸ‘‰ **Read [`docs/PLAN.md`](docs/PLAN.md) first.** diff --git a/docs/PLAN.md b/docs/PLAN.md new file mode 100644 index 0000000..e39b4e3 --- /dev/null +++ b/docs/PLAN.md @@ -0,0 +1,184 @@ +# PodMan β€” Master Plan (v1) + +> Living doc. A deeper, API-accurate v2 (exact Gemini/LiveKit SDK calls, starter code, +> DO deploy steps) is being generated by the research workflow and will be merged in. + +--- + +## 0. TL;DR + +**PodMan** is an ambient AI teammate. Engineers join a **pod**, share screen + mic, and +PodMan watches everyone's screen in realtime, understands what each person is doing +(Gemini vision), fuses it with the team's GitHub state, and **interrupts like Jarvis to +prevent merge collisions before anyone pushes**, offering to open a sync PR. + +- **Track:** Continual Learning +- **Prizes targeted (stacked):** Gemini 3.5 ($5k cash), LiveKit (keyboards), DigitalOcean (credits) +- **Hero moment:** two laptops editing the same file β†’ PodMan *speaks up live* and offers the fix. + +--- + +## 1. Why this fits the track (and stays eligible) + +### Track = Continual Learning +The official definition rewards systems that "continuously improve from real-world use… +becoming more useful the more they are used with as little user intervention as possible." +PodMan does exactly this: +- Builds and **continuously refines a live model of the team** β€” who owns which files/areas, + what's in-flight, recurring conflict patterns, each engineer's working style. +- **Self-improves its own intervention policy** from outcomes: did the collision it predicted + actually happen? Did the team accept the suggested PR? It tunes its thresholds/prompts so it + nags less and helps more over time. +- Grows a per-team **skill/memory store** (vector memory) that makes later sessions sharper. + +### ⚠️ Disqualification traps β€” and how we dodge them +| Risk | Mitigation | +|---|---| +| **"Dashboard is the main feature" = auto-DQ** | The pod grid is *secondary*. The hero is PodMan's **proactive voice/card interventions**. In the demo we barely show the grid; we show PodMan *acting*. | +| Repo must be **public** | Make the GitHub repo public from the start. | +| **Only what you built** during the event | Everything in this monorepo is new, timestamped by commits. Demo narrates "built today." | +| New work only | No pre-existing project reuse. | + +--- + +## 2. Prize-stacking map + +| Prize | How PodMan earns it | +|---|---| +| **Best Gemini 3.5 ($5,000 cash)** | Realtime screen understanding via Gemini 3.5 Flash vision + PodMan's voice via Gemini Live API. Bonus: Live Translate so a multilingual pod hears PodMan in their language. | +| **Best LiveKit (keyboards)** | LiveKit is the realtime backbone: screen-share + mic + cam tracks in, PodMan voice + data-channel cards out. It's load-bearing, not bolted on. | +| **Best DigitalOcean (credits)** | Backend PodMan agent + LiveKit agent deployed on DigitalOcean; claim the $200 credits. | + +--- + +## 3. Architecture (v1 β€” refined by workflow) + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ Engineer laptops (Chrome PWA) ──────────────┐ +β”‚ getDisplayMedia (screen) + mic + cam β”‚ +β”‚ publish tracks ─────────────┐ β–² PodMan voice β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ β”‚ data-channel cards + β”Œβ”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ LiveKit room β”‚ (one room per pod) + β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β–²β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ subscribeβ”‚ publish voice/data + β”Œβ”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ BACKEND: PodMan agent (DigitalOcean) β”‚ + β”‚ β”‚ + β”‚ 1. grab frames from each screen track β”‚ + β”‚ 2. Gemini 3.5 vision β†’ structured β”‚ + β”‚ "engineer context" (file, symbol, β”‚ + β”‚ feature, action) β”‚ + β”‚ 3. GitHub client β†’ branches/PRs/commits β”‚ + β”‚ 4. COLLISION DETECTOR (fuse 2+3) β”‚ + β”‚ 5. continual-learning memory (Atlas + β”‚ + β”‚ Voyage vectors): team model + policy β”‚ + β”‚ 6. PodMan brain (Gemini) β†’ intervention β”‚ + β”‚ 7. speak (Gemini Live/TTS) + send card β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +**The critical reconciliation:** GitHub only knows *pushed* state. The "X is editing this and +hasn't pushed" signal comes from **vision on the live screen** (filename in the editor tab, +visible diff/gutter), optionally cross-checked by an *optional* lightweight local `git status` +reporter the engineer can run. Vision is the headline; the local reporter is a nice-to-have. + +### Continual-learning loop +1. **Observe** β€” per-engineer context every few seconds (sampled frames, not every frame). +2. **Store** β€” append observations to the team model; embed file/feature notes into Voyage + vectors in Atlas for retrieval. +3. **Predict** β€” collision detector + PodMan brain decide if/when to intervene. +4. **Outcome** β€” record whether the warning was acted on / was a true positive. +5. **Adapt** β€” adjust intervention thresholds, ownership attribution, and prompt context from + outcomes β†’ fewer false alarms, better targeting over the session. *(This is the "gets + better the more you use it" story judges want.)* + +--- + +## 4. Stack per folder + +| Folder | Stack | +|---|---| +| `frontend/` | React + Vite + TypeScript, `livekit-client`, PWA (vite-plugin-pwa), Tailwind | +| `backend/` | Node + TypeScript, LiveKit server SDK + agents, `@google/genai`, GitHub (Octokit or GitHub MCP), Express/ws | +| `database/` | MongoDB Atlas (team model, observations, outcomes) + Voyage embeddings for vector recall | +| `infra/` | DigitalOcean App Platform / Droplet, Dockerfile, app spec | +| `shared/` | TS types: `Pod`, `EngineerContext`, `Collision`, `Intervention` | + +--- + +## 5. 20-hour build order (MVP-first) + +1. **Plumbing** β€” monorepo installs, shared types, env wiring, LiveKit token endpoint. *(Karti)* +2. **Capture** β€” frontend: join pod + publish screen/mic; render PodMan card + play voice. *(Zander)* +3. **Eyes** β€” backend: subscribe to a screen track, grab a frame, Gemini vision β†’ `EngineerContext`. *(Ramis)* +4. **Brain + collision** β€” fuse two engineers' contexts + GitHub state β†’ detect same-file/feature; PodMan brain composes the intervention. *(Yahya)* +5. **Voice + action** β€” PodMan speaks (Live API/TTS) into the room + "Open sync PR" via GitHub. *(Yahya + Ramis)* +6. **Memory/continual learning** β€” store observations + outcomes; show the team model improving. *(Karti)* +7. **Deploy on DO + polish demo** β€” everyone. Rehearse the live demo 3Γ—. + +> If behind: cut webcam, cut multilingual, cut the local git reporter, **mock the QR join**, +> hardcode the demo repo. Never cut: realtime screenβ†’visionβ†’PodMan-speaks loop. + +--- + +## 6. Team split (4 max β€” must drop to 4!) + +> ⚠️ Roster has 5 (Karti, Ramis, Yahya, Zander, Shakthi). **Max team size is 4.** Decide who's +> the official 4 before submission, or one stays unofficial/support. + +| Person | Owns | +|---|---| +| **Karti** | Repo/infra/plumbing, shared types, memory + continual-learning store, DO deploy | +| **Zander** | Frontend PWA: pod join, capture, PodMan card UI + voice playback | +| **Ramis** | Backend realtime: LiveKit room subscribe + frame grab + Gemini vision pipeline | +| **Yahya** | PodMan brain: collision detector, intervention policy, GitHub PR action, voice out | + +--- + +## 7. Demo script (3 min β€” refined by workflow) + +1. **(0:00)** Two laptops on screen. Both engineers "join the pod" (QR mock). PodMan greets them by voice. +2. **(0:30)** Engineer A opens `auth.ts` and starts editing. PodMan quietly notes it (show the team model tick). +3. **(1:00)** Engineer B opens the *same* `auth.ts` and edits a related function β€” **neither has pushed.** +4. **(1:20) MONEY MOMENT** β€” PodMan *interrupts by voice*: "Heads up β€” Karti and Yahya are both in `auth.ts`, Yahya has unpushed changes. Here's the diff. Want me to open a sync PR?" Card appears with the diff. +5. **(1:50)** One click β†’ PodMan opens a draft PR via GitHub (show it on github.com). +6. **(2:20)** Show it **learned**: PodMan now knows Karti owns auth; second scenario it's faster/quieter where appropriate β†’ "more useful the more you use it." +7. **(2:45)** One-liner close: "PodMan β€” the teammate that sees what git can't." + +--- + +## 8. Env vars (v1 β€” finalized by workflow) + +``` +# LiveKit +LIVEKIT_URL= +LIVEKIT_API_KEY= +LIVEKIT_API_SECRET= + +# Gemini +GEMINI_API_KEY= +GEMINI_VISION_MODEL=gemini-3.5-flash # confirm exact id from research +GEMINI_LIVE_MODEL= # confirm from research + +# GitHub +GITHUB_TOKEN= +GITHUB_REPO=owner/name + +# MongoDB Atlas + Voyage +MONGODB_URI= +VOYAGE_API_KEY= +``` + +--- + +## 9. Open risks + +| Risk | Mitigation | +|---|---| +| Realtime vision latency/cost | Sample ~1 frame/sec or on-change; downscale frames; cache last context | +| LiveKit ↔ Gemini frame plumbing is the hardest part | Build & de-risk it **first** (step 3); have a screenshot-fallback path | +| "Unpushed" detection is fuzzy | Lead with vision; optional local `git status` reporter for accuracy | +| On-stage flakiness | Pre-stage the demo repo, rehearse 3Γ—, have a recorded backup of the money moment | +| Dashboard-DQ optics | Keep UI minimal; demo PodMan *acting*, not a grid |