OSS/podman
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs: add PodMan vision README and master plan

Browse Source 
Define the product (ambient AI teammate that prevents merge collisions
before push), architecture, stack per folder, build order, and demo script.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Kartikeya
2026-06-27 14:04:54 -07:00
parent 1bb15724c6
commit 3223e331f4
2 changed files with 211 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+27 -13
View File
@@ -1,18 +1,32 @@
# Podman
# 🛰️ PodMan — Jarvis for engineering teams
Monorepo for the Podman project.
Ambient AI teammate for the **2026 AI Engineer World's Fair Hackathon** (track: **Continual Learning**).
## Structure
Engineers join a **pod** and share their screen + mic. PodMan watches every screen in
**realtime**, understands what each person is working on, fuses that with the team's
GitHub state, and **proactively prevents collisions before code is even pushed** —
speaking up like Jarvis: _"Karti and Yahya are both editing `auth.ts`, Yahya has unpushed
changes — here's the diff, want a sync PR?"_
| Directory | Purpose |
| ----------- | ------------------------------------------------ |
| `backend/` | Server-side application code and APIs |
| `frontend/` | Client-side application / UI |
| `database/` | Schema, migrations, and seed data |
| `infra/` | Infrastructure, deployment, and IaC |
| `shared/` | Code shared across packages (types, utils, etc.)|
| `docs/` | Project documentation |
> **The moat:** unpushed local code is invisible to the GitHub API. The realtime
> screen-vision layer is the *only* way to catch a collision before the push. That is the
> whole product — and the reason this is an ambient agent, not a dashboard.
## Getting started
## Monorepo layout
This is the initial scaffold. Each package will be filled in as development progresses.
| Folder | What |
|---|---|
| `frontend/` | React + Vite PWA — join pod, screen/mic/cam capture, PodMan voice + intervention cards |
| `backend/` | PodMan agent: LiveKit room subscriber, Gemini vision, GitHub fusion, collision detector, voice out |
| `infra/` | DigitalOcean deploy (App Platform / Droplet), IaC, Dockerfiles |
| `database/` | MongoDB Atlas schema + Voyage vector memory for continual learning |
| `shared/` | Shared TypeScript types (pod, engineer context, intervention) |
| `docs/` | `PLAN.md` (the north star), demo script |
## Prizes we're stacking
- 🏆 **Best Gemini 3.5** — $5,000 cash (realtime vision + Live API voice)
- 🏆 **Best LiveKit** — Keychron keyboards (realtime transport is the core)
- 🏆 **Best DigitalOcean** — credits (deploy target)
👉 **Read [`docs/PLAN.md`](docs/PLAN.md) first.**
docs/PLAN.md
+184
View File
@@ -0,0 +1,184 @@
# PodMan — Master Plan (v1)
> Living doc. A deeper, API-accurate v2 (exact Gemini/LiveKit SDK calls, starter code,
> DO deploy steps) is being generated by the research workflow and will be merged in.
---
## 0. TL;DR
**PodMan** is an ambient AI teammate. Engineers join a **pod**, share screen + mic, and
PodMan watches everyone's screen in realtime, understands what each person is doing
(Gemini vision), fuses it with the team's GitHub state, and **interrupts like Jarvis to
prevent merge collisions before anyone pushes**, offering to open a sync PR.
- **Track:** Continual Learning
- **Prizes targeted (stacked):** Gemini 3.5 ($5k cash), LiveKit (keyboards), DigitalOcean (credits)
- **Hero moment:** two laptops editing the same file → PodMan *speaks up live* and offers the fix.
---
## 1. Why this fits the track (and stays eligible)
### Track = Continual Learning
The official definition rewards systems that "continuously improve from real-world use…
becoming more useful the more they are used with as little user intervention as possible."
PodMan does exactly this:
- Builds and **continuously refines a live model of the team** — who owns which files/areas,
what's in-flight, recurring conflict patterns, each engineer's working style.
- **Self-improves its own intervention policy** from outcomes: did the collision it predicted
actually happen? Did the team accept the suggested PR? It tunes its thresholds/prompts so it
nags less and helps more over time.
- Grows a per-team **skill/memory store** (vector memory) that makes later sessions sharper.
### ⚠️ Disqualification traps — and how we dodge them
| Risk | Mitigation |
|---|---|
| **"Dashboard is the main feature" = auto-DQ** | The pod grid is *secondary*. The hero is PodMan's **proactive voice/card interventions**. In the demo we barely show the grid; we show PodMan *acting*. |
| Repo must be **public** | Make the GitHub repo public from the start. |
| **Only what you built** during the event | Everything in this monorepo is new, timestamped by commits. Demo narrates "built today." |
| New work only | No pre-existing project reuse. |
---
## 2. Prize-stacking map
| Prize | How PodMan earns it |
|---|---|
| **Best Gemini 3.5 ($5,000 cash)** | Realtime screen understanding via Gemini 3.5 Flash vision + PodMan's voice via Gemini Live API. Bonus: Live Translate so a multilingual pod hears PodMan in their language. |
| **Best LiveKit (keyboards)** | LiveKit is the realtime backbone: screen-share + mic + cam tracks in, PodMan voice + data-channel cards out. It's load-bearing, not bolted on. |
| **Best DigitalOcean (credits)** | Backend PodMan agent + LiveKit agent deployed on DigitalOcean; claim the $200 credits. |
---
## 3. Architecture (v1 — refined by workflow)
```
┌────────────── Engineer laptops (Chrome PWA) ──────────────┐
│ getDisplayMedia (screen) + mic + cam │
│ publish tracks ─────────────┐ ▲ PodMan voice │
└──────────────────────────────┼─────────┼──────────────────┘
│ │ data-channel cards
┌───────▼─────────┴────────┐
│ LiveKit room │ (one room per pod)
└───────┬─────────▲──────────┘
│ subscribe│ publish voice/data
┌───────▼─────────┴──────────────────────┐
│ BACKEND: PodMan agent (DigitalOcean) │
│ │
│ 1. grab frames from each screen track │
│ 2. Gemini 3.5 vision → structured │
│ "engineer context" (file, symbol, │
│ feature, action) │
│ 3. GitHub client → branches/PRs/commits │
│ 4. COLLISION DETECTOR (fuse 2+3) │
│ 5. continual-learning memory (Atlas + │
│ Voyage vectors): team model + policy │
│ 6. PodMan brain (Gemini) → intervention │
│ 7. speak (Gemini Live/TTS) + send card │
└──────────────────────────────────────────┘
```
**The critical reconciliation:** GitHub only knows *pushed* state. The "X is editing this and
hasn't pushed" signal comes from **vision on the live screen** (filename in the editor tab,
visible diff/gutter), optionally cross-checked by an *optional* lightweight local `git status`
reporter the engineer can run. Vision is the headline; the local reporter is a nice-to-have.
### Continual-learning loop
1. **Observe** — per-engineer context every few seconds (sampled frames, not every frame).
2. **Store** — append observations to the team model; embed file/feature notes into Voyage
vectors in Atlas for retrieval.
3. **Predict** — collision detector + PodMan brain decide if/when to intervene.
4. **Outcome** — record whether the warning was acted on / was a true positive.
5. **Adapt** — adjust intervention thresholds, ownership attribution, and prompt context from
outcomes → fewer false alarms, better targeting over the session. *(This is the "gets
better the more you use it" story judges want.)*
---
## 4. Stack per folder
| Folder | Stack |
|---|---|
| `frontend/` | React + Vite + TypeScript, `livekit-client`, PWA (vite-plugin-pwa), Tailwind |
| `backend/` | Node + TypeScript, LiveKit server SDK + agents, `@google/genai`, GitHub (Octokit or GitHub MCP), Express/ws |
| `database/` | MongoDB Atlas (team model, observations, outcomes) + Voyage embeddings for vector recall |
| `infra/` | DigitalOcean App Platform / Droplet, Dockerfile, app spec |
| `shared/` | TS types: `Pod`, `EngineerContext`, `Collision`, `Intervention` |
---
## 5. 20-hour build order (MVP-first)
1. **Plumbing** — monorepo installs, shared types, env wiring, LiveKit token endpoint. *(Karti)*
2. **Capture** — frontend: join pod + publish screen/mic; render PodMan card + play voice. *(Zander)*
3. **Eyes** — backend: subscribe to a screen track, grab a frame, Gemini vision → `EngineerContext`. *(Ramis)*
4. **Brain + collision** — fuse two engineers' contexts + GitHub state → detect same-file/feature; PodMan brain composes the intervention. *(Yahya)*
5. **Voice + action** — PodMan speaks (Live API/TTS) into the room + "Open sync PR" via GitHub. *(Yahya + Ramis)*
6. **Memory/continual learning** — store observations + outcomes; show the team model improving. *(Karti)*
7. **Deploy on DO + polish demo** — everyone. Rehearse the live demo 3×.
> If behind: cut webcam, cut multilingual, cut the local git reporter, **mock the QR join**,
> hardcode the demo repo. Never cut: realtime screen→vision→PodMan-speaks loop.
---
## 6. Team split (4 max — must drop to 4!)
> ⚠️ Roster has 5 (Karti, Ramis, Yahya, Zander, Shakthi). **Max team size is 4.** Decide who's
> the official 4 before submission, or one stays unofficial/support.
| Person | Owns |
|---|---|
| **Karti** | Repo/infra/plumbing, shared types, memory + continual-learning store, DO deploy |
| **Zander** | Frontend PWA: pod join, capture, PodMan card UI + voice playback |
| **Ramis** | Backend realtime: LiveKit room subscribe + frame grab + Gemini vision pipeline |
| **Yahya** | PodMan brain: collision detector, intervention policy, GitHub PR action, voice out |
---
## 7. Demo script (3 min — refined by workflow)
1. **(0:00)** Two laptops on screen. Both engineers "join the pod" (QR mock). PodMan greets them by voice.
2. **(0:30)** Engineer A opens `auth.ts` and starts editing. PodMan quietly notes it (show the team model tick).
3. **(1:00)** Engineer B opens the *same* `auth.ts` and edits a related function — **neither has pushed.**
4. **(1:20) MONEY MOMENT** — PodMan *interrupts by voice*: "Heads up — Karti and Yahya are both in `auth.ts`, Yahya has unpushed changes. Here's the diff. Want me to open a sync PR?" Card appears with the diff.
5. **(1:50)** One click → PodMan opens a draft PR via GitHub (show it on github.com).
6. **(2:20)** Show it **learned**: PodMan now knows Karti owns auth; second scenario it's faster/quieter where appropriate → "more useful the more you use it."
7. **(2:45)** One-liner close: "PodMan — the teammate that sees what git can't."
---
## 8. Env vars (v1 — finalized by workflow)
```
# LiveKit
LIVEKIT_URL=
LIVEKIT_API_KEY=
LIVEKIT_API_SECRET=
# Gemini
GEMINI_API_KEY=
GEMINI_VISION_MODEL=gemini-3.5-flash # confirm exact id from research
GEMINI_LIVE_MODEL= # confirm from research
# GitHub
GITHUB_TOKEN=
GITHUB_REPO=owner/name
# MongoDB Atlas + Voyage
MONGODB_URI=
VOYAGE_API_KEY=
```
---
## 9. Open risks
| Risk | Mitigation |
|---|---|
| Realtime vision latency/cost | Sample ~1 frame/sec or on-change; downscale frames; cache last context |
| LiveKit ↔ Gemini frame plumbing is the hardest part | Build & de-risk it **first** (step 3); have a screenshot-fallback path |
| "Unpushed" detection is fuzzy | Lead with vision; optional local `git status` reporter for accuracy |
| On-stage flakiness | Pre-stage the demo repo, rehearse 3×, have a recorded backup of the money moment |
| Dashboard-DQ optics | Keep UI minimal; demo PodMan *acting*, not a grid |