OSS/podman
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs: add README architecture section

Browse Source
This commit is contained in:
Yahya Alhinai
2026-06-27 23:54:25 +00:00
parent a853ef7709
commit 1294b8428d
1 changed files with 200 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+198 -47
View File
@@ -1,62 +1,212 @@
# PodMan — Real-time AI Team Coordination Agent
# PodMan - Real-time AI Team Coordination Agent
**2026 AI Engineer World's Fair Hackathon** — Track: **Continual Learning**
[![TypeScript](https://img.shields.io/badge/TypeScript-6.x-3178C6?logo=typescript&logoColor=white)](https://www.typescriptlang.org/)
[![React](https://img.shields.io/badge/React-19-61DAFB?logo=react&logoColor=111)](https://react.dev/)
[![LiveKit](https://img.shields.io/badge/LiveKit-realtime-000000?logo=livekit&logoColor=white)](https://livekit.io/)
[![MongoDB](https://img.shields.io/badge/MongoDB-memory-47A248?logo=mongodb&logoColor=white)](https://www.mongodb.com/)
[![Gemini](https://img.shields.io/badge/Gemini-vision-8E75B2?logo=googlegemini&logoColor=white)](https://ai.google.dev/)
[![DigitalOcean](https://img.shields.io/badge/DigitalOcean-deploy-0080FF?logo=digitalocean&logoColor=white)](https://www.digitalocean.com/)
PodMan is an ambient AI teammate. Engineers join a pod room with earbuds in. Each engineer's browser PWA captures their screen every 30 seconds. PodMan watches, understands what each person is working on, detects coordination gaps — blockers, dependencies, duplicate work — and speaks up proactively before anyone has to ask.
**2026 AI Engineer World's Fair Hackathon** - Track: **Continual Learning**
> "Carol, looks like you're waiting on auth. Alice is actively building it — hang tight."
> "Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate."
PodMan is a non-intrusive AI teammate for active coding. It watches consented
LiveKit screen-share context, combines it with local git truth and shared team
memory, and coordinates teammates before a problem becomes a GitHub problem.
Nobody sent a message. Nobody pinged on Slack. PodMan just knew.
> GitHub sees pushed work. PodMan sees work while it is still happening.
PodMan is not a dashboard and not a raw screenshot analyzer. Its job is to
notice useful coordination moments, remember what helped before, and route the
least intrusive intervention: a small card first, a Hermes message when teammates
need coordination, and voice only for urgent escalation.
---
## Architecture
PodMan is split into a browser PWA, an HTTP API service, a LiveKit agent worker,
and a persistence/action layer. The screen signal flows through LiveKit, not a
manual screenshot upload endpoint.
```mermaid
flowchart LR
subgraph Laptop["Engineer laptop"]
PWA["React PWA"]
Screen["Screen share track"]
Git["Git watcher<br/>scripts/podman-agent.mjs"]
end
subgraph Realtime["LiveKit room"]
Room["Pod room"]
Data["Data topic<br/>podman.intervention"]
end
subgraph Backend["PodMan backend"]
API["API service<br/>/api/token /api/pods /api/outcome"]
Agent["Agent worker<br/>@livekit/rtc-node"]
Vision["Gemini Vision<br/>structured JSON"]
Detector["Coordination detector<br/>collisions, blockers, dead ends"]
end
subgraph Memory["Memory and actions"]
Mongo["MongoDB<br/>observations, outcomes, pods"]
GitHub["GitHub<br/>repo state + sync PR artifact"]
Hermes["Hermes action layer<br/>cards, messages, urgent voice"]
end
PWA -->|"POST /api/token"| API
API -->|"LiveKit JWT"| PWA
PWA --> Screen
Screen --> Room
Room --> Agent
Agent --> Vision
Vision --> Detector
Git --> Mongo
Detector --> Mongo
Detector --> GitHub
Detector --> Hermes
Hermes --> Data
Data --> PWA
PWA -->|"POST /api/outcome"| API
API --> Mongo
```
### Runtime shape
| Layer | Runtime | Responsibility |
| ------------ | ------------------- | --------------------------------------------------------------------------------------------------- |
| Frontend PWA | React + Vite | Join pods, publish screen share, show live room state, render interventions |
| Backend API | Express | Mint LiveKit tokens, manage pods, record outcomes, expose memory stats, create sync PR artifacts |
| Agent worker | `@livekit/rtc-node` | Join the room as PodMan, subscribe to screen-share tracks, sample frames, publish intervention data |
| Vision loop | Gemini | Convert sampled IDE frames into structured work context |
| Team memory | MongoDB | Store observations, collisions, interventions, outcomes, pods, and git watcher state |
| Git watcher | Node script | Poll each laptop's local git state so dirty/unpushed work is not guessed from vision alone |
| Action layer | Hermes concept | Route cards, teammate messages, optional research summaries, and urgent voice escalation |
| Deployment | DigitalOcean | Static site for frontend, HTTP service for API, worker for the LiveKit agent |
### Data flow
```mermaid
sequenceDiagram
autonumber
participant Dev as Engineer PWA
participant API as Backend API
participant LK as LiveKit Room
participant Agent as PodMan Agent
participant Gemini as Gemini Vision
participant Mongo as MongoDB Memory
participant Hermes as Hermes / Action Layer
participant GH as GitHub
Dev->>API: POST /api/token
API-->>Dev: LiveKit URL + JWT
Dev->>LK: Join pod room
Dev->>LK: Publish screen-share track
Agent->>LK: Subscribe to screen-share video
Agent->>Gemini: Sampled JPEG frame
Gemini-->>Agent: Structured work context
Agent->>Mongo: Record observation
Agent->>GH: Read public repo state
Agent->>Mongo: Recall prior patterns
Agent->>Hermes: Create intervention
Hermes->>LK: Publish small data packet
LK-->>Dev: Render card / message / urgent voice cue
Dev->>API: POST /api/outcome
API->>Mongo: Store learning signal
```
### Why this architecture matters
- **LiveKit is the realtime spine.** Screens and intervention data move through a
shared room, so PodMan can react before code is pushed.
- **Gemini is the perception layer.** The agent samples frames and asks Gemini
for structured JSON such as current file, symbol, activity, unpushed hints,
and confidence.
- **MongoDB is the learning loop.** Outcomes and repeated patterns make later
interventions quieter and more useful.
- **Local git is the truth source.** The watcher reports dirty files and branch
state directly from each laptop, which avoids relying on vision for facts
GitHub cannot see.
- **Hermes keeps it non-intrusive.** Most events are cards. Team messages and
voice are escalation paths, not the default.
---
## How it works
1. Engineers open the PWA in their browser and join a pod room
2. PWA captures a screen frame every 30s via `getDisplayMedia`, POSTs it to Hermes
3. **Hermes** (server-side orchestrator) calls Gemini Vision to extract structured context per engineer — current file, inferred task, terminal state
4. Hermes writes context to MongoDB Atlas, updates the ownership map
5. Hermes runs event detection across all engineers — dependency ready, blocker, duplicate work
6. When an event fires, Hermes generates a spoken nudge and publishes it into the LiveKit room via Gemini Live 2.5
7. Engineers hear PodMan through their earbuds
1. Engineers open the PWA and join a pod room.
2. The backend API mints a LiveKit token via `POST /api/token`.
3. The PWA publishes screen share into the pod room when the engineer chooses
"Share my screen".
4. The PodMan agent worker joins the same room and subscribes to screen-share
tracks.
5. The agent samples frames, sends them to Gemini Vision, and records structured
observations in MongoDB.
6. Each engineer runs the git watcher so PodMan has deterministic dirty/unpushed
state.
7. The detector combines live screen context, git truth, GitHub state, and team
memory.
8. PodMan sends the smallest useful intervention: card first, Hermes message for
coordination, voice only when urgent.
9. The user's response is saved as an outcome, closing the continual-learning
loop.
The ownership map persists across sessions — making PodMan faster and smarter each time the team works together.
---
## Public interfaces
| Interface | Purpose |
| ----------------------------------------------------------- | ---------------------------------------------- |
| `GET /health` | API health check |
| `POST /api/token` | Mint LiveKit room tokens |
| `POST /api/sync-pr` | Create a visible sync PR artifact |
| `POST /api/outcome` | Store accepted/dismissed intervention outcomes |
| `GET /api/memory/stats` | Show memory collection counts |
| `GET/POST/PATCH/DELETE /api/pods` | Pod CRUD |
| `POST/DELETE /api/pods/:id/members` | Pod membership |
| LiveKit topic `podman.intervention` | Intervention data channel |
| Wire messages `COLLISION`, `ACK`, `GIT_REPORT`, `VOICE_CUE` | Shared agent/PWA message contract |
---
## Monorepo layout
| Folder | What |
|---|---|
| `frontend/` | React + Vite PWA — join pod, screen capture, nudge feed, live teammate status |
| `backend/` | Hermes orchestrator: `/ingest` endpoint, Gemini vision pipeline, event detector, LiveKit agent |
| `database/` | MongoDB Atlas schema — engineer states, ownership map, events, nudges |
| `infra/` | DigitalOcean App Platform deploy spec + Dockerfile |
| `shared/` | Shared TypeScript types |
| `docs/` | Full specs — read these first |
| ----------- | -------------------------------------------------------------------------------- |
| `frontend/` | React + Vite PWA for pods, LiveKit room UI, screen share, and intervention cards |
| `backend/` | Express API plus separate LiveKit agent worker |
| `shared/` | Shared TypeScript types and LiveKit data message contracts |
| `database/` | MongoDB setup and seed utilities |
| `infra/` | DigitalOcean App Platform specs and Dockerfile |
| `scripts/` | Local git watcher for demo laptops |
| `docs/` | Canonical plan and deeper sponsor/integration notes |
---
## Docs
| File | What |
|---|---|
| [`docs/idea.md`](docs/idea.md) | Full concept, value prop, demo moment |
| [`docs/PLAN.md`](docs/PLAN.md) | 12-hour build plan, team assignments, build order |
| [`docs/gemini.md`](docs/gemini.md) | Gemini Vision + event detection + Gemini Live 2.5 voice |
| [`docs/livekit.md`](docs/livekit.md) | LiveKit room structure, Hermes agent, voice delivery |
| [`docs/mongodb.md`](docs/mongodb.md) | MongoDB collections, schemas, continual learning hook |
| [`docs/digitalocean.md`](docs/digitalocean.md) | DO deploy config, env vars, fallback plan |
| [`docs/demo-setup.md`](docs/demo-setup.md) | Demo laptop setup, pre-staging checklist |
| ---------------------------------------------- | ----------------------------------------- |
| [`docs/PLAN.md`](docs/PLAN.md) | Canonical master plan and source of truth |
| [`docs/idea.md`](docs/idea.md) | Product concept and demo framing |
| [`docs/livekit.md`](docs/livekit.md) | LiveKit notes and room model |
| [`docs/gemini.md`](docs/gemini.md) | Gemini vision and voice notes |
| [`docs/mongodb.md`](docs/mongodb.md) | MongoDB memory design |
| [`docs/digitalocean.md`](docs/digitalocean.md) | Deployment notes |
| [`docs/demo-setup.md`](docs/demo-setup.md) | Demo laptop and stage checklist |
---
## Prizes targeted
- **Best Gemini** — Gemini Vision (screen understanding) + Gemini Live 2.5 (voice output via LiveKit Agents)
- **Best LiveKit** — LiveKit is the real-time backbone for room presence and voice delivery
- **Best DigitalOcean** — Hermes deployed on DigitalOcean App Platform
- **Best Gemini:** structured vision over live IDE context, with voice as an
optional escalation path.
- **Best LiveKit:** realtime screen-share tracks, presence, data packets, and
eventual voice in one pod room.
- **Best DigitalOcean:** frontend static site, API service, and LiveKit agent
worker deployment.
- **MongoDB + Voyage story:** persistent memory first, vector recall once exact
signature recall is proven.
---
@@ -64,40 +214,41 @@ The ownership map persists across sessions — making PodMan faster and smarter
```bash
cp .env.example .env
# fill in LIVEKIT_*, GEMINI_API_KEY, MONGODB_URI
# fill in LIVEKIT_*, GEMINI_*, GITHUB_*, and MONGODB_URI
pnpm install
pnpm --filter backend dev # Hermes on :8787
pnpm --filter frontend dev # PWA on :5173
pnpm --filter @podman/backend dev # API on :8787
pnpm --filter @podman/backend dev:agent # PodMan LiveKit agent
pnpm --filter @podman/frontend dev # PWA on :5173
```
---
## Git watcher — run this on every demo laptop
## Git watcher - run this on every demo laptop
Each engineer runs this in a terminal before the demo. It polls the local git working tree every 15 seconds and writes git state to MongoDB so PodMan has deterministic dirty/unpushed truth that vision alone cannot reliably infer.
Each engineer runs this in a terminal before the demo. It polls the local git
working tree every 15 seconds and writes git state to MongoDB so PodMan has
deterministic dirty/unpushed truth that vision alone cannot reliably infer.
```bash
# from the repo root
node scripts/podman-agent.mjs --name <yourname> --pod <podId>
```
**Demo setup (one command per laptop):**
**Demo setup:**
```bash
# Alice's laptop
node scripts/podman-agent.mjs --name alice --pod demo-pod
# Bob's laptop
node scripts/podman-agent.mjs --name bob --pod demo-pod
# Carol's laptop
node scripts/podman-agent.mjs --name carol --pod demo-pod
```
The script logs one line per cycle — branch, changed file count, and latest commit. Leave it running in a background terminal tab throughout the session. Stop with `Ctrl+C`.
The script logs one line per cycle: branch, changed file count, and latest
commit. Leave it running in a background terminal tab throughout the session.
Stop with `Ctrl+C`.
**Requirements:**
- `MONGODB_URI` must be set — either exported in the shell or present in `backend/.env`
- Run `pnpm install` first so `mongodb` is in workspace `node_modules`
- Must be run from the repo root (the script resolves `backend/.env` relative to its own path)
- `MONGODB_URI` must be exported in the shell or present in `backend/.env`.
- Run `pnpm install` first so workspace dependencies are available.
- Run from the repo root.