Add member work history and learning docs
This commit is contained in:
Yahya Alhinai
@@ -7,11 +7,16 @@ metadata:
|
||||
|
||||
# PodMan — System Design
|
||||
|
||||
Status: historical reference. Current implementation truth lives in
|
||||
[`../../PLAN.md`](../../PLAN.md), [`../../mongodb.md`](../../mongodb.md),
|
||||
[`../../continual-learning/`](../../continual-learning/), and
|
||||
[`../../graph-discovery/`](../../graph-discovery/).
|
||||
|
||||
## Concept
|
||||
|
||||
PodMan is a real-time AI team coordination agent for software teams. Engineers join a LiveKit room with earbuds. Each engineer's browser PWA captures their screen every 30s and sends it to Hermes (server-side orchestrator on DigitalOcean). Hermes uses Gemini Vision to extract structured context per engineer, detects coordination events, and speaks proactive nudges into the room via Gemini Live 2.5 through LiveKit. MongoDB Atlas stores team state and an ownership map that persists across sessions.
|
||||
PodMan is a real-time AI team coordination agent for software teams. Engineers join a consented LiveKit room and publish screen share when they want PodMan to observe active work. The backend agent samples the LiveKit screen track, uses Gemini Vision to extract structured context, detects coordination risks, and sends intervention cards, Hermes messages, or urgent voice cues through LiveKit. MongoDB Atlas stores observations, collisions, interventions, outcomes, latest engineer state, and the Team memory graph.
|
||||
|
||||
**Track:** Continual Learning — the ownership map makes PodMan faster and smarter each session with no user configuration.
|
||||
**Track:** Continual Learning — accepted and dismissed outcomes make later exact-signature recall and graph memory more useful.
|
||||
|
||||
---
|
||||
|
||||
@@ -19,26 +24,27 @@ PodMan is a real-time AI team coordination agent for software teams. Engineers j
|
||||
|
||||
```
|
||||
┌──────────────── Engineer laptop (Browser PWA) ──────────────────┐
|
||||
│ getDisplayMedia → frame every 30s │
|
||||
│ HTTP POST /ingest → { screenshot, engineerId, podId } │
|
||||
│ LiveKit room joined → receives voice audio from Hermes │
|
||||
│ Earbuds: hears PodMan proactive nudges │
|
||||
│ getDisplayMedia → LiveKit screen-share track │
|
||||
│ Local git watcher → MongoDB engineer_states │
|
||||
│ LiveKit room joined → receives cards, messages, voice cues │
|
||||
│ Earbuds: hears PodMan urgent voice cues │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│ POST /ingest
|
||||
│ LiveKit media + data
|
||||
▼
|
||||
┌────────────────── HERMES (DigitalOcean) ─────────────────────────┐
|
||||
│ 1. Receive frame → Gemini Vision → EngineerContext │
|
||||
│ 2. Write context to MongoDB (per-user state) │
|
||||
│ 3. Update ownership map (file → engineer) │
|
||||
│ 4. Run event detector over all active contexts │
|
||||
│ 5. If event detected → Gemini generates voice message │
|
||||
│ 6. Push audio into LiveKit room via Gemini Live 2.5 │
|
||||
│ 1. Subscribe to screen-share track → Gemini Vision │
|
||||
│ 2. Write observations and per-user state to MongoDB │
|
||||
│ 3. Fuse local git truth from engineer_states │
|
||||
│ 4. Run collision detector over active contexts │
|
||||
│ 5. If risk detected → card/message first, voice only if urgent │
|
||||
│ 6. Push data and optional audio into LiveKit room │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│ read/write
|
||||
▼
|
||||
MongoDB Atlas
|
||||
(engineer_states, ownership_map,
|
||||
events, nudges)
|
||||
(engineer_states, observations,
|
||||
collisions, interventions, outcomes,
|
||||
team_model, graph_nodes, graph_edges)
|
||||
```
|
||||
|
||||
---
|
||||
@@ -48,51 +54,53 @@ PodMan is a real-time AI team coordination agent for software teams. Engineers j
|
||||
### PWA (local agent)
|
||||
|
||||
- Joins LiveKit room via existing `joinPod` flow
|
||||
- Captures frame every 30s via `getDisplayMedia`, compresses to JPEG (1280×720, quality 0.7)
|
||||
- POSTs `{ engineerId, podId, screenshotBase64, capturedAt }` to `POST /ingest`
|
||||
- Receives Hermes audio track (automatic via LiveKit)
|
||||
- Listens for data channel messages → renders nudge feed
|
||||
- Publishes screen share through LiveKit after explicit user action
|
||||
- Receives Hermes audio track through LiveKit when voice is urgent
|
||||
- Listens for data channel messages → renders intervention feed
|
||||
- Two screens: join screen (built), active session screen (to build)
|
||||
|
||||
### Hermes (orchestrator)
|
||||
|
||||
- Express server + LiveKit Agent on DigitalOcean
|
||||
- `POST /ingest`: receives frame, queues for vision
|
||||
- LiveKit agent worker receives sampled screen-share frames and queues them for vision
|
||||
- Vision pipeline: Gemini 2.0 Flash → `EngineerContext`
|
||||
- Confidence gate: discard frames with confidence < 0.6
|
||||
- State writer: upsert `engineer_states` + `ownership_map` in MongoDB
|
||||
- State writer: write `observations`, `collisions`, `interventions`, `outcomes`, and `engineer_states`
|
||||
- Event detector: Gemini text prompt over all active states
|
||||
- Nudge generator: Gemini text → 1–2 sentence spoken message
|
||||
- Voice publisher: Gemini Live 2.5 via LiveKit Agents → audio into room
|
||||
- Data channel: sends structured nudge payload alongside audio
|
||||
- Cooldown: 3 min between nudges per pod
|
||||
- Message generator: Gemini text → short intervention message
|
||||
- Voice publisher: Gemini TTS via LiveKit audio into room for urgent escalation
|
||||
- Data channel: sends structured intervention payload
|
||||
- Cooldown: 3 min between voice cues per pod
|
||||
|
||||
### Gemini usage
|
||||
|
||||
- **Vision:** `gemini-2.0-flash` — screen → `{ currentFile, inferredTask, terminalVisible, recentTerminalOutput, confidence }`
|
||||
- **Event detection:** `gemini-2.0-flash` — all engineer states → `{ event, involvedEngineers, file, reason }`
|
||||
- **Nudge generation:** `gemini-2.0-flash` — event → spoken message text
|
||||
- **Message generation:** `gemini-2.0-flash` — risk → intervention text
|
||||
- **Voice:** `gemini-3.1-flash-tts-preview` via LiveKit audio publication — text → audio
|
||||
|
||||
### MongoDB Atlas (4 collections)
|
||||
### MongoDB Atlas
|
||||
|
||||
- `engineer_states`: latest context per engineer, upserted each ingest
|
||||
- `ownership_map`: file → primaryOwner + contributors, persists across sessions (continual learning)
|
||||
- `events`: all detected coordination events
|
||||
- `nudges`: all voice nudges sent + cooldown history
|
||||
- `engineer_states`: latest context per engineer
|
||||
- `observations`: structured perception records
|
||||
- `collisions`: detected coordination risks
|
||||
- `interventions`: cards, messages, and voice cues sent or suggested
|
||||
- `outcomes`: accepted and dismissed learning signals
|
||||
- `team_model`: durable per-pod summary and seeded graph
|
||||
- `graph_nodes` / `graph_edges`: normalized graph records for `$graphLookup`
|
||||
|
||||
### LiveKit
|
||||
|
||||
- One room per pod
|
||||
- Engineers publish screen track (used client-side for capture — Hermes does not subscribe)
|
||||
- Hermes joins as `podman-hermes`, publishes audio + data channel messages
|
||||
- Engineers publish screen-share tracks
|
||||
- PodMan joins as an agent participant, subscribes to screen share, and publishes audio + data channel messages
|
||||
- Engineers receive audio automatically
|
||||
|
||||
---
|
||||
|
||||
## Event types
|
||||
|
||||
| Event | Trigger | Example nudge |
|
||||
| Event | Trigger | Example intervention |
|
||||
| ------------------ | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
|
||||
| `BLOCKER_DETECTED` | Engineer stuck (error in terminal, same file N frames) + teammate can help | "Carol, looks like you're waiting on auth. Alice is actively building it — hang tight." |
|
||||
| `DEPENDENCY_READY` | Engineer A completes work that Engineer B was waiting on | "Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate." |
|
||||
@@ -102,13 +110,17 @@ PodMan is a real-time AI team coordination agent for software teams. Engineers j
|
||||
|
||||
## Continual learning story
|
||||
|
||||
The `ownership_map` collection persists across sessions. On Hermes startup:
|
||||
The `team_model` graph and accepted outcomes persist across sessions. On graph
|
||||
load:
|
||||
|
||||
1. Load ownership map for this pod from Atlas
|
||||
2. Build in-memory cache: `Map<file, { primaryOwner, contributors }>`
|
||||
3. Event detection uses priors immediately — no ramp-up phase
|
||||
1. Materialize from live MongoDB records when real activity exists.
|
||||
2. Fall back to seeded `team_model.graph`.
|
||||
3. Fall back to a labeled demo graph for stage stability.
|
||||
4. Exact signature recall uses accepted and dismissed outcomes before vector recall.
|
||||
|
||||
**Demo:** Session 1 takes 3 min to first nudge. Session 2 fires in < 30 seconds. That is the learning, visible on stage.
|
||||
**Demo:** The first collision writes an outcome. The second similar collision
|
||||
recalls that memory and changes the graph or behavior. That is the learning
|
||||
visible on stage.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user