OSS/podman

Fork 0

Files

T

Yahya Alhinai 89893110f1 feat: activate gemini voice tts path

2026-06-28 02:11:02 +00:00

7.2 KiB

Raw Blame History

name, description, metadata

name

description

metadata

podman-design

Full system design for PodMan — real-time AI team coordination agent using Gemini Vision, Gemini Live 2.5, LiveKit, and MongoDB Atlas

type
project

PodMan — System Design

Concept

PodMan is a real-time AI team coordination agent for software teams. Engineers join a LiveKit room with earbuds. Each engineer's browser PWA captures their screen every 30s and sends it to Hermes (server-side orchestrator on DigitalOcean). Hermes uses Gemini Vision to extract structured context per engineer, detects coordination events, and speaks proactive nudges into the room via Gemini Live 2.5 through LiveKit. MongoDB Atlas stores team state and an ownership map that persists across sessions.

Track: Continual Learning — the ownership map makes PodMan faster and smarter each session with no user configuration.

Architecture

┌──────────────── Engineer laptop (Browser PWA) ──────────────────┐
│  getDisplayMedia → frame every 30s                               │
│  HTTP POST /ingest → { screenshot, engineerId, podId }           │
│  LiveKit room joined → receives voice audio from Hermes          │
│  Earbuds: hears PodMan proactive nudges                          │
└──────────────────────────────────────────────────────────────────┘
                          │ POST /ingest
                          ▼
┌────────────────── HERMES (DigitalOcean) ─────────────────────────┐
│  1. Receive frame → Gemini Vision → EngineerContext              │
│  2. Write context to MongoDB (per-user state)                    │
│  3. Update ownership map (file → engineer)                       │
│  4. Run event detector over all active contexts                  │
│  5. If event detected → Gemini generates voice message           │
│  6. Push audio into LiveKit room via Gemini Live 2.5             │
└──────────────────────────────────────────────────────────────────┘
                          │ read/write
                          ▼
                   MongoDB Atlas
         (engineer_states, ownership_map,
          events, nudges)

Components

PWA (local agent)

Joins LiveKit room via existing joinPod flow
Captures frame every 30s via getDisplayMedia, compresses to JPEG (1280×720, quality 0.7)
POSTs { engineerId, podId, screenshotBase64, capturedAt } to POST /ingest
Receives Hermes audio track (automatic via LiveKit)
Listens for data channel messages → renders nudge feed
Two screens: join screen (built), active session screen (to build)

Hermes (orchestrator)

Express server + LiveKit Agent on DigitalOcean
POST /ingest: receives frame, queues for vision
Vision pipeline: Gemini 2.0 Flash → EngineerContext
Confidence gate: discard frames with confidence < 0.6
State writer: upsert engineer_states + ownership_map in MongoDB
Event detector: Gemini text prompt over all active states
Nudge generator: Gemini text → 1–2 sentence spoken message
Voice publisher: Gemini Live 2.5 via LiveKit Agents → audio into room
Data channel: sends structured nudge payload alongside audio
Cooldown: 3 min between nudges per pod

Gemini usage

Vision: gemini-2.0-flash — screen → { currentFile, inferredTask, terminalVisible, recentTerminalOutput, confidence }
Event detection: gemini-2.0-flash — all engineer states → { event, involvedEngineers, file, reason }
Nudge generation: gemini-2.0-flash — event → spoken message text
Voice: gemini-3.1-flash-tts-preview via LiveKit audio publication — text → audio

MongoDB Atlas (4 collections)

engineer_states: latest context per engineer, upserted each ingest
ownership_map: file → primaryOwner + contributors, persists across sessions (continual learning)
events: all detected coordination events
nudges: all voice nudges sent + cooldown history

LiveKit

One room per pod
Engineers publish screen track (used client-side for capture — Hermes does not subscribe)
Hermes joins as podman-hermes, publishes audio + data channel messages
Engineers receive audio automatically

Event types

Event	Trigger	Example nudge
`BLOCKER_DETECTED`	Engineer stuck (error in terminal, same file N frames) + teammate can help	"Carol, looks like you're waiting on auth. Alice is actively building it — hang tight."
`DEPENDENCY_READY`	Engineer A completes work that Engineer B was waiting on	"Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate."
`DUPLICATE_WORK`	2+ engineers on same file simultaneously	"Alice and Bob — you're both in login.tsx. Coordinate before pushing."

Continual learning story

The ownership_map collection persists across sessions. On Hermes startup:

Load ownership map for this pod from Atlas
Build in-memory cache: Map<file, { primaryOwner, contributors }>
Event detection uses priors immediately — no ramp-up phase

Demo: Session 1 takes 3 min to first nudge. Session 2 fires in < 30 seconds. That is the learning, visible on stage.

Demo flow (3 min)

(0:00) Three engineers join pod. PodMan greets by voice.
(0:20) Alice opens auth/middleware.ts. Hermes infers ownership.
(0:45) Bob opens frontend/login.tsx. Carol's terminal shows connection refused.
(1:20) BLOCKER_DETECTED: "Carol, looks like you're waiting on auth. Alice is actively building it — hang tight."
(2:00) DEPENDENCY_READY: "Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate."
(2:20) Optional: session 2 warm-start comparison.
(2:45) Close: "PodMan — the teammate that sees what Slack can't."

Key risks

Risk	Mitigation
Gemini Vision accuracy	Large font, single editor window, confidence gate
Gemini Live 2.5 + LiveKit Agents integration	Build together hour 5–7, have TTS fallback
Frame POST latency	JPEG compression, target < 500ms
Event false positives	3-min cooldown, pre-staged demo
DO deploy failure	Hermes runs local, PWA defaults to localhost:8787

7.2 KiB Raw Blame History Unescape Escape