OSS/podman
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs: split architecture graph into infra + Gemini diagrams

Browse Source 
Replace the single all-in-one architecture flowchart with two focused
diagrams: (1) how LiveKit, Hermes, MongoDB, and DigitalOcean wire
together, and (2) Gemini's five surfaces with their triggers, models,
and call sites.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LuV8W8oNYRsDWKoqK8Mkqc
This commit is contained in:
Ramis
2026-06-28 08:11:43 -07:00
parent 9fb252ec6a
commit f4ad3bec63
1 changed files with 79 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+79 -39
View File
@@ -96,53 +96,93 @@ A browser PWA, an HTTP API service, LiveKit agent workers, and a
memory/action layer. The screen signal flows through LiveKit, never a manual memory/action layer. The screen signal flows through LiveKit, never a manual
screenshot upload. screenshot upload.
### Infrastructure: LiveKit · Hermes · MongoDB · DigitalOcean
How the real-time, action, memory, and hosting layers wire together. Everything
under the droplet boundary is one systemd-supervised DigitalOcean box; LiveKit
and Mongo Atlas are managed clouds it talks to.
```mermaid ```mermaid
flowchart LR flowchart LR
subgraph Laptop["Engineer laptop"] Dev["Engineer browser<br/>React PWA + screen share"]
PWA["React PWA"]
Screen["Screen share track"]
Git["Git watcher<br/>scripts/podman-agent.mjs"]
end
subgraph Realtime["LiveKit room"] subgraph DO["DigitalOcean droplet — systemd-supervised"]
Room["Pod room"] Caddy["Caddy<br/>serves static frontend · proxies /api"]
Data["Data topic<br/>podman.intervention"] API["API service<br/>/api/token · /api/pods · /api/outcome"]
Audio["Audio tracks<br/>Gemini TTS + Lyria"]
end
subgraph Backend["PodMan backend"]
API["API service<br/>/api/token /api/pods /api/outcome"]
Agent["Vision agent worker<br/>@livekit/rtc-node"] Agent["Vision agent worker<br/>@livekit/rtc-node"]
Convo["Live conversation agent<br/>Gemini Live API (Python)"] Detector["Coordination detector<br/>collisions + blockers"]
Vision["Gemini Vision<br/>structured JSON"] Hermes["Hermes action layer<br/>cards · messages · urgent voice"]
Detector["Coordination detector<br/>collisions, blockers"] PyAgent["Live conversation agent<br/>LiveKit Agents (Python)"]
end end
subgraph Memory["Memory and actions"] subgraph LK["LiveKit room — managed cloud"]
Mongo["MongoDB Atlas<br/>observations, outcomes, vectors"] Tracks["Screen-share video tracks"]
GitHub["GitHub<br/>repo state + sync PR artifact"] DataCh["Data topic<br/>podman.intervention"]
Hermes["Hermes action layer<br/>cards, messages, urgent voice"] AudioCh["Audio tracks<br/>TTS + Lyria music"]
end end
PWA -->|"POST /api/token"| API Mongo[("MongoDB Atlas<br/>observations · collisions<br/>interventions · outcomes · vectors")]
API -->|"LiveKit JWT"| PWA
PWA --> Screen Dev -->|static assets| Caddy
Screen --> Room Dev -->|"POST /api/*"| Caddy --> API
Room --> Agent API -->|LiveKit JWT| Dev
Agent --> Vision Dev -->|publish screen| Tracks
Vision --> Detector
Git --> Mongo Agent -->|subscribe frames| Tracks
Detector --> Mongo Agent --> Detector --> Hermes
Detector --> GitHub Hermes -->|cards / messages| DataCh
Detector --> Hermes Hermes -->|urgent voice| AudioCh
Hermes --> Data PyAgent <-->|join room · speak| AudioCh
Hermes --> Audio DataCh --> Dev
Convo --> Audio AudioCh --> Dev
Convo --> Mongo Dev -->|"POST /api/outcome"| Caddy
Data --> PWA
Audio --> PWA API <--> Mongo
PWA -->|"POST /api/outcome"| API Agent -->|write observations| Mongo
API --> Mongo Detector -->|read recall · write collisions| Mongo
Hermes -->|write interventions| Mongo
PyAgent -->|tool reads| Mongo
```
> Mongo connectivity is **required at boot** — the API and both agents ping Atlas
> on startup and exit loudly rather than degrade silently.
### Gemini: roles, models, and triggers
Five distinct Gemini surfaces, each fired by a specific event in the pod and
mapped to the file that calls it.
```mermaid
flowchart LR
subgraph Triggers["Trigger (event in the pod)"]
Frame["Sampled IDE frame<br/>from LiveKit screen track"]
NewEvent["New observation<br/>or collision"]
Talk["Engineer speaks<br/>in the room"]
Urgent["Hermes raises an<br/>urgent intervention"]
Session["Pod session<br/>active"]
end
subgraph G["Gemini surface · model · code"]
Vision["Vision<br/>gemini-2.0-flash<br/>backend/src/vision/gemini.ts"]
Embed["Embeddings<br/>gemini-embedding-001<br/>backend/src/memory/vectors.ts"]
Live["Live voice agent<br/>gemini-3.1-flash-live-preview<br/>agents/podman-live-conversation/agent.py"]
TTS["Urgent TTS<br/>gemini-3.1-flash-tts-preview<br/>backend/src/voice/live.ts"]
Lyria["Background music<br/>lyria-3-clip-preview<br/>backend/src/voice/music.ts"]
end
subgraph Out["Output"]
Ctx["Structured work context<br/>{file, symbol, activity, mode, research}"]
Recall["$vectorSearch recall<br/>prior intervention + outcome"]
Answer["Spoken answer over<br/>repo / git / memory tools"]
Voice["Spoken alert<br/>in the room"]
Score["Per-pod ambient<br/>score"]
end
Frame --> Vision --> Ctx
NewEvent --> Embed --> Recall
Talk --> Live --> Answer
Urgent --> TTS --> Voice
Session --> Lyria --> Score
``` ```