docs: finalize PodMan architecture and write full integration specs
Replaces v1 plan with locked architecture: - Hermes orchestrator: POST /ingest → Gemini Vision → MongoDB → event detection → Gemini Live 2.5 voice via LiveKit Agents - Four MongoDB collections: engineer_states, ownership_map, events, nudges - Continual learning via ownership_map persisting across sessions New files: docs/idea.md, docs/gemini.md, docs/livekit.md, docs/mongodb.md, docs/digitalocean.md, docs/demo-setup.md Updated: README.md, docs/PLAN.md (12-hour build plan), database/README.md, infra/README.md, .env.example Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FFbfi4Cmb7BY75Wtne7bZn
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
# LiveKit Integration Spec
|
||||
|
||||
LiveKit is the real-time backbone for PodMan. It handles room presence and voice delivery. It is load-bearing — not decorative.
|
||||
|
||||
---
|
||||
|
||||
## Room structure
|
||||
|
||||
- One LiveKit room per project pod: `room = podId`
|
||||
- Engineers join as named participants (e.g. `alice`, `bob`)
|
||||
- Hermes joins as `podman-hermes`
|
||||
- All participants stay connected for the duration of the session
|
||||
|
||||
---
|
||||
|
||||
## Engineer side (PWA)
|
||||
|
||||
**Joining:**
|
||||
1. PWA calls `POST /pods/:podId/token` → receives `{ token, url }`
|
||||
2. LiveKit client connects to the room with the token
|
||||
3. PWA publishes screen track via `getDisplayMedia` (used client-side for frame capture — Hermes does NOT subscribe to this track)
|
||||
4. PWA sets mic enabled for ambient presence
|
||||
|
||||
**Receiving:**
|
||||
- LiveKit client automatically receives Hermes audio track
|
||||
- No special subscription needed — LiveKit delivers audio to all participants
|
||||
- PWA also listens for data channel messages from Hermes for UI card updates
|
||||
|
||||
**Data channel listener (PWA):**
|
||||
```ts
|
||||
room.on(RoomEvent.DataReceived, (payload, participant) => {
|
||||
if (participant?.identity !== 'podman-hermes') return;
|
||||
const nudge = JSON.parse(new TextDecoder().decode(payload));
|
||||
// nudge: { type, message, involvedEngineers, file, sentAt }
|
||||
appendNudgeToFeed(nudge);
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Hermes side (LiveKit Agent)
|
||||
|
||||
**Framework:** LiveKit Agents (Node.js)
|
||||
|
||||
**Startup:**
|
||||
1. Hermes mints its own token via the same `createPodToken` function with `identity: 'podman-hermes'`
|
||||
2. Connects to the room on pod creation / first engineer joining
|
||||
3. Registers as a LiveKit Agent with Gemini Live 2.5 as voice provider
|
||||
|
||||
**Voice delivery:**
|
||||
1. Nudge message text is ready (from Gemini text generation)
|
||||
2. Hermes passes text to Gemini Live 2.5 via LiveKit Agents voice pipeline
|
||||
3. Audio streams into the room in real-time
|
||||
4. All participants hear it
|
||||
|
||||
**Data channel message (sent alongside audio):**
|
||||
```ts
|
||||
const nudge = {
|
||||
type: 'DEPENDENCY_READY' | 'BLOCKER_DETECTED' | 'DUPLICATE_WORK',
|
||||
message: string, // the spoken text
|
||||
involvedEngineers: string[],
|
||||
file: string | null,
|
||||
sentAt: string, // ISO timestamp
|
||||
};
|
||||
room.localParticipant.publishData(
|
||||
new TextEncoder().encode(JSON.stringify(nudge)),
|
||||
{ reliable: true }
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Token endpoint
|
||||
|
||||
Already implemented at `POST /pods/:podId/token`.
|
||||
|
||||
Hermes uses the same endpoint. Grants:
|
||||
- `roomJoin: true`
|
||||
- `canPublish: true` (for audio track)
|
||||
- `canPublishData: true` (for data channel)
|
||||
- `canSubscribe: true`
|
||||
|
||||
---
|
||||
|
||||
## Gemini Live 2.5 model
|
||||
|
||||
- Model ID: `gemini-live-2.5-flash` — confirm exact ID from LiveKit Agents + Gemini docs at build time
|
||||
- LiveKit Agents has native Gemini Live integration — no manual audio encoding needed
|
||||
- Hermes passes text string → Agents handles streaming audio publication
|
||||
|
||||
---
|
||||
|
||||
## What LiveKit does NOT do in PodMan
|
||||
|
||||
- Hermes does NOT subscribe to engineer screen tracks (frame capture happens client-side)
|
||||
- No video tracks from Hermes
|
||||
- No mic transcription (not needed for v1)
|
||||
- No SFU mixing — standard room behavior is sufficient
|
||||
Reference in New Issue
Block a user