docs: reposition around team-coordination, sync specs to code, prune stale docs
Reframe README around the real bottleneck (human coordination, not engineering ability) with the "five-minute meeting" cost story; rewrite the demo script to match. Update gemini/livekit/mongodb/digitalocean specs to reflect shipped code (Gemini Live agent, TTS, embeddings, Lyria via Interactions API; current API routes; hermes_jobs collections). Add hermes.md. Rename graph.md -> cont_learning.md. Remove outdated/dead docs (agent-learning scaffolding, graph-discovery, handoffs, superpowers, idea/plan/demo-setup) and the bundled LiveKit starter under examples/. Rewire CLAUDE.md doc-first gate off the deleted PLAN.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LuV8W8oNYRsDWKoqK8Mkqc
This commit is contained in:
+57
-69
@@ -1,15 +1,24 @@
|
||||
# LiveKit Integration Spec
|
||||
|
||||
LiveKit is the real-time backbone for PodMan. It handles room presence and voice delivery. It is load-bearing — not decorative.
|
||||
Status: active / matches code.
|
||||
|
||||
LiveKit is the real-time backbone for PodMan. It carries the **screen-share
|
||||
perception input**, the **intervention data channel**, and **all room audio**
|
||||
(Gemini TTS escalations, the Lyria score, and the live conversation agent). It is
|
||||
load-bearing, not decorative.
|
||||
|
||||
---
|
||||
|
||||
## Room structure
|
||||
|
||||
- One LiveKit room per project pod: `room = podId`
|
||||
- Engineers join as named participants (e.g. `alice`, `bob`)
|
||||
- Hermes joins as `podman-hermes`
|
||||
- All participants stay connected for the duration of the session
|
||||
- One LiveKit room per pod: `room = podId`.
|
||||
- Engineers join as named participants (e.g. `alice`, `bob`).
|
||||
- PodMan runs **multiple agent identities** in/around a room:
|
||||
- `podman-hermes` — the main vision + intervention agent (`@livekit/rtc-node`).
|
||||
- `podman-live-conversation` — the Gemini Live voice agent (Python).
|
||||
- short-lived `podman-hermes-job-*` publishers for async job events.
|
||||
- A fixed identity matters: a second `podman-hermes` evicts the first and they
|
||||
flap, dropping interventions. systemd keeps exactly one alive in production.
|
||||
|
||||
---
|
||||
|
||||
@@ -17,100 +26,79 @@ LiveKit is the real-time backbone for PodMan. It handles room presence and voice
|
||||
|
||||
**Joining:**
|
||||
|
||||
1. PWA calls `POST /pods/:podId/token` → receives `{ token, url }`
|
||||
2. LiveKit client connects to the room with the token
|
||||
3. PWA publishes screen track via `getDisplayMedia`
|
||||
4. PWA sets mic enabled for ambient presence
|
||||
1. PWA calls `POST /api/token` with `{ podId, identity }` → `{ token, url }`.
|
||||
2. LiveKit client connects with the token.
|
||||
3. PWA publishes the screen track via `getDisplayMedia`.
|
||||
4. PWA enables mic for ambient presence (used by the conversation agent).
|
||||
|
||||
**Receiving:**
|
||||
|
||||
- LiveKit client subscribes to remote Hermes audio tracks and attaches them to
|
||||
a hidden audio sink in the DOM.
|
||||
- Browser autoplay restrictions still apply. The PWA calls `room.startAudio()`
|
||||
from user gestures such as first room click, `Enable audio`, `Test PodMan
|
||||
voice`, and `Share screen`.
|
||||
- PWA also listens for data channel messages from Hermes for UI card updates and
|
||||
`VOICE_CUE` fallback text.
|
||||
- Subscribes to remote agent audio tracks (TTS, Lyria, conversation) and attaches
|
||||
them to a hidden audio sink.
|
||||
- Browser autoplay restrictions apply: the PWA calls `room.startAudio()` from a
|
||||
user gesture (`Enable audio`, `Test PodMan voice`, `Share screen`, first room
|
||||
click).
|
||||
- Listens on the data channel for cards and `VOICE_CUE` fallback text.
|
||||
|
||||
**Data channel listener (PWA):**
|
||||
|
||||
```ts
|
||||
room.on(RoomEvent.DataReceived, (payload, participant) => {
|
||||
if (participant?.identity !== 'podman-hermes') return;
|
||||
const intervention = JSON.parse(new TextDecoder().decode(payload));
|
||||
// intervention: COLLISION, HERMES_MESSAGE, VOICE_CUE, ACK, or GIT_REPORT
|
||||
appendInterventionToFeed(intervention);
|
||||
if (!participant?.identity.startsWith('podman-')) return;
|
||||
const msg = JSON.parse(new TextDecoder().decode(payload));
|
||||
// msg.type: COLLISION | ACK | GIT_REPORT | VOICE_CUE | HERMES_JOB_EVENT
|
||||
appendInterventionToFeed(msg);
|
||||
});
|
||||
```
|
||||
|
||||
All data messages share the `podman.intervention` topic (`DATA_TOPIC`).
|
||||
|
||||
---
|
||||
|
||||
## Hermes side (PodMan LiveKit participant)
|
||||
## Agent side (`podman-hermes`)
|
||||
|
||||
**Framework:** `@livekit/rtc-node`
|
||||
**Framework:** `@livekit/rtc-node`. **Code:** `backend/src/agent/podman.ts`,
|
||||
`backend/src/action/hermes.ts`, `backend/src/voice/live.ts`.
|
||||
|
||||
**Startup:**
|
||||
1. Subscribes to engineers' screen-share tracks and samples frames for Gemini
|
||||
Vision.
|
||||
2. Detects collisions, gates them through the learning policy, then publishes a
|
||||
card/message on the data channel.
|
||||
3. For critical collisions, generates Gemini TTS audio and publishes it as a
|
||||
microphone-source audio track, held for the audio duration plus a tail/hold
|
||||
window so subscribers finish playout. Voice publishing logs frame count,
|
||||
estimated duration, queued playout, and hold time for diagnostics.
|
||||
|
||||
1. Hermes mints its own token via the same `createPodToken` function with `identity: 'podman-hermes'`
|
||||
2. Connects to the configured room as `podman-hermes`
|
||||
3. Publishes data-channel cards/messages and Gemini TTS audio tracks
|
||||
---
|
||||
|
||||
**Voice delivery:**
|
||||
## Live conversation agent (`podman-live-conversation`)
|
||||
|
||||
1. Urgent intervention text is ready (from Gemini text generation)
|
||||
2. Hermes sends a natural-speaking prompt to Gemini TTS
|
||||
3. Gemini returns PCM audio using the configured voice
|
||||
4. Hermes publishes the audio as a LiveKit microphone-source track
|
||||
5. Hermes keeps the track published for the generated audio duration plus tail
|
||||
silence and a hold window. This avoids browser-side cutoff when LiveKit's
|
||||
queued playout signal returns before subscribers finish playing buffered
|
||||
audio.
|
||||
6. All participants hear it after browser audio has been unlocked
|
||||
**Framework:** LiveKit Agents for Python (`AgentSession`, `function_tool`,
|
||||
`google.realtime.RealtimeModel`). **Code:**
|
||||
`agents/podman-live-conversation/agent.py`.
|
||||
|
||||
**Data channel message (sent alongside audio):**
|
||||
|
||||
```ts
|
||||
const intervention = {
|
||||
type: 'DEPENDENCY_READY' | 'BLOCKER_DETECTED' | 'DUPLICATE_WORK',
|
||||
message: string, // the spoken text
|
||||
involvedEngineers: string[],
|
||||
file: string | null,
|
||||
sentAt: string, // ISO timestamp
|
||||
};
|
||||
room.localParticipant.publishData(
|
||||
new TextEncoder().encode(JSON.stringify(intervention)),
|
||||
{ reliable: true }
|
||||
);
|
||||
```
|
||||
Joins the pod room on demand (`POST /api/pods/:id/live-conversation/start`),
|
||||
streams speech-to-speech with Gemini Live, and answers using repo/git/memory
|
||||
function tools. It can delegate long tasks to the async Hermes job runner and
|
||||
narrate progress. See `docs/hermes.md`.
|
||||
|
||||
---
|
||||
|
||||
## Token endpoint
|
||||
|
||||
Already implemented at `POST /api/token`.
|
||||
|
||||
Hermes uses the same endpoint. Grants:
|
||||
`POST /api/token` mints room tokens for engineers and agents alike. Grants:
|
||||
|
||||
- `roomJoin: true`
|
||||
- `canPublish: true` (for audio track)
|
||||
- `canPublishData: true` (for data channel)
|
||||
- `canPublish: true` (audio + screen)
|
||||
- `canPublishData: true` (data channel)
|
||||
- `canSubscribe: true`
|
||||
|
||||
---
|
||||
|
||||
## Gemini voice model
|
||||
|
||||
- Model ID: `gemini-3.1-flash-tts-preview`
|
||||
- Default voice: `Charon` (`GEMINI_TTS_VOICE`)
|
||||
- Hermes generates Gemini TTS audio and publishes it as a LiveKit audio track.
|
||||
- Voice publishing logs generated frame count, estimated duration, queued
|
||||
playout, and the final subscriber hold time for diagnostics.
|
||||
- The backend keeps a Gemini Live path for future model availability, but the verified deployment path uses TTS.
|
||||
Short-lived job publishers use `canSubscribe: false`.
|
||||
|
||||
---
|
||||
|
||||
## What LiveKit does NOT do in PodMan
|
||||
|
||||
- No video tracks from Hermes
|
||||
- No mic transcription (not needed for v1)
|
||||
- No SFU mixing — standard room behavior is sufficient
|
||||
- No video tracks published by agents.
|
||||
- No mic transcription outside the live conversation agent.
|
||||
- No custom SFU mixing — standard room behavior is sufficient.
|
||||
|
||||
Reference in New Issue
Block a user