OSS/podman
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: use gemini tts for livekit voice

Browse Source
This commit is contained in:
Yahya Alhinai
2026-06-28 05:43:38 +00:00
parent c771e1594c
commit 721fe2b8d9
15 changed files with 82 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/PLAN.md
+6 -5
View File
@@ -319,6 +319,7 @@ LIVEKIT_API_SECRET=
GEMINI_API_KEY=
GEMINI_VISION_MODEL=
GEMINI_LIVE_MODEL=
GEMINI_TTS_VOICE=
GITHUB_TOKEN=
GITHUB_REPO=karti-ai/podman
@@ -361,9 +362,8 @@ Keep all non-`VITE_` secrets server-side.
- Use low media resolution for ambient screen watching; reserve higher
resolution for debugging or targeted inspection.
- Never expose `GEMINI_API_KEY` to the browser.
- Gemini Live API is still a risk for the first demo path. Use card + Hermes
message first; add browser TTS or pre-generated voice fallback before relying
on Gemini Live for stage audio.
- Use card + Hermes message first. For urgent stage audio, default to Gemini TTS
published through LiveKit; keep browser TTS only as an explicit fallback flag.
- Keep model IDs in env so preview/availability changes do not require code
changes.
@@ -484,7 +484,8 @@ artifact.
- Add visible live inference captions in the PWA.
- Add a small memory stats panel backed by `/api/memory/stats`.
- Add browser-side TTS or pre-generated voice fallback for urgent interventions.
- Keep browser-side TTS as an explicit demo fallback only; Gemini TTS over
LiveKit is the default urgent-voice path.
- Add Hermes notification bridge once the target channel is chosen.
- Improve research cards with compatibility, install effort, docs quality, repo
health, and security/trust signals.
@@ -504,7 +505,7 @@ artifact.
- Full auth/accounts.
- Slack/Linear/Jira integrations unless Hermes requires one immediately.
- Complex dashboards.
- Server-published audio if browser/pre-generated voice proves escalation.
- Live voice polish beyond the Gemini TTS urgent-alert path.
- Vector Search if exact Mongo recall demonstrates the learning beat.
---
docs/digitalocean.md
+1
View File
@@ -70,6 +70,7 @@ LIVEKIT_API_SECRET=...
GEMINI_API_KEY=...
GEMINI_VISION_MODEL=gemini-2.0-flash
GEMINI_LIVE_MODEL=gemini-3.1-flash-tts-preview
GEMINI_TTS_VOICE=Charon
GITHUB_TOKEN=...
GITHUB_REPO=karti-ai/podman
docs/gemini.md
+11 -10
View File
@@ -112,23 +112,24 @@ Respond with the message text only.
## 4. Voice Output — Gemini TTS via LiveKit
**Model:** `gemini-3.1-flash-tts-preview`
**Default voice:** `Charon`
**Integration:** Hermes generates Gemini TTS audio and publishes it as a LiveKit audio track. The code still preserves a Gemini Live path for future available Live models.
**Integration:** Hermes asks Gemini TTS for short PCM audio, then publishes that audio into the room as a short LiveKit audio track. The code still preserves a Gemini Live path for future available Live models.
**Flow:**
1. Nudge message text generated (step 3)
2. Hermes passes text to Gemini Live via LiveKit Agents
3. Gemini Live streams audio back in real-time
4. LiveKit publishes audio into the pod room
5. All participants hear it through their audio output
2. Hermes wraps it in a natural-speaking prompt for Gemini TTS
3. Gemini returns audio with the configured prebuilt voice
4. Hermes publishes the audio into the LiveKit room
5. The frontend still renders the `VOICE_CUE` text, but browser TTS is off unless explicitly enabled
**Why Gemini Live (not plain TTS):**
**Why Gemini TTS first:**
- Streams audio directly — no intermediate WAV file conversion
- Latency ~300–500ms from text to first audio packet
- Natural-sounding voice
- Strong prize story: Gemini Live 2.5 is the headline model
- Natural voice quality is better than browser `speechSynthesis`
- Tone and pacing can be steered directly in the prompt
- The voice name is configurable with `GEMINI_TTS_VOICE`
- LiveKit remains the delivery layer, so teammates hear the same room audio
---
docs/idea.md
+3 -3
View File
@@ -43,7 +43,7 @@ Small software teams: hackathon squads, startup engineering teams, student dev t
- `BLOCKER_DETECTED` — engineer appears stuck; another teammate can unblock
- `DUPLICATE_WORK` — 2+ engineers working on the same file simultaneously
- Generate a 1–2 sentence proactive voice nudge
- Deliver it into the LiveKit room via Gemini Live 2.5
- Deliver it into the LiveKit room as Gemini TTS audio
---
@@ -60,7 +60,7 @@ The system gets demonstrably more useful the more it is used, with no user confi
## Architecture (one paragraph)
Each engineer opens a browser PWA on their laptop. The PWA captures a screen frame every 30 seconds via `getDisplayMedia` and POSTs it to Hermes, the server-side orchestrator running on DigitalOcean. Hermes calls Gemini Vision to extract structured context, writes it to MongoDB Atlas, updates the ownership map, and runs event detection across all active engineers. When a coordination event fires, Hermes generates a short spoken message and publishes it as audio into the team's LiveKit room via Gemini Live 2.5. Engineers hear PodMan through their earbuds. No Slack. No tab switching. No interruption to the editor flow.
Each engineer opens a browser PWA on their laptop. The PWA captures live IDE context through LiveKit screen sharing and scheduled local git reports. Hermes, the server-side orchestrator running on DigitalOcean, calls Gemini Vision to extract structured context, writes it to MongoDB Atlas, updates the ownership map, and runs event detection across all active engineers. When a coordination event fires, Hermes generates a short spoken message, asks Gemini TTS for natural audio, and publishes that audio into the team's LiveKit room. Engineers hear PodMan through their earbuds. No Slack. No tab switching. No interruption to the editor flow.
---
@@ -88,6 +88,6 @@ Each engineer opens a browser PWA on their laptop. The PWA captures a screen fra
| Prize | How PodMan earns it |
| --------------------- | ----------------------------------------------------------------------------------------------------- |
| Best Gemini 3.5 / 2.5 | Gemini Vision for screen understanding + Gemini Live 2.5 for voice output |
| Best Gemini 3.5 / 2.5 | Gemini Vision for screen understanding + Gemini TTS for urgent voice output |
| Best LiveKit | LiveKit is the real-time backbone for room presence and voice delivery — load-bearing, not decorative |
| Best DigitalOcean | Hermes deployed on DigitalOcean App Platform; MongoDB Atlas on DO-adjacent infrastructure |
docs/livekit.md
+8 -6
View File
@@ -41,22 +41,23 @@ room.on(RoomEvent.DataReceived, (payload, participant) => {
---
## Hermes side (LiveKit Agent)
## Hermes side (PodMan LiveKit participant)
**Framework:** LiveKit Agents (Node.js)
**Framework:** `@livekit/rtc-node`
**Startup:**
1. Hermes mints its own token via the same `createPodToken` function with `identity: 'podman-hermes'`
2. Connects to the configured room as `podman-hermes`
3. Registers as a LiveKit Agent with Gemini Live 2.5 as voice provider
3. Publishes data-channel cards/messages and short Gemini TTS audio tracks
**Voice delivery:**
1. Nudge message text is ready (from Gemini text generation)
2. Hermes passes text to Gemini Live 2.5 via LiveKit Agents voice pipeline
3. Audio streams into the room in real-time
4. All participants hear it
2. Hermes sends a natural-speaking prompt to Gemini TTS
3. Gemini returns PCM audio using the configured voice
4. Hermes publishes the audio as a short LiveKit track
5. All participants hear it
**Data channel message (sent alongside audio):**
@@ -92,6 +93,7 @@ Hermes uses the same endpoint. Grants:
## Gemini voice model
- Model ID: `gemini-3.1-flash-tts-preview`
- Default voice: `Charon` (`GEMINI_TTS_VOICE`)
- Hermes generates Gemini TTS audio and publishes it as a LiveKit audio track.
- The backend keeps a Gemini Live path for future model availability, but the verified deployment path uses TTS.