fix(voice): keep LiveKit TTS tracks alive
This commit is contained in:
Yahya Alhinai
@@ -0,0 +1,259 @@
|
||||
# Gemini Hacker Starter
|
||||
|
||||
A minimal starting point for building with **Gemini 3.1**, **NanoBanana 2**, and **Lyria RealTime** on LiveKit. Get a working multimodal agent running in under 10 minutes, then make it your own.
|
||||
|
||||
Built for the **Google DeepMind × YC Hackathon**.
|
||||
|
||||
---
|
||||
|
||||
## What's included
|
||||
|
||||
| Model | What it does in this starter |
|
||||
|---|---|
|
||||
| **Gemini 3.1 Flash Audio** | Real-time voice conversation with native audio and video understanding |
|
||||
| **NanoBanana 2** (`gemini-3.1-flash-image-preview`) | Generates images from text prompts — agent calls it as a function tool and sends the result to your browser |
|
||||
| **Lyria RealTime** (`models/lyria-realtime-exp`) | Streams generative music into the LiveKit room as a live audio track |
|
||||
|
||||
The agent can see your camera, hear you speak, generate images on demand, and play real-time music — all through a single LiveKit room.
|
||||
|
||||
---
|
||||
|
||||
## Install the LiveKit MCP server
|
||||
|
||||
Install this before you start. It gives your AI coding assistant direct access to LiveKit documentation so you get accurate, current help as you build.
|
||||
|
||||
**Cursor** — click to install:
|
||||
|
||||
[](https://cursor.com/en-US/install-mcp?name=livekit-docs&config=eyJ1cmwiOiJodHRwczovL2RvY3MubGl2ZWtpdC5pby9tY3AifQ%3D%3D)
|
||||
|
||||
Or add manually to your MCP settings:
|
||||
|
||||
```json
|
||||
{
|
||||
"livekit-docs": {
|
||||
"url": "https://docs.livekit.io/mcp"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Claude Code**
|
||||
|
||||
```bash
|
||||
claude mcp add --transport http livekit-docs https://docs.livekit.io/mcp
|
||||
```
|
||||
|
||||
**Gemini CLI**
|
||||
|
||||
```bash
|
||||
gemini mcp add --transport http livekit-docs https://docs.livekit.io/mcp
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.10–3.13
|
||||
- Node.js 18+
|
||||
- [uv](https://docs.astral.sh/uv/getting-started/installation/) (Python package manager)
|
||||
- LiveKit CLI:
|
||||
- macOS: `brew install livekit-cli`
|
||||
- Linux: `curl -sSL https://get.livekit.io/cli | bash`
|
||||
- Windows: `winget install LiveKit.LiveKitCLI`
|
||||
- [LiveKit Cloud account](https://cloud.livekit.io) (free)
|
||||
- Google API key with access to Gemini 3.1, NanoBanana 2, and Lyria
|
||||
|
||||
---
|
||||
|
||||
## Quick start
|
||||
|
||||
### 1. Set up the agent
|
||||
|
||||
```bash
|
||||
cd agent
|
||||
uv sync
|
||||
cp .env.example .env.local
|
||||
```
|
||||
|
||||
Edit `.env.local` with your credentials:
|
||||
|
||||
```env
|
||||
LIVEKIT_URL=wss://your-project.livekit.cloud
|
||||
LIVEKIT_API_KEY=your_key
|
||||
LIVEKIT_API_SECRET=your_secret
|
||||
GOOGLE_API_KEY=your_google_api_key
|
||||
```
|
||||
|
||||
Or use the LiveKit CLI to pull credentials from your cloud project automatically:
|
||||
|
||||
```bash
|
||||
lk cloud auth
|
||||
lk app env -w -d .env.local
|
||||
```
|
||||
|
||||
### 2. Set up the frontend
|
||||
|
||||
```bash
|
||||
cd ../frontend
|
||||
pnpm install
|
||||
cp .env.example .env.local
|
||||
```
|
||||
|
||||
Edit `frontend/.env.local`:
|
||||
|
||||
```env
|
||||
LIVEKIT_URL=wss://your-project.livekit.cloud
|
||||
LIVEKIT_API_KEY=your_key
|
||||
LIVEKIT_API_SECRET=your_secret
|
||||
```
|
||||
|
||||
Or use the LiveKit CLI:
|
||||
|
||||
```bash
|
||||
lk app env -w
|
||||
```
|
||||
|
||||
### 3. Run the agent
|
||||
|
||||
```bash
|
||||
cd agent
|
||||
uv run agent.py dev
|
||||
```
|
||||
|
||||
### 4. Run the frontend
|
||||
|
||||
In a new terminal:
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
pnpm dev
|
||||
```
|
||||
|
||||
Open [http://localhost:3000](http://localhost:3000), click **Start hacking**, and talk to your agent.
|
||||
|
||||
---
|
||||
|
||||
## Try it out
|
||||
|
||||
Once running, try these prompts:
|
||||
|
||||
- *"Generate an image of a neon-lit street at night in the style of a Studio Ghibli film"*
|
||||
- *"Play some calm ambient music"*
|
||||
- *"Stop the music"*
|
||||
- *"What do you see through my camera?"*
|
||||
- *"Generate a logo for a company called Quantum Noodle"*
|
||||
|
||||
---
|
||||
|
||||
## Customization
|
||||
|
||||
All the extension points are marked with `# HACK HERE:` comments in `agent/agent.py`. Here are the main ones.
|
||||
|
||||
### Change the agent's persona
|
||||
|
||||
Edit `PERSONA_INSTRUCTIONS` at the top of `agent/agent.py`:
|
||||
|
||||
```python
|
||||
PERSONA_INSTRUCTIONS = """You are a live sports commentator.
|
||||
Watch the game through the user's camera and provide real-time strategic analysis.
|
||||
Call out key moments, track the score, and keep energy high."""
|
||||
```
|
||||
|
||||
### Add a function tool
|
||||
|
||||
```python
|
||||
from livekit.agents import function_tool, RunContext
|
||||
|
||||
@function_tool()
|
||||
async def search_the_web(self, context: RunContext, query: str) -> str:
|
||||
"""Search the web for current information.
|
||||
|
||||
Args:
|
||||
query: The search query
|
||||
"""
|
||||
# your implementation here
|
||||
return "results..."
|
||||
```
|
||||
|
||||
### Adjust video frame rate
|
||||
|
||||
By default, video frames are sampled based on voice activity. For continuous commentary (e.g., watching a game), use a constant frame rate:
|
||||
|
||||
```python
|
||||
from livekit.agents import voice
|
||||
|
||||
session = AgentSession(
|
||||
llm=google.realtime.RealtimeModel(...),
|
||||
video_sampler=voice.VoiceActivityVideoSampler(speaking_fps=1.0, silent_fps=1.0),
|
||||
)
|
||||
```
|
||||
|
||||
### Swap the Gemini voice
|
||||
|
||||
Change the `voice` parameter in `agent.py`:
|
||||
|
||||
```python
|
||||
llm=google.realtime.RealtimeModel(
|
||||
model=REALTIME_MODEL,
|
||||
voice="Kore", # Options: Aoede, Charon, Fenrir, Kore, Puck
|
||||
)
|
||||
```
|
||||
|
||||
### Customize image generation
|
||||
|
||||
The `generate_image` tool in `HackathonAgent` sends the result as a data message to the frontend. You can extend it to:
|
||||
- Apply a style prefix to every prompt (e.g., always render in watercolor)
|
||||
- Send multiple images
|
||||
- Log prompts and images for a gallery view
|
||||
|
||||
### Customize Lyria music
|
||||
|
||||
The `start_music` tool accepts a `prompt` (text description) and `bpm`. You can extend it to expose more Lyria controls like `density`, `brightness`, and `scale`. See the [Lyria RealTime docs](https://ai.google.dev/gemini-api/docs/music-generation) for all available config options.
|
||||
|
||||
---
|
||||
|
||||
## Project ideas
|
||||
|
||||
These are just starting points. Build whatever seems interesting.
|
||||
|
||||
**Live foley engine** — Agent watches your video feed and generates matching ambient sounds and music in real time using Lyria. Point the camera at rain, a fire, a crowd — the agent creates a matching soundscape.
|
||||
|
||||
**Live game asset generator** — Sketch character designs or level layouts on paper, show them to the camera, and ask the agent to render polished versions using NanoBanana 2.
|
||||
|
||||
**Interactive storytelling** — Narrate a scene out loud. The agent listens, generates an image of what you describe, and plays mood-appropriate music — all simultaneously.
|
||||
|
||||
**Spatial design tool** — Point your camera at a room and describe how you'd redesign it. The agent generates photo-realistic renders of the redesigned space.
|
||||
|
||||
**Accessibility scene describer** — Agent watches a live video feed and generates detailed audio descriptions plus spatial soundscapes for visually impaired users.
|
||||
|
||||
**Real-time style transfer** — Capture frames from the camera, send them through the image model with style prompts, and stream the stylized output back to the screen continuously.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Frontend (Next.js + Agents UI)
|
||||
├── Microphone + camera → LiveKit room → agent receives audio/video
|
||||
├── Agent speech → LiveKit room → browser plays audio
|
||||
├── "generated-image" data message → browser renders image panel
|
||||
└── Lyria audio track → browser plays music
|
||||
|
||||
Agent (Python)
|
||||
├── Gemini 3.1 Flash Audio — realtime voice + vision
|
||||
├── generate_image tool → NanoBanana 2 → publish_data("generated-image")
|
||||
├── start_music tool → Lyria RealTime → publish AudioTrack
|
||||
└── stop_music tool → unpublish AudioTrack
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Resources
|
||||
|
||||
- [LiveKit Agents documentation](https://docs.livekit.io/agents/)
|
||||
- [Gemini Live API documentation](https://ai.google.dev/gemini-api/docs/live)
|
||||
- [Lyria RealTime documentation](https://ai.google.dev/gemini-api/docs/music-generation)
|
||||
- [Lyria RealTime cookbook](https://github.com/google-gemini/cookbook/blob/main/quickstarts/Get_started_LyriaRealTime.ipynb)
|
||||
- [LiveKit Cloud](https://cloud.livekit.io)
|
||||
- [Google AI Studio](https://aistudio.google.com)
|
||||
|
||||
Good luck — build something weird.
|
||||
Reference in New Issue
Block a user