Kartikeya
ab8ea07c12
The 'Test audio' button becomes 'Background Music': each pod gets a calm looping track from Gemini Lyria 3 that sings the pod name once up front then stays instrumental. Backend GET /api/pods/:id/music generates via the Gemini interactions endpoint and caches the MP3 per pod in Mongo (pod_music); the frontend fetches and loops it via Web Audio, published pod-wide on the existing podman-beat track. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
80 lines
4.0 KiB
Markdown
80 lines
4.0 KiB
Markdown
# Shared Background Music — pod-wide audio + connectivity check
|
|
|
|
> Spec for the `frontend/src/livekit/useBeat.ts` + `PodView.tsx` background-music
|
|
> behavior, the `lib/beat.ts` audio source, the `GET /api/pods/:id/music`
|
|
> endpoint, and the additive `BEAT_STOP` data message. Satisfies the
|
|
> documentation-first gate for those files.
|
|
|
|
## Why
|
|
|
|
The **Background Music** button is PodMan's pod-wide audio: a calm, looping
|
|
background track unique to each pod, generated by Gemini **Lyria 3**. It opens
|
|
with the pod's name sung once, then settles into a soft instrumental bed. It also
|
|
doubles as the pre-flight check that the LiveKit audio path works for the whole
|
|
pod — the same path the urgent Gemini-TTS voice escalation rides on. Its on/off
|
|
state is shared pod-wide so a judge sees it flip on every screen at once.
|
|
|
|
## Music generation (backend)
|
|
|
|
- `GET /api/pods/:id/music` → streams the pod's background-music MP3
|
|
(`audio/mpeg`). On first request it calls Lyria 3 (`lyria-3-clip-preview`) via
|
|
the Gemini **interactions** endpoint with a prompt that sings the pod name in
|
|
the first ~3s then stays instrumental, and **caches** the MP3 in the
|
|
`pod_music` Mongo collection (keyed by pod id; regenerated if the pod name
|
|
changes). Subsequent requests are instant. The Gemini key stays server-side.
|
|
- `backend/src/voice/music.ts` owns generation + caching (`getPodMusic`).
|
|
- Override the model with `GEMINI_MUSIC_MODEL` (default `lyria-3-clip-preview`).
|
|
|
|
## Behavior (frontend)
|
|
|
|
- Any participant clicks **Background Music** → the client fetches the pod's MP3
|
|
and loops it (`lib/beat.ts` `startMusic`, Web Audio `AudioBufferSource.loop`),
|
|
publishing it as the `podman-beat` track. Everyone auto-subscribes and hears
|
|
it; the publisher hears it locally too.
|
|
- The shared on/off state is **derived from the track's presence**, not a synced
|
|
flag — so it self-syncs across joins/leaves and can't drift. The publisher is
|
|
the **owner**.
|
|
- Anyone can stop it:
|
|
- Owner clicks **Stop music** → unpublishes its own track directly.
|
|
- Non-owner clicks **Stop (`<owner>`)** → sends `BEAT_STOP`; the owner
|
|
unpublishes. (LiveKit forbids unpublishing another participant's track.)
|
|
- `PodView` warms the cache with a fire-and-forget fetch on mount so the first
|
|
click plays instantly.
|
|
|
|
## State derivation (source of truth = the track)
|
|
|
|
`useBeat(room, musicUrl)` returns `{ beat, toggleBeat }` where `beat` is
|
|
`{ on, by, mine }`, recomputed from the presence of a track named `podman-beat`
|
|
across `localParticipant` + `remoteParticipants` on these events:
|
|
`LocalTrackPublished/Unpublished`, `TrackPublished/Unpublished`,
|
|
`TrackSubscribed/Unsubscribed`, `ParticipantConnected/Disconnected`. Owner
|
|
disconnect and late-join sync therefore need no extra messaging.
|
|
|
|
## Contract (additive)
|
|
|
|
`shared/src/messages.ts` — `{ type: 'BEAT_STOP' }` on the existing
|
|
`podman.intervention` data topic (any participant → owner: stop the shared
|
|
track). Additive to the `DataMessage` union; existing consumers ignore unknown
|
|
types.
|
|
|
|
## Known limitation (LiveKit constraint)
|
|
|
|
A client can only unpublish **its own** tracks, so a non-owner's **Stop** is a
|
|
`BEAT_STOP` _request_ the owner must honor. If the owner disconnects **uncleanly**
|
|
(crash / network drop), the SFU keeps the track published until it times the
|
|
participant out — during that window the music keeps playing and non-owners
|
|
can't stop it. A clean disconnect clears it immediately via
|
|
`ParticipantDisconnected`. Demo mitigation: have the same person who starts it
|
|
also stop it.
|
|
|
|
## Files
|
|
|
|
- `backend/src/voice/music.ts` — Lyria generation + `pod_music` cache.
|
|
- `backend/src/server.ts` — `GET /api/pods/:id/music` (streams MP3).
|
|
- `frontend/src/lib/api.ts` — `podMusicUrl(id)` helper.
|
|
- `frontend/src/lib/beat.ts` — `startMusic(url)` (loops the MP3); legacy
|
|
`startBeat()` (synthesized kick/hat) kept as a fallback.
|
|
- `frontend/src/livekit/useBeat.ts` — `useBeat(room, musicUrl)` hook.
|
|
- `frontend/src/components/PodView.tsx` — button label + cache warm-up.
|
|
- `shared/src/messages.ts` — `BEAT_STOP` message (additive).
|