feat: pod-wide Lyria background music (replaces test-audio drums)
The 'Test audio' button becomes 'Background Music': each pod gets a calm looping track from Gemini Lyria 3 that sings the pod name once up front then stays instrumental. Backend GET /api/pods/:id/music generates via the Gemini interactions endpoint and caches the MP3 per pod in Mongo (pod_music); the frontend fetches and loops it via Web Audio, published pod-wide on the existing podman-beat track. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Kartikeya
+52
-37
@@ -1,64 +1,79 @@
|
||||
# Shared Test Audio — pod-wide connectivity check
|
||||
# Shared Background Music — pod-wide audio + connectivity check
|
||||
|
||||
> Spec for the `frontend/src/livekit/useBeat.ts` + `PodView.tsx` test-audio
|
||||
> behavior and the additive `BEAT_STOP` data message. Satisfies the
|
||||
> Spec for the `frontend/src/livekit/useBeat.ts` + `PodView.tsx` background-music
|
||||
> behavior, the `lib/beat.ts` audio source, the `GET /api/pods/:id/music`
|
||||
> endpoint, and the additive `BEAT_STOP` data message. Satisfies the
|
||||
> documentation-first gate for those files.
|
||||
|
||||
## Why
|
||||
|
||||
The **Test audio** button is PodMan's pre-flight check that the LiveKit audio
|
||||
path works for the whole pod — the same path the urgent Gemini-TTS voice
|
||||
escalation rides on. Today the beat is published correctly but its on/off state
|
||||
is **local to the publisher**: teammates can't see it's playing and can't stop
|
||||
it. This makes it a shared, pod-wide toggle so a judge sees the state flip on
|
||||
every screen at once.
|
||||
The **Background Music** button is PodMan's pod-wide audio: a calm, looping
|
||||
background track unique to each pod, generated by Gemini **Lyria 3**. It opens
|
||||
with the pod's name sung once, then settles into a soft instrumental bed. It also
|
||||
doubles as the pre-flight check that the LiveKit audio path works for the whole
|
||||
pod — the same path the urgent Gemini-TTS voice escalation rides on. Its on/off
|
||||
state is shared pod-wide so a judge sees it flip on every screen at once.
|
||||
|
||||
## Behavior
|
||||
## Music generation (backend)
|
||||
|
||||
- Any participant clicks **Test audio** → they publish the `podman-beat` audio
|
||||
track (Web Audio, `lib/beat.ts`). Everyone auto-subscribes and hears it.
|
||||
- `GET /api/pods/:id/music` → streams the pod's background-music MP3
|
||||
(`audio/mpeg`). On first request it calls Lyria 3 (`lyria-3-clip-preview`) via
|
||||
the Gemini **interactions** endpoint with a prompt that sings the pod name in
|
||||
the first ~3s then stays instrumental, and **caches** the MP3 in the
|
||||
`pod_music` Mongo collection (keyed by pod id; regenerated if the pod name
|
||||
changes). Subsequent requests are instant. The Gemini key stays server-side.
|
||||
- `backend/src/voice/music.ts` owns generation + caching (`getPodMusic`).
|
||||
- Override the model with `GEMINI_MUSIC_MODEL` (default `lyria-3-clip-preview`).
|
||||
|
||||
## Behavior (frontend)
|
||||
|
||||
- Any participant clicks **Background Music** → the client fetches the pod's MP3
|
||||
and loops it (`lib/beat.ts` `startMusic`, Web Audio `AudioBufferSource.loop`),
|
||||
publishing it as the `podman-beat` track. Everyone auto-subscribes and hears
|
||||
it; the publisher hears it locally too.
|
||||
- The shared on/off state is **derived from the track's presence**, not a synced
|
||||
flag — so it self-syncs across joins/leaves and can't drift from reality. The
|
||||
publisher is the **owner**.
|
||||
flag — so it self-syncs across joins/leaves and can't drift. The publisher is
|
||||
the **owner**.
|
||||
- Anyone can stop it:
|
||||
- Owner clicks **Stop audio** → unpublishes its own track directly.
|
||||
- Non-owner clicks **Stop (`<owner>`'s)** → sends `BEAT_STOP`; the owner
|
||||
unpublishes. (LiveKit forbids unpublishing another participant's track, so a
|
||||
request is the only way.)
|
||||
- The Status card shows `publishing` / `<owner> playing` / `ready`, and the
|
||||
waveform animates (`active`) for everyone while the test is live.
|
||||
- Owner clicks **Stop music** → unpublishes its own track directly.
|
||||
- Non-owner clicks **Stop (`<owner>`)** → sends `BEAT_STOP`; the owner
|
||||
unpublishes. (LiveKit forbids unpublishing another participant's track.)
|
||||
- `PodView` warms the cache with a fire-and-forget fetch on mount so the first
|
||||
click plays instantly.
|
||||
|
||||
## State derivation (source of truth = the track)
|
||||
|
||||
`useBeat(room)` returns `{ on, by, mine }`, recomputed from the presence of a
|
||||
track named `podman-beat` across `localParticipant` + `remoteParticipants` on
|
||||
these events: `LocalTrackPublished/Unpublished`, `TrackPublished/Unpublished`,
|
||||
`useBeat(room, musicUrl)` returns `{ beat, toggleBeat }` where `beat` is
|
||||
`{ on, by, mine }`, recomputed from the presence of a track named `podman-beat`
|
||||
across `localParticipant` + `remoteParticipants` on these events:
|
||||
`LocalTrackPublished/Unpublished`, `TrackPublished/Unpublished`,
|
||||
`TrackSubscribed/Unsubscribed`, `ParticipantConnected/Disconnected`. Owner
|
||||
disconnect and late-join sync therefore need no extra messaging.
|
||||
|
||||
## Contract (additive)
|
||||
|
||||
`shared/src/messages.ts` — one new message on the existing `podman.intervention`
|
||||
data topic:
|
||||
|
||||
| { type: 'BEAT_STOP' } // any participant → owner: stop the shared beat
|
||||
|
||||
Additive to the `DataMessage` union; existing consumers ignore unknown types.
|
||||
**No backend / API change.**
|
||||
`shared/src/messages.ts` — `{ type: 'BEAT_STOP' }` on the existing
|
||||
`podman.intervention` data topic (any participant → owner: stop the shared
|
||||
track). Additive to the `DataMessage` union; existing consumers ignore unknown
|
||||
types.
|
||||
|
||||
## Known limitation (LiveKit constraint)
|
||||
|
||||
A client can only unpublish **its own** tracks, so a non-owner's **Stop** is a
|
||||
`BEAT_STOP` _request_ the owner must honor. If the owner disconnects **uncleanly**
|
||||
(crash / network drop), the SFU keeps the track published until it times the
|
||||
participant out — during that window the beat keeps playing and non-owners can't
|
||||
stop it. A clean disconnect clears it immediately via `ParticipantDisconnected`.
|
||||
Demo mitigation: have the same person who starts the test also stop it.
|
||||
participant out — during that window the music keeps playing and non-owners
|
||||
can't stop it. A clean disconnect clears it immediately via
|
||||
`ParticipantDisconnected`. Demo mitigation: have the same person who starts it
|
||||
also stop it.
|
||||
|
||||
## Files
|
||||
|
||||
- `backend/src/voice/music.ts` — Lyria generation + `pod_music` cache.
|
||||
- `backend/src/server.ts` — `GET /api/pods/:id/music` (streams MP3).
|
||||
- `frontend/src/lib/api.ts` — `podMusicUrl(id)` helper.
|
||||
- `frontend/src/lib/beat.ts` — `startMusic(url)` (loops the MP3); legacy
|
||||
`startBeat()` (synthesized kick/hat) kept as a fallback.
|
||||
- `frontend/src/livekit/useBeat.ts` — `useBeat(room, musicUrl)` hook.
|
||||
- `frontend/src/components/PodView.tsx` — button label + cache warm-up.
|
||||
- `shared/src/messages.ts` — `BEAT_STOP` message (additive).
|
||||
- `frontend/src/livekit/useBeat.ts` — `useBeat(room)` hook.
|
||||
- `frontend/src/components/PodView.tsx` — button label, status line, waveform
|
||||
`active` driven by the hook.
|
||||
- `frontend/src/lib/beat.ts` — unchanged (existing Web-Audio beat source).
|
||||
|
||||
Reference in New Issue
Block a user