Add research overlap nudges
This commit is contained in:
Yahya Alhinai
+50
-35
@@ -2,8 +2,8 @@
|
||||
|
||||
**Theme:** Continual Learning. **Hard limit:** 4:00. Practice to land at 3:45.
|
||||
|
||||
**The one-line story:** writing code isn't the bottleneck anymore — *coordinating
|
||||
who's writing what* is. PodMan is a pair programmer for the whole team: it watches
|
||||
**The one-line story:** writing code isn't the bottleneck anymore — _coordinating
|
||||
who's writing what_ is. PodMan is a pair programmer for the whole team: it watches
|
||||
every member's work in real time, gives everyone live status without anyone having
|
||||
to interrupt anyone, and learns your team's dynamics so it nudges less and helps
|
||||
more over time.
|
||||
@@ -25,7 +25,7 @@ recovery time across every teammate, every day, and that is the value.
|
||||
> everyone's work live, so anyone can see another's status without interrupting
|
||||
> them — and it learns your team as it goes."
|
||||
|
||||
*On screen:* the pod view, two teammates joined, screen-share tiles live.
|
||||
_On screen:_ the pod view, two teammates joined, screen-share tiles live.
|
||||
|
||||
### 0:30–1:05 — Real-time team awareness (LiveKit + Gemini Vision)
|
||||
|
||||
@@ -35,27 +35,41 @@ recovery time across every teammate, every day, and that is the value.
|
||||
symbol, activity — not a chatbot, a perception layer."
|
||||
- Show the live activity stream filling in (Signals vs Reasoning sections).
|
||||
- Land the value: "This is the part that replaces 'what are you working on?' —
|
||||
every teammate's current work is just *visible*, in real time. Nobody had to
|
||||
every teammate's current work is just _visible_, in real time. Nobody had to
|
||||
ask."
|
||||
|
||||
*Built-by-us callout:* `backend/src/vision/gemini.ts`, the LiveKit agent worker.
|
||||
_Built-by-us callout:_ `backend/src/vision/gemini.ts`, the LiveKit agent worker.
|
||||
|
||||
### 1:05–1:50 — The catch (detection + first intervention)
|
||||
### 1:05–1:40 — The catch (detection + first intervention)
|
||||
|
||||
- Have alice and bob both edit the **same file** with unpushed changes.
|
||||
- "Normally nobody notices until merge time. GitHub can't see this — nothing's
|
||||
pushed. Our detector fuses live screen context with **local git truth** from a
|
||||
watcher on each laptop."
|
||||
- A collision card appears: *"alice + bob both on detector.ts (unpushed)."*
|
||||
- Let the **Gemini TTS** urgent voice fire once over LiveKit: *"alice and bob are
|
||||
both editing detector.ts. Please sync before pushing."*
|
||||
- A collision card appears: _"alice + bob both on detector.ts (unpushed)."_
|
||||
- Let the **Gemini TTS** urgent voice fire once over LiveKit: _"alice and bob are
|
||||
both editing detector.ts. Please sync before pushing."_
|
||||
- Land the value: "That's a merge conflict and a wasted afternoon caught before it
|
||||
happened — and neither of them had to be tracking the other."
|
||||
|
||||
*Built-by-us callout:* `collision/detector.ts`, `action/hermes.ts`,
|
||||
_Built-by-us callout:_ `collision/detector.ts`, `action/hermes.ts`,
|
||||
`voice/live.ts`.
|
||||
|
||||
### 1:50–2:50 — Continual learning (the theme — the money shot)
|
||||
### 1:40–2:10 — Cross-channel overlap (research + code)
|
||||
|
||||
- Keep alice editing `livekit.py`.
|
||||
- Have bob share a browser tab on LiveKit docs/SDK pages.
|
||||
- A collaboration nudge appears: _"🤝 bob is researching LiveKit agents
|
||||
(docs.livekit.io) while alice edits livekit.py — sync up before duplicating
|
||||
effort."_
|
||||
- Land the value: "This is not a merge conflict. PodMan caught duplicated effort
|
||||
across channels — code on one screen, research on another — and nudged the team
|
||||
before two people solved the same problem twice."
|
||||
|
||||
_Built-by-us callout:_ `vision/gemini.ts`, `collision/research.ts`,
|
||||
`memory/vectors.ts`.
|
||||
|
||||
### 2:10–2:50 — Continual learning (the theme — the money shot)
|
||||
|
||||
This is the differentiator. Two beats, both from pre-seeded memory:
|
||||
|
||||
@@ -71,15 +85,15 @@ This is the differentiator. Two beats, both from pre-seeded memory:
|
||||
that adapts on the recalled outcome."
|
||||
- Optional: show `/api/memory/stats` counts climbing — accumulated experience.
|
||||
|
||||
*Built-by-us callout:* `memory/vectors.ts` ($vectorSearch), `memory/policy.ts`
|
||||
_Built-by-us callout:_ `memory/vectors.ts` ($vectorSearch), `memory/policy.ts`
|
||||
(outcome-conditioned gate), `memory/store.ts`.
|
||||
|
||||
### 2:50–3:30 — The five-minute meeting, killed (Gemini Live API)
|
||||
|
||||
- Frame it: "Instead of breaking a teammate's focus to ask what they're up to,
|
||||
you ask PodMan."
|
||||
- Open the live voice conversation. Ask out loud: *"PodMan, what is everyone
|
||||
working on, and where is the collision detector implemented?"*
|
||||
- Open the live voice conversation. Ask out loud: _"PodMan, what is everyone
|
||||
working on, and where is the collision detector implemented?"_
|
||||
- It answers with **real tool calls** — `search_repo`, git history, current
|
||||
collisions — not guesses.
|
||||
- "This is the **Gemini Live API**, streaming speech-to-speech over LiveKit, with
|
||||
@@ -87,7 +101,7 @@ This is the differentiator. Two beats, both from pre-seeded memory:
|
||||
and live state. That's the status sync, answered in seconds, with zero recovery
|
||||
tax on anyone else."
|
||||
|
||||
*Built-by-us callout:* `agents/podman-live-conversation/agent.py`.
|
||||
_Built-by-us callout:_ `agents/podman-live-conversation/agent.py`.
|
||||
|
||||
### 3:30–3:50 — Stack + close
|
||||
|
||||
@@ -106,23 +120,23 @@ This is the differentiator. Two beats, both from pre-seeded memory:
|
||||
|
||||
## Sponsor-prize coverage (say each at least once)
|
||||
|
||||
| Prize | Spoken moment | Segment |
|
||||
| --- | --- | --- |
|
||||
| **Gemini** | Vision perception, Live API agent w/ tools, TTS voice, Lyria score | 0:30, 1:05, 2:50, 3:30 |
|
||||
| **LiveKit** | "real screen shares over LiveKit", agent subscribes, TTS audio track, live voice | 0:30, 1:05, 3:30 |
|
||||
| **MongoDB** | "Atlas vector search recalling past events" | 1:50 |
|
||||
| **DigitalOcean** | "all on DigitalOcean, systemd-supervised workers" | 3:30 |
|
||||
| Prize | Spoken moment | Segment |
|
||||
| ---------------- | -------------------------------------------------------------------------------- | ---------------------- |
|
||||
| **Gemini** | Vision perception, Live API agent w/ tools, TTS voice, Lyria score | 0:30, 1:05, 2:50, 3:30 |
|
||||
| **LiveKit** | "real screen shares over LiveKit", agent subscribes, TTS audio track, live voice | 0:30, 1:05, 3:30 |
|
||||
| **MongoDB** | "Atlas vector search recalling past events" | 1:50 |
|
||||
| **DigitalOcean** | "all on DigitalOcean, systemd-supervised workers" | 3:30 |
|
||||
|
||||
---
|
||||
|
||||
## If something breaks (live recovery)
|
||||
|
||||
| Failure | Recovery |
|
||||
| --- | --- |
|
||||
| Voice doesn't fire | Cut to the card; say the line aloud; cards are the default path anyway. |
|
||||
| Live conversation drops | Skip 2:50–3:30; lean longer on the learning beat. |
|
||||
| Collision won't trigger | Use the backup recording for that beat; keep narrating. |
|
||||
| Agent flapping | Pre-checked — but if so, `systemctl restart podman-platform-agent`. |
|
||||
| Failure | Recovery |
|
||||
| ----------------------- | ----------------------------------------------------------------------- |
|
||||
| Voice doesn't fire | Cut to the card; say the line aloud; cards are the default path anyway. |
|
||||
| Live conversation drops | Skip 2:50–3:30; lean longer on the learning beat. |
|
||||
| Collision won't trigger | Use the backup recording for that beat; keep narrating. |
|
||||
| Agent flapping | Pre-checked — but if so, `systemctl restart podman-platform-agent`. |
|
||||
|
||||
**Rule:** never debug on stage. Narrate, fall back to recording, keep moving.
|
||||
|
||||
@@ -130,12 +144,13 @@ This is the differentiator. Two beats, both from pre-seeded memory:
|
||||
|
||||
## Tight timing summary
|
||||
|
||||
| Time | Beat |
|
||||
| --- | --- |
|
||||
| 0:00 | Problem (coordination cost) + hook + original-work line |
|
||||
| 0:30 | Real-time team awareness — LiveKit + Gemini Vision |
|
||||
| 1:05 | The catch — collision caught before merge |
|
||||
| 1:50 | **Continual learning — quiet + escalate** |
|
||||
| Time | Beat |
|
||||
| ---- | ---------------------------------------------------------- |
|
||||
| 0:00 | Problem (coordination cost) + hook + original-work line |
|
||||
| 0:30 | Real-time team awareness — LiveKit + Gemini Vision |
|
||||
| 1:05 | The catch — collision caught before merge |
|
||||
| 1:40 | Cross-channel overlap — research + code nudge |
|
||||
| 2:10 | **Continual learning — quiet + escalate** |
|
||||
| 2:50 | The five-minute meeting, killed — Gemini Live conversation |
|
||||
| 3:30 | DigitalOcean + Lyria + close |
|
||||
| 3:50 | Buffer |
|
||||
| 3:30 | DigitalOcean + Lyria + close |
|
||||
| 3:50 | Buffer |
|
||||
|
||||
@@ -33,14 +33,24 @@ screen-share track (not an HTTP upload — frames arrive over LiveKit).
|
||||
|
||||
```ts
|
||||
{
|
||||
mode: 'editing' | 'research', // browser/docs/SDK research vs editor work
|
||||
currentFile: string, // open file path, e.g. src/auth/session.ts
|
||||
currentSymbol: string, // function/class under the cursor
|
||||
activity: string, // editing | reading | debugging | terminal | PR review
|
||||
hasUnpushedChanges: boolean, // dirty git gutter / modified markers visible
|
||||
researchTopic: string, // e.g. "LiveKit agents setup", for research mode
|
||||
researchSource: string, // source domain, e.g. "docs.livekit.io"
|
||||
confidence: number // 0..1
|
||||
}
|
||||
```
|
||||
|
||||
When a frame shows a browser/docs/SDK page instead of an editor, Gemini Vision
|
||||
classifies it as `mode: "research"` and extracts the topic/source. That feeds the
|
||||
cross-channel overlap detector: one teammate researching LiveKit docs while
|
||||
another edits `livekit.py` becomes a collaboration nudge, not a merge-conflict
|
||||
alert. Editor/IDE frames remain `mode: "editing"` and use the existing file,
|
||||
symbol, activity, and dirty-change fields.
|
||||
|
||||
**Latency/cost levers (in code):**
|
||||
|
||||
- `thinkingConfig: { thinkingBudget: 0 }` — minimal thinking for the ambient loop.
|
||||
|
||||
@@ -26,6 +26,10 @@ intervention that fits:
|
||||
(`publishHermesIntervention` / `publishHermesMessage`). Default path.
|
||||
- **Urgent voice** — only for `critical` collisions. `speak()` generates Gemini
|
||||
TTS audio and publishes it as a LiveKit audio track.
|
||||
- **Research overlap nudge** — a collaboration card when one engineer is editing
|
||||
a file while another is researching the same topic in docs/browser context.
|
||||
This uses `suggestedAction.kind = "ping_teammate"` and is spoken once for the
|
||||
demo beat, but it is explicitly **not** a merge conflict.
|
||||
|
||||
Intervention text is short and deterministic (template, not an LLM call):
|
||||
`Conflict: alice + bob both on detector.ts (unpushed). Seen before.` The spoken
|
||||
@@ -33,6 +37,9 @@ line is phrased for natural TTS prosody. Each intervention is persisted to the
|
||||
`interventions` collection; the teammate's accept/dismiss returns via
|
||||
`POST /api/outcome`.
|
||||
|
||||
Research-overlap text is also deterministic:
|
||||
`🤝 bob is researching LiveKit agents (docs.livekit.io) while alice edits livekit.py — sync up before duplicating effort.`
|
||||
|
||||
A per-pod cooldown (`NUDGE_COOLDOWN_MS`, default 3 min) and a single-shot
|
||||
"active conflict" guard prevent repeat nagging; a conflict re-arms once it
|
||||
resolves.
|
||||
|
||||
@@ -69,6 +69,10 @@ Key fields:
|
||||
- `symbol`
|
||||
- `engineers`
|
||||
- `severity`
|
||||
- `overlapKind` — optional; `file`/undefined for same-file collisions,
|
||||
`research` for code-edit ↔ research overlaps.
|
||||
- `researchTopic`, `researchSource`, `researcher`, `editor` — optional fields
|
||||
present only for research overlaps.
|
||||
- `memorySignature`
|
||||
- `githubState`
|
||||
- `detectedAt`
|
||||
|
||||
Reference in New Issue
Block a user