docs: finalize PodMan architecture and write full integration specs
Replaces v1 plan with locked architecture: - Hermes orchestrator: POST /ingest → Gemini Vision → MongoDB → event detection → Gemini Live 2.5 voice via LiveKit Agents - Four MongoDB collections: engineer_states, ownership_map, events, nudges - Continual learning via ownership_map persisting across sessions New files: docs/idea.md, docs/gemini.md, docs/livekit.md, docs/mongodb.md, docs/digitalocean.md, docs/demo-setup.md Updated: README.md, docs/PLAN.md (12-hour build plan), database/README.md, infra/README.md, .env.example Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FFbfi4Cmb7BY75Wtne7bZn
This commit is contained in:
Ramis
+134
@@ -0,0 +1,134 @@
|
||||
# Gemini Integration Spec
|
||||
|
||||
PodMan uses Gemini for two distinct jobs: **vision** (understanding screens) and **voice** (speaking nudges).
|
||||
|
||||
---
|
||||
|
||||
## 1. Vision — Screen Understanding
|
||||
|
||||
**Model:** `gemini-2.0-flash` (fast, cheap, strong multimodal)
|
||||
|
||||
**Trigger:** every 30s per active engineer, when Hermes receives a `POST /ingest` frame
|
||||
|
||||
**Input:** base64-encoded JPEG, max 1280×720, ~50–80KB after compression
|
||||
|
||||
**Prompt:**
|
||||
|
||||
```
|
||||
You are analyzing a software engineer's screen during a coding session.
|
||||
Extract the following JSON. If you cannot determine a field with confidence above 0.7, set it to null.
|
||||
|
||||
{
|
||||
"currentFile": "string | null", // active file visible in editor tab or title bar
|
||||
"inferredTask": "string | null", // 1 sentence: what the engineer appears to be doing
|
||||
"terminalVisible": true | false, // is a terminal or CLI panel visible
|
||||
"recentTerminalOutput": "string | null", // last meaningful line of terminal output if visible
|
||||
"confidence": 0.0–1.0 // your overall confidence in this extraction
|
||||
}
|
||||
|
||||
Respond with valid JSON only. No explanation. No markdown.
|
||||
```
|
||||
|
||||
**Confidence gate:** if `confidence < 0.6`, Hermes discards the frame — no state update, no event detection triggered.
|
||||
|
||||
**Rate limit:** 1 call per engineer per 30s. With 3 engineers = 6 calls/min ≈ $0.002/min at Flash pricing.
|
||||
|
||||
**Demo setup requirement:** editors must have large font (18pt+), single window, file name clearly visible in tab. This is the primary reliability lever.
|
||||
|
||||
---
|
||||
|
||||
## 2. Event Detection — Coordination Awareness
|
||||
|
||||
**Model:** `gemini-2.0-flash` (text only, fast)
|
||||
|
||||
**Trigger:** after every successful state write to MongoDB, Hermes runs event detection over all active engineer contexts.
|
||||
|
||||
**Input:** JSON snapshot of all engineers' current states + ownership map
|
||||
|
||||
**Prompt:**
|
||||
|
||||
```
|
||||
You are a team coordination agent. Below is the current state of each engineer on the team.
|
||||
|
||||
Engineer states:
|
||||
{{engineerStates}}
|
||||
|
||||
Ownership map (who owns which files):
|
||||
{{ownershipMap}}
|
||||
|
||||
Detect if any of these coordination events are occurring:
|
||||
- DEPENDENCY_READY: an engineer who was blocked or waiting now has what they need because another engineer completed relevant work
|
||||
- BLOCKER_DETECTED: an engineer appears stuck (same file, error in terminal, no progress) and another teammate could help
|
||||
- DUPLICATE_WORK: two or more engineers are working on the same file simultaneously
|
||||
|
||||
If an event is detected, respond with:
|
||||
{
|
||||
"event": "DEPENDENCY_READY" | "BLOCKER_DETECTED" | "DUPLICATE_WORK" | null,
|
||||
"involvedEngineers": ["engineerId", ...],
|
||||
"file": "string | null",
|
||||
"reason": "1 sentence explanation"
|
||||
}
|
||||
|
||||
If no event, respond with { "event": null }.
|
||||
Respond with valid JSON only.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Nudge Generation — Voice Message
|
||||
|
||||
**Model:** `gemini-2.0-flash` (text only)
|
||||
|
||||
**Trigger:** when event detection returns a non-null event
|
||||
|
||||
**Input:** event type + engineer names + file + reason
|
||||
|
||||
**Prompt:**
|
||||
|
||||
```
|
||||
You are PodMan, a friendly AI teammate. Generate a short spoken message (1–2 sentences max) to notify the team about this coordination event.
|
||||
|
||||
Event: {{eventType}}
|
||||
Engineers involved: {{engineerNames}}
|
||||
File: {{file}}
|
||||
Context: {{reason}}
|
||||
|
||||
Rules:
|
||||
- Use first names only
|
||||
- Be direct and specific
|
||||
- Do not use filler words
|
||||
- Sound natural when spoken aloud
|
||||
- Do not start with "Hey" or "Attention"
|
||||
|
||||
Respond with the message text only.
|
||||
```
|
||||
|
||||
**Example output:**
|
||||
> "Carol — Alice just got the auth endpoint running. You're clear to integrate."
|
||||
|
||||
---
|
||||
|
||||
## 4. Voice Output — Gemini Live 2.5 via LiveKit
|
||||
|
||||
**Model:** `gemini-live-2.5-flash` (confirm exact ID from LiveKit Agents docs)
|
||||
|
||||
**Integration:** LiveKit Agents framework — Hermes runs as a LiveKit Agent with Gemini Live 2.5 as the voice provider
|
||||
|
||||
**Flow:**
|
||||
1. Nudge message text generated (step 3)
|
||||
2. Hermes passes text to Gemini Live via LiveKit Agents
|
||||
3. Gemini Live streams audio back in real-time
|
||||
4. LiveKit publishes audio into the pod room
|
||||
5. All participants hear it through their audio output
|
||||
|
||||
**Why Gemini Live (not plain TTS):**
|
||||
- Streams audio directly — no intermediate WAV file conversion
|
||||
- Latency ~300–500ms from text to first audio packet
|
||||
- Natural-sounding voice
|
||||
- Strong prize story: Gemini Live 2.5 is the headline model
|
||||
|
||||
---
|
||||
|
||||
## Cooldown
|
||||
|
||||
Per-pod cooldown of **3 minutes** between nudges. Prevents spam if multiple events fire simultaneously. Implemented in Hermes, not in Gemini.
|
||||
Reference in New Issue
Block a user