diff --git a/CLAUDE.md b/CLAUDE.md index 75ea0cd..076f88b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,10 +1,12 @@ # CLAUDE.md ## Mission + You are the execution copilot for a 24-hour hackathon project with **12 hours max effective build time left**. Your job is not to be creative for creativity’s sake. Your job is to help the team **ship one technically impressive, demo-stable, judge-friendly project** under extreme time pressure. Optimize for: + - fast execution - technical depth that is visible in a 3-minute demo - originality beyond generic AI wrappers @@ -13,9 +15,10 @@ Optimize for: Do not encourage side quests, overengineering, or speculative future work. -*** +--- ## Hackathon context + Use these facts as hard constraints: - The project is for a hackathon focused on **self-improving AI, continual learning, agent infrastructure, and recursive intelligence**. @@ -32,9 +35,10 @@ Use these facts as hard constraints: - generic analyzers/coaches in banned categories - The project should feel like a **real agent system that improves from usage**, not a prompt demo. -*** +--- ## Required stack for this workspace + Assume the team is using: - **DigitalOcean** for hosting / deployment @@ -45,6 +49,7 @@ Assume the team is using: Prefer solutions that make these technologies central, not decorative. ### Stack roles + Use this default mapping unless explicitly changed: - **Gemini** @@ -70,9 +75,10 @@ Use this default mapping unless explicitly changed: If proposing architecture, keep it aligned with this stack. -*** +--- ## Project strategy rules + Always reason from these principles: 1. **One killer workflow beats five weak ones.** @@ -84,6 +90,7 @@ Always reason from these principles: 7. **Do not optimize for completeness. Optimize for a convincing end-to-end loop.** When asked for ideas, prefer: + - B2B or developer tooling - agent infrastructure - self-improving workflows @@ -92,18 +99,21 @@ When asked for ideas, prefer: - strong before/after demo moments Reject ideas that are: + - broad consumer apps - generic copilots - “chat with your data” tools - mostly static dashboards - impossible to build in 12 hours -*** +--- ## Execution mode + When helping in this workspace, be brutally practical. ### Always do these + - force prioritization - identify the shortest demoable path - separate **must-have**, **nice-to-have**, and **cut** @@ -112,39 +122,47 @@ When helping in this workspace, be brutally practical. - prefer mocked or constrained environments over flaky real-world integrations if demo reliability improves ### Never do these + - suggest large refactors unless absolutely necessary - encourage adding multiple product surfaces - recommend training a meaningful large model from scratch - pretend a weak feature is impressive - propose “future work” as if it helps judging -*** +--- ## Time constraint protocol + Assume **12 effective build hours remain** unless told otherwise. Every recommendation must pass this filter: ### Must-have test + Can this be built and demo-polished in <= 12 hours? If not, simplify or kill it. ### Priority buckets + Whenever planning features, organize into: #### Must-have demo path + The minimum end-to-end flow required for a winning demo. #### Nice-to-have + Useful only if the core path is done early. #### Cut immediately + Features that sound good but jeopardize shipping. Default behavior: if uncertain, put it in **Cut immediately**. -*** +--- ## Preferred architecture pattern + Unless there is a strong reason not to, bias toward this system shape: 1. **User provides a task / goal** @@ -158,6 +176,7 @@ Unless there is a strong reason not to, bias toward this system shape: This is the default “self-improving loop.” ### Good examples of improvement signals + - task success/failure - test pass rate - latency or step count @@ -166,7 +185,9 @@ This is the default “self-improving loop.” - human thumbs up/down only if necessary, but prefer automatic verification ### Best demo structure + A strong demo usually shows: + - Run 1 fails or is mediocre - System inspects memory / trace - System updates strategy @@ -174,12 +195,14 @@ A strong demo usually shows: That is more convincing than a static success case. -*** +--- ## Working style for code help + When asked to help code, optimize for momentum. ### Code generation rules + - Write small, composable files. - Prefer boring reliable frameworks over fancy ones. - Minimize moving parts. @@ -188,12 +211,14 @@ When asked to help code, optimize for momentum. - Add logging for every agent step, tool call, verifier result, and memory write. ### Debugging rules + - Start from the smallest failing surface. - Propose the most likely root cause first. - Give concrete fixes, not broad theories. - Prefer copy-pasteable commands and patches. ### UI rules + - UI exists to support the demo. - It should clearly show: - current task @@ -204,9 +229,10 @@ When asked to help code, optimize for momentum. - Do not build a dashboard-heavy product shell. - Avoid Streamlit. -*** +--- ## How to evaluate proposals + If asked whether something is a good idea, score it explicitly on: - **Problem Quality** — is the pain real and costly? @@ -218,9 +244,10 @@ If asked whether something is a good idea, score it explicitly on: If an idea is weak, say so directly and suggest a tighter version. -*** +--- ## Strong defaults for this workspace + Unless told otherwise, assume the team should build something in this family: - self-improving agent workflow @@ -231,9 +258,10 @@ Unless told otherwise, assume the team should build something in this family: Do not drift into generic SaaS CRUD apps. -*** +--- ## Demo-first development order + If asked what to build next, default to this sequence: 1. Define one narrow workflow @@ -247,9 +275,10 @@ If asked what to build next, default to this sequence: If something threatens steps 1-6, cut it. -*** +--- ## Communication style + Be direct, concise, and critical. Do not be sycophantic. Do not praise mediocre ideas. @@ -258,14 +287,17 @@ When there is risk, say exactly what the risk is. When there is a simpler path, recommend it. Preferred response pattern: + 1. verdict 2. why 3. what to do now -*** +--- ## If asked to choose between options + Default decision criteria: + - faster to demo - easier to explain - more visually impressive @@ -274,14 +306,15 @@ Default decision criteria: If two options are close, choose the one with the **better live demo**. -*** +--- ## Non-negotiable reminder + The team does **not** have time to build a platform. The team has time to build **one sharp, memorable, end-to-end loop**. Everything should serve that outcome. -*** +--- ## Documentation-first enforcement — HARD RULE @@ -299,7 +332,7 @@ This is not a guideline. This is a gate. **Do not write the code.** Instead: -1. Say explicitly: *"This isn't in the current plan. Let me understand what you're trying to do."* +1. Say explicitly: _"This isn't in the current plan. Let me understand what you're trying to do."_ 2. Ask what problem they're solving and whether it's required for the demo path. 3. Evaluate whether it fits within scope or replaces something planned. 4. If it's valid: **update `docs/PLAN.md` and the relevant spec first**, then proceed to code. @@ -317,7 +350,7 @@ This is not a guideline. This is a gate. 4 engineers are building simultaneously. If one person deviates from the plan, others build against wrong assumptions. An unplanned change to `backend/src/index.ts` can silently break another engineer's work mid-build. The docs are the contract between teammates — Claude's job is to enforce them, not work around them. -*** +--- ## Team context — 4 people working simultaneously @@ -329,4 +362,4 @@ This repo is actively used by **4 engineers at the same time**. Claude sessions - **Treat integration points as contracts.** The shared types in `shared/src/` and the API shapes of `POST /ingest`, `GET /pods/:podId/token`, and `GET /pods/:podId/state` are the interfaces between all teammates — do not change their signatures unilaterally. - **Flag merge risk explicitly** before editing a shared file (e.g., `backend/src/index.ts`, `frontend/src/App.tsx`). Say so, then proceed only if the user confirms. - **Prefer additive changes** — new files, new functions — over modifying existing ones. This minimizes merge conflicts in a concurrent team. -- **When proposing new files**, verify they match the file names listed in the relevant task in `docs/PLAN.md`. Do not invent new paths. \ No newline at end of file +- **When proposing new files**, verify they match the file names listed in the relevant task in `docs/PLAN.md`. Do not invent new paths. diff --git a/backend/src/agent.ts b/backend/src/agent.ts index 91f1741..cd2e6e6 100644 --- a/backend/src/agent.ts +++ b/backend/src/agent.ts @@ -43,7 +43,8 @@ async function main() { room.on( RoomEvent.TrackSubscribed, (track: RemoteTrack, pub: RemoteTrackPublication, participant: RemoteParticipant) => { - if (track.kind !== TrackKind.KIND_VIDEO || pub.source !== TrackSource.SOURCE_SCREENSHARE) return; + if (track.kind !== TrackKind.KIND_VIDEO || pub.source !== TrackSource.SOURCE_SCREENSHARE) + return; const id = participant.identity; const stream = new VideoStream(track); void (async () => { diff --git a/backend/src/agent/podman.ts b/backend/src/agent/podman.ts index d42cd70..b673248 100644 --- a/backend/src/agent/podman.ts +++ b/backend/src/agent/podman.ts @@ -26,9 +26,12 @@ export class PodMan { const msg = JSON.parse(new TextDecoder().decode(payload)) as DataMessage; if (msg.type === 'GIT_REPORT') { const c = this.contexts.get(msg.report.engineerId); - if (c) c.hasUnpushedChanges = msg.report.unpushedCount > 0 || msg.report.dirtyFiles.length > 0; + if (c) + c.hasUnpushedChanges = msg.report.unpushedCount > 0 || msg.report.dirtyFiles.length > 0; } - } catch { /* ignore malformed */ } + } catch { + /* ignore malformed */ + } }); } @@ -58,7 +61,8 @@ export class PodMan { await recordCollision(collision); const action = preferredAction(collision, prior); const names = collision.engineers.join(' and '); - const message = `${names} are both editing ${collision.file}` + + const message = + `${names} are both editing ${collision.file}` + (collision.githubState?.unpushed ? ' and one has unpushed changes.' : '.') + (prior ? ` I've seen this conflict pattern before.` : ''); @@ -75,10 +79,10 @@ export class PodMan { await recordIntervention(intervention); const data: DataMessage = { type: 'COLLISION', collision, intervention }; - await this.room.localParticipant?.publishData( - this.encoder.encode(JSON.stringify(data)), - { reliable: true, topic: DATA_TOPIC }, - ); + await this.room.localParticipant?.publishData(this.encoder.encode(JSON.stringify(data)), { + reliable: true, + topic: DATA_TOPIC, + }); await speak(this.room, message); // gemini-3.1-flash-live voice into the room } } diff --git a/backend/src/collision/detector.ts b/backend/src/collision/detector.ts index 7d09900..7b26f84 100644 --- a/backend/src/collision/detector.ts +++ b/backend/src/collision/detector.ts @@ -21,8 +21,7 @@ export function detectCollisions( const engineers = [...new Set(group.map((g) => g.engineerId))]; if (engineers.length < 2) continue; - const anyUnpushed = - group.some((g) => g.hasUnpushedChanges) || github.unpushed === true; + const anyUnpushed = group.some((g) => g.hasUnpushedChanges) || github.unpushed === true; if (!anyUnpushed) continue; // the crux GitHub alone cannot answer out.push({ diff --git a/backend/src/github/client.ts b/backend/src/github/client.ts index 0d0b66e..a725529 100644 --- a/backend/src/github/client.ts +++ b/backend/src/github/client.ts @@ -33,7 +33,12 @@ export async function createSyncPr(input: { headBranch: string; file: string; su const { owner, repo } = repoParts(); const { data: mainRef } = await gh.rest.git.getRef({ owner, repo, ref: 'heads/main' }); const branch = `podman-sync-${Date.now()}`; - await gh.rest.git.createRef({ owner, repo, ref: `refs/heads/${branch}`, sha: mainRef.object.sha }); + await gh.rest.git.createRef({ + owner, + repo, + ref: `refs/heads/${branch}`, + sha: mainRef.object.sha, + }); const { data: pr } = await gh.rest.pulls.create({ owner, repo, diff --git a/backend/src/memory/policy.ts b/backend/src/memory/policy.ts index 4693a37..e1de4f9 100644 --- a/backend/src/memory/policy.ts +++ b/backend/src/memory/policy.ts @@ -4,10 +4,7 @@ import type { Collision, SuggestedActionKind } from '@podman/shared'; * Policy gate: decides whether PodMan should intervene. * Stub: always intervene on warn/critical. */ -export function shouldIntervene( - collision: Collision, - _prior: unknown, -): boolean { +export function shouldIntervene(collision: Collision, _prior: unknown): boolean { return collision.severity !== 'info'; } @@ -15,9 +12,6 @@ export function shouldIntervene( * Preferred action selection based on collision + prior history. * Stub: open sync PR for critical, ping teammate otherwise. */ -export function preferredAction( - collision: Collision, - _prior: unknown, -): SuggestedActionKind { +export function preferredAction(collision: Collision, _prior: unknown): SuggestedActionKind { return collision.severity === 'critical' ? 'open_sync_pr' : 'ping_teammate'; } diff --git a/backend/src/memory/store.ts b/backend/src/memory/store.ts index c44b82e..b784bee 100644 --- a/backend/src/memory/store.ts +++ b/backend/src/memory/store.ts @@ -15,11 +15,15 @@ async function persist(name: string, fn: () => Promise): Promise } export async function recordObservation(ctx: EngineerContext): Promise { - await persist('observation', async () => (await collections()).observations.insertOne({ ...ctx })); + await persist('observation', async () => + (await collections()).observations.insertOne({ ...ctx }), + ); } export async function recordCollision(collision: Collision): Promise { - await persist('collision', async () => (await collections()).collisions.insertOne({ ...collision })); + await persist('collision', async () => + (await collections()).collisions.insertOne({ ...collision }), + ); } export async function recordIntervention(intervention: Intervention): Promise { diff --git a/backend/src/pods/store.ts b/backend/src/pods/store.ts index 27c9728..3c70be1 100644 --- a/backend/src/pods/store.ts +++ b/backend/src/pods/store.ts @@ -39,7 +39,8 @@ function cleanMembers(members: unknown): string[] { for (const raw of members) { if (typeof raw !== 'string') throw new Error('member names must be strings'); const name = raw.trim(); - if (name.length > MAX_MEMBER_LEN) throw new Error(`member name too long (max ${MAX_MEMBER_LEN})`); + if (name.length > MAX_MEMBER_LEN) + throw new Error(`member name too long (max ${MAX_MEMBER_LEN})`); const key = name.toLowerCase(); if (name && !seen.has(key)) { seen.add(key); diff --git a/backend/src/vision/gemini.ts b/backend/src/vision/gemini.ts index 7fe6fc7..f3aca84 100644 --- a/backend/src/vision/gemini.ts +++ b/backend/src/vision/gemini.ts @@ -7,13 +7,28 @@ const ai = new GoogleGenAI({ apiKey: env.GEMINI_API_KEY }); const SCHEMA = { type: Type.OBJECT, properties: { - currentFile: { type: Type.STRING, description: 'open file path if visible, e.g. src/auth/session.ts' }, + currentFile: { + type: Type.STRING, + description: 'open file path if visible, e.g. src/auth/session.ts', + }, currentSymbol: { type: Type.STRING, description: 'function/class under the cursor' }, - activity: { type: Type.STRING, description: 'editing | reading | debugging | terminal | PR review' }, - hasUnpushedChanges: { type: Type.BOOLEAN, description: 'dirty git gutter / modified markers visible' }, + activity: { + type: Type.STRING, + description: 'editing | reading | debugging | terminal | PR review', + }, + hasUnpushedChanges: { + type: Type.BOOLEAN, + description: 'dirty git gutter / modified markers visible', + }, confidence: { type: Type.NUMBER, description: '0..1 confidence in this read' }, }, - propertyOrdering: ['currentFile', 'currentSymbol', 'activity', 'hasUnpushedChanges', 'confidence'], + propertyOrdering: [ + 'currentFile', + 'currentSymbol', + 'activity', + 'hasUnpushedChanges', + 'confidence', + ], } as const; export async function analyzeFrame( @@ -27,7 +42,9 @@ export async function analyzeFrame( { role: 'user', parts: [ - { text: "You are PodMan watching an engineer's screen. Identify what file/symbol they are working on and whether there are uncommitted edits. JSON only." }, + { + text: "You are PodMan watching an engineer's screen. Identify what file/symbol they are working on and whether there are uncommitted edits. JSON only.", + }, { inlineData: { mimeType: 'image/jpeg', data: jpeg.toString('base64') } }, ], }, diff --git a/database/README.md b/database/README.md index f88a575..4ed6aed 100644 --- a/database/README.md +++ b/database/README.md @@ -6,12 +6,12 @@ This is what makes PodMan "more useful the more you use it" — the ownership ma ## Collections -| Collection | Holds | Notes | -|---|---|---| -| `engineer_states` | Latest `EngineerContext` per engineer | Upserted on every `/ingest` call | -| `ownership_map` | File → primary owner + contributors | Persists across sessions — the continual learning artifact | -| `events` | Detected coordination events (DEPENDENCY_READY, BLOCKER_DETECTED, DUPLICATE_WORK) | Append-only | -| `nudges` | Every voice nudge sent to the room | Used for cooldown checks (3 min between nudges) | +| Collection | Holds | Notes | +| ----------------- | --------------------------------------------------------------------------------- | ---------------------------------------------------------- | +| `engineer_states` | Latest `EngineerContext` per engineer | Upserted on every `/ingest` call | +| `ownership_map` | File → primary owner + contributors | Persists across sessions — the continual learning artifact | +| `events` | Detected coordination events (DEPENDENCY_READY, BLOCKER_DETECTED, DUPLICATE_WORK) | Append-only | +| `nudges` | Every voice nudge sent to the room | Used for cooldown checks (3 min between nudges) | Full schemas with indexes in [`docs/mongodb.md`](../docs/mongodb.md). diff --git a/database/init.ts b/database/init.ts index 9d394e6..72227d2 100644 --- a/database/init.ts +++ b/database/init.ts @@ -26,7 +26,9 @@ async function main() { name: 'vector_index', type: 'vectorSearch', definition: { - fields: [{ type: 'vector', path: 'embedding', numDimensions: 1024, similarity: 'cosine' }], + fields: [ + { type: 'vector', path: 'embedding', numDimensions: 1024, similarity: 'cosine' }, + ], }, }, ], @@ -38,4 +40,7 @@ async function main() { console.log('PodMan DB initialized.'); await client.close(); } -main().catch((e) => { console.error(e); process.exit(1); }); +main().catch((e) => { + console.error(e); + process.exit(1); +}); diff --git a/docs/demo-setup.md b/docs/demo-setup.md index 68af3c6..bc8fb75 100644 --- a/docs/demo-setup.md +++ b/docs/demo-setup.md @@ -18,18 +18,21 @@ Pre-stage checklist for the 3-minute live demo. Do this on all 3 laptops before ## Laptop setup (all 3 machines) ### Editor settings + - Font size: **18pt or larger** — Gemini Vision must read file names and code - Single editor window — no split panes, no overlapping terminals - File tab visible with full file name shown (not truncated) - Light or dark theme is fine — avoid low-contrast themes ### Browser + - Chrome (best `getDisplayMedia` support) - PWA tab open and joined to `demo-pod` - Earbuds / headphones plugged in and tested - Volume: medium — PodMan voice should be clearly audible but not startle ### Screen layout + - Editor takes 2/3 of screen - Terminal takes bottom 1/3 (always visible) - No other windows on top @@ -41,14 +44,17 @@ Pre-stage checklist for the 3-minute live demo. Do this on all 3 laptops before Pre-create these files in the demo repo before the demo: **Alice's machine:** + - Open `auth/middleware.ts` — has visible function stubs - Terminal shows nothing running initially, then `Server running on :3001` at the right moment **Bob's machine:** + - Open `frontend/login.tsx` — has visible form component code - Terminal idle **Carol's machine:** + - Open `frontend/integration.ts` or similar - Terminal shows: `curl http://localhost:3001/auth` → `curl: (7) Failed to connect` @@ -56,18 +62,18 @@ Pre-create these files in the demo repo before the demo: ## Demo script timing -| Time | Action | Who | -|---|---|---| -| 0:00 | All three join `demo-pod` | All | -| 0:05 | PodMan greets by voice | Hermes auto | -| 0:20 | Alice opens `auth/middleware.ts`, starts typing | Alice | -| 0:45 | Bob opens `frontend/login.tsx` | Bob | -| 0:50 | Carol runs `curl` command, sees error | Carol | -| ~1:20 | BLOCKER_DETECTED nudge fires | Hermes auto | -| 1:50 | Alice starts her server (`node server.js`) | Alice | -| ~2:00 | DEPENDENCY_READY nudge fires | Hermes auto | -| 2:20 | Optional: show session 2 ownership warm-start | Presenter | -| 2:45 | Close | Presenter | +| Time | Action | Who | +| ----- | ----------------------------------------------- | ----------- | +| 0:00 | All three join `demo-pod` | All | +| 0:05 | PodMan greets by voice | Hermes auto | +| 0:20 | Alice opens `auth/middleware.ts`, starts typing | Alice | +| 0:45 | Bob opens `frontend/login.tsx` | Bob | +| 0:50 | Carol runs `curl` command, sees error | Carol | +| ~1:20 | BLOCKER_DETECTED nudge fires | Hermes auto | +| 1:50 | Alice starts her server (`node server.js`) | Alice | +| ~2:00 | DEPENDENCY_READY nudge fires | Hermes auto | +| 2:20 | Optional: show session 2 ownership warm-start | Presenter | +| 2:45 | Close | Presenter | --- diff --git a/docs/digitalocean.md b/docs/digitalocean.md index a716537..5ec4c7e 100644 --- a/docs/digitalocean.md +++ b/docs/digitalocean.md @@ -114,6 +114,7 @@ static_sites: ## Fallback plan (if App Platform deploy fails on stage) Run Hermes locally: + ```bash cd backend && pnpm dev ``` diff --git a/docs/gemini.md b/docs/gemini.md index 1307f88..b2a57e0 100644 --- a/docs/gemini.md +++ b/docs/gemini.md @@ -104,6 +104,7 @@ Respond with the message text only. ``` **Example output:** + > "Carol — Alice just got the auth endpoint running. You're clear to integrate." --- @@ -115,6 +116,7 @@ Respond with the message text only. **Integration:** LiveKit Agents framework — Hermes runs as a LiveKit Agent with Gemini Live 2.5 as the voice provider **Flow:** + 1. Nudge message text generated (step 3) 2. Hermes passes text to Gemini Live via LiveKit Agents 3. Gemini Live streams audio back in real-time @@ -122,6 +124,7 @@ Respond with the message text only. 5. All participants hear it through their audio output **Why Gemini Live (not plain TTS):** + - Streams audio directly — no intermediate WAV file conversion - Latency ~300–500ms from text to first audio packet - Natural-sounding voice diff --git a/docs/idea.md b/docs/idea.md index 2d8164f..973ff7d 100644 --- a/docs/idea.md +++ b/docs/idea.md @@ -86,8 +86,8 @@ Each engineer opens a browser PWA on their laptop. The PWA captures a screen fra ## Prize alignment -| Prize | How PodMan earns it | -|---|---| -| Best Gemini 3.5 / 2.5 | Gemini Vision for screen understanding + Gemini Live 2.5 for voice output | -| Best LiveKit | LiveKit is the real-time backbone for room presence and voice delivery — load-bearing, not decorative | -| Best DigitalOcean | Hermes deployed on DigitalOcean App Platform; MongoDB Atlas on DO-adjacent infrastructure | +| Prize | How PodMan earns it | +| --------------------- | ----------------------------------------------------------------------------------------------------- | +| Best Gemini 3.5 / 2.5 | Gemini Vision for screen understanding + Gemini Live 2.5 for voice output | +| Best LiveKit | LiveKit is the real-time backbone for room presence and voice delivery — load-bearing, not decorative | +| Best DigitalOcean | Hermes deployed on DigitalOcean App Platform; MongoDB Atlas on DO-adjacent infrastructure | diff --git a/docs/livekit.md b/docs/livekit.md index a4a41b1..f32a107 100644 --- a/docs/livekit.md +++ b/docs/livekit.md @@ -16,17 +16,20 @@ LiveKit is the real-time backbone for PodMan. It handles room presence and voice ## Engineer side (PWA) **Joining:** + 1. PWA calls `POST /pods/:podId/token` → receives `{ token, url }` 2. LiveKit client connects to the room with the token 3. PWA publishes screen track via `getDisplayMedia` (used client-side for frame capture — Hermes does NOT subscribe to this track) 4. PWA sets mic enabled for ambient presence **Receiving:** + - LiveKit client automatically receives Hermes audio track - No special subscription needed — LiveKit delivers audio to all participants - PWA also listens for data channel messages from Hermes for UI card updates **Data channel listener (PWA):** + ```ts room.on(RoomEvent.DataReceived, (payload, participant) => { if (participant?.identity !== 'podman-hermes') return; @@ -43,17 +46,20 @@ room.on(RoomEvent.DataReceived, (payload, participant) => { **Framework:** LiveKit Agents (Node.js) **Startup:** + 1. Hermes mints its own token via the same `createPodToken` function with `identity: 'podman-hermes'` 2. Connects to the room on pod creation / first engineer joining 3. Registers as a LiveKit Agent with Gemini Live 2.5 as voice provider **Voice delivery:** + 1. Nudge message text is ready (from Gemini text generation) 2. Hermes passes text to Gemini Live 2.5 via LiveKit Agents voice pipeline 3. Audio streams into the room in real-time 4. All participants hear it **Data channel message (sent alongside audio):** + ```ts const nudge = { type: 'DEPENDENCY_READY' | 'BLOCKER_DETECTED' | 'DUPLICATE_WORK', @@ -75,6 +81,7 @@ room.localParticipant.publishData( Already implemented at `POST /pods/:podId/token`. Hermes uses the same endpoint. Grants: + - `roomJoin: true` - `canPublish: true` (for audio track) - `canPublishData: true` (for data channel) diff --git a/docs/mongodb.md b/docs/mongodb.md index d79f261..71b19bd 100644 --- a/docs/mongodb.md +++ b/docs/mongodb.md @@ -63,6 +63,7 @@ Tracks who works on which files. Built up over the session. **Persists across se **Index:** `{ podId: 1, file: 1 }` (unique) **Upsert logic:** + - On each context update where `currentFile` is non-null: - Increment `observationCount` - Update `primaryOwner` to the engineer with the most recent `lastSeenAt` on this file diff --git a/docs/superpowers/specs/2026-06-27-podman-design.md b/docs/superpowers/specs/2026-06-27-podman-design.md index b0678f0..a0df8a0 100644 --- a/docs/superpowers/specs/2026-06-27-podman-design.md +++ b/docs/superpowers/specs/2026-06-27-podman-design.md @@ -46,6 +46,7 @@ PodMan is a real-time AI team coordination agent for software teams. Engineers j ## Components ### PWA (local agent) + - Joins LiveKit room via existing `joinPod` flow - Captures frame every 30s via `getDisplayMedia`, compresses to JPEG (1280×720, quality 0.7) - POSTs `{ engineerId, podId, screenshotBase64, capturedAt }` to `POST /ingest` @@ -54,6 +55,7 @@ PodMan is a real-time AI team coordination agent for software teams. Engineers j - Two screens: join screen (built), active session screen (to build) ### Hermes (orchestrator) + - Express server + LiveKit Agent on DigitalOcean - `POST /ingest`: receives frame, queues for vision - Vision pipeline: Gemini 2.0 Flash → `EngineerContext` @@ -66,18 +68,21 @@ PodMan is a real-time AI team coordination agent for software teams. Engineers j - Cooldown: 3 min between nudges per pod ### Gemini usage + - **Vision:** `gemini-2.0-flash` — screen → `{ currentFile, inferredTask, terminalVisible, recentTerminalOutput, confidence }` - **Event detection:** `gemini-2.0-flash` — all engineer states → `{ event, involvedEngineers, file, reason }` - **Nudge generation:** `gemini-2.0-flash` — event → spoken message text - **Voice:** `gemini-live-2.5-flash` via LiveKit Agents — text → streaming audio ### MongoDB Atlas (4 collections) + - `engineer_states`: latest context per engineer, upserted each ingest - `ownership_map`: file → primaryOwner + contributors, persists across sessions (continual learning) - `events`: all detected coordination events - `nudges`: all voice nudges sent + cooldown history ### LiveKit + - One room per pod - Engineers publish screen track (used client-side for capture — Hermes does not subscribe) - Hermes joins as `podman-hermes`, publishes audio + data channel messages @@ -87,17 +92,18 @@ PodMan is a real-time AI team coordination agent for software teams. Engineers j ## Event types -| Event | Trigger | Example nudge | -|---|---|---| +| Event | Trigger | Example nudge | +| ------------------ | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | `BLOCKER_DETECTED` | Engineer stuck (error in terminal, same file N frames) + teammate can help | "Carol, looks like you're waiting on auth. Alice is actively building it — hang tight." | -| `DEPENDENCY_READY` | Engineer A completes work that Engineer B was waiting on | "Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate." | -| `DUPLICATE_WORK` | 2+ engineers on same file simultaneously | "Alice and Bob — you're both in login.tsx. Coordinate before pushing." | +| `DEPENDENCY_READY` | Engineer A completes work that Engineer B was waiting on | "Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate." | +| `DUPLICATE_WORK` | 2+ engineers on same file simultaneously | "Alice and Bob — you're both in login.tsx. Coordinate before pushing." | --- ## Continual learning story The `ownership_map` collection persists across sessions. On Hermes startup: + 1. Load ownership map for this pod from Atlas 2. Build in-memory cache: `Map` 3. Event detection uses priors immediately — no ramp-up phase @@ -120,10 +126,10 @@ The `ownership_map` collection persists across sessions. On Hermes startup: ## Key risks -| Risk | Mitigation | -|---|---| -| Gemini Vision accuracy | Large font, single editor window, confidence gate | -| Gemini Live 2.5 + LiveKit Agents integration | Build together hour 5–7, have TTS fallback | -| Frame POST latency | JPEG compression, target < 500ms | -| Event false positives | 3-min cooldown, pre-staged demo | -| DO deploy failure | Hermes runs local, PWA defaults to localhost:8787 | +| Risk | Mitigation | +| -------------------------------------------- | ------------------------------------------------- | +| Gemini Vision accuracy | Large font, single editor window, confidence gate | +| Gemini Live 2.5 + LiveKit Agents integration | Build together hour 5–7, have TTS fallback | +| Frame POST latency | JPEG compression, target < 500ms | +| Event false positives | 3-min cooldown, pre-staged demo | +| DO deploy failure | Hermes runs local, PWA defaults to localhost:8787 | diff --git a/eslint.config.mjs b/eslint.config.mjs index ca10783..7a3bec1 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -7,6 +7,17 @@ export default tseslint.config( }, js.configs.recommended, ...tseslint.configs.recommended, + { + files: ['scripts/**/*.mjs'], + languageOptions: { + globals: { + console: 'readonly', + process: 'readonly', + setTimeout: 'readonly', + URL: 'readonly', + }, + }, + }, { rules: { '@typescript-eslint/no-unused-vars': [ diff --git a/frontend/src/App.tsx b/frontend/src/App.tsx index 3ca1614..a7e2859 100644 --- a/frontend/src/App.tsx +++ b/frontend/src/App.tsx @@ -192,20 +192,11 @@ export default function App() { {joinedPod ? ( - + ) : showReconnecting ? (

Reconnecting to your pod…

-
diff --git a/frontend/src/components/PodView.tsx b/frontend/src/components/PodView.tsx index d22ffe7..a96372c 100644 --- a/frontend/src/components/PodView.tsx +++ b/frontend/src/components/PodView.tsx @@ -170,9 +170,7 @@ export function PodView({ onClick={toggleBeat} disabled={!room} className={`rounded-md px-4 py-2 text-sm font-medium disabled:opacity-50 ${ - playingBeat - ? 'bg-red-600 hover:bg-red-500' - : 'bg-emerald-600 hover:bg-emerald-500' + playingBeat ? 'bg-red-600 hover:bg-red-500' : 'bg-emerald-600 hover:bg-emerald-500' }`} > {playingBeat ? '⏹ Stop beat' : '▶ Play beat'} @@ -185,7 +183,9 @@ export function PodView({ {sharing ? '🛑 Stop sharing' : '📺 Share my screen'} - {playingBeat ? '🔊 broadcasting beat to the pod' : 'press “Play beat” — everyone should hear it'} + {playingBeat + ? '🔊 broadcasting beat to the pod' + : 'press “Play beat” — everyone should hear it'} {note &&

{note}

} diff --git a/frontend/src/livekit/useInterventions.ts b/frontend/src/livekit/useInterventions.ts index 9359fd7..28e6b9f 100644 --- a/frontend/src/livekit/useInterventions.ts +++ b/frontend/src/livekit/useInterventions.ts @@ -15,7 +15,9 @@ export function useInterventions(room: Room | null) { if (msg.type === 'COLLISION') setActive(msg.intervention); }; room.on(RoomEvent.DataReceived, onData); - return () => { room.off(RoomEvent.DataReceived, onData); }; + return () => { + room.off(RoomEvent.DataReceived, onData); + }; }, [room]); const respond = useCallback( diff --git a/frontend/src/livekit/useScreenPublish.ts b/frontend/src/livekit/useScreenPublish.ts index ec91311..e1e49fc 100644 --- a/frontend/src/livekit/useScreenPublish.ts +++ b/frontend/src/livekit/useScreenPublish.ts @@ -7,14 +7,17 @@ export function useScreenPublish() { const [connected, setConnected] = useState(false); const [sharing, setSharing] = useState(false); - const join = useCallback(async (pod: string, identity: string, name: string, githubLogin?: string) => { - const { token, url } = await fetchToken({ room: pod, identity, name, githubLogin }); - const room = new Room({ adaptiveStream: true, dynacast: true }); - await room.connect(url, token); - roomRef.current = room; - setConnected(true); - return room; - }, []); + const join = useCallback( + async (pod: string, identity: string, name: string, githubLogin?: string) => { + const { token, url } = await fetchToken({ room: pod, identity, name, githubLogin }); + const room = new Room({ adaptiveStream: true, dynacast: true }); + await room.connect(url, token); + roomRef.current = room; + setConnected(true); + return room; + }, + [], + ); const startSharing = useCallback(async () => { const room = roomRef.current;