OSS/podman
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

chore: remove docs/generated, expand PLAN.md into full implementation plan

Browse Source 
- Remove all docs/generated/* (stale research output from prior workflow)
- Rewrite PLAN.md as ordered implementation plan: 10 tasks by dependency + demo criticality
- Each task has owner, estimate, dependencies, specific files to create/modify
- Cut line separates must-ship from nice-to-have

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFbfi4Cmb7BY75Wtne7bZn
This commit is contained in:
Ramis
2026-06-27 15:26:27 -07:00
parent 923ab1cf58
commit 34d923374a
19 changed files with 172 additions and 1681 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+4
View File
@@ -35,3 +35,7 @@ Thumbs.db
coverage/ coverage/
.cache/ .cache/
.turbo/ .turbo/
# Ramis
.remember/
.claude/
docs/.gitkeep
View File
docs/PLAN.md
+168 -104
View File
@@ -1,152 +1,216 @@
# PodMan — Build Plan (12 hours) # PodMan — Implementation Plan (12 hours)
> Replaces the original v1 plan. Architecture finalized. See `docs/idea.md` for concept, individual integration specs for details. > Architecture locked. See `docs/idea.md` for concept, integration specs for details.
> Tasks ordered by dependency and demo criticality. Never cut items above the cut line.
--- ---
## What we are building ## What we are building
PodMan is a real-time AI team coordination agent. Engineers join a LiveKit room with earbuds. Each engineer's browser PWA captures their screen every 30s and sends it to Hermes (server-side orchestrator). Hermes uses Gemini Vision to extract structured context, detects coordination events (dependency ready, blocker, duplicate work), and speaks proactive nudges into the room via Gemini Live 2.5. MongoDB Atlas stores team state and an ownership map that persists across sessions — making PodMan faster and smarter each session. Engineers join a LiveKit room with earbuds. Each engineer's browser PWA captures their screen every 30s and POSTs it to **Hermes** (server-side orchestrator on DigitalOcean). Hermes calls Gemini Vision to extract structured context per engineer, detects coordination events, generates a spoken nudge, and publishes it into the LiveKit room via Gemini Live 2.5. MongoDB Atlas stores state and an ownership map that persists across sessions.
**Hero demo moment:** PodMan detects Carol is blocked waiting for Alice's auth endpoint, warns Carol, then notifies Carol and Bob the moment Alice's server starts — without anyone sending a message. **Demo:** Alice builds auth endpoint. Carol is blocked. PodMan notices, warns Carol. Alice's server starts. PodMan tells Carol and Bob they're clear to integrate. No Slack. No asking.
--- ---
## Must-have demo path ## Critical path — must ship for demo
The minimum end-to-end flow required for a winning demo: ### 1. Environment + health check
**Owner:** Karti | **Est:** 1h | **Blocks:** everything
1. 3 engineers join a pod room via PWA (browser tab) - [ ] All `.env` vars populated: `LIVEKIT_*`, `GEMINI_API_KEY`, `MONGODB_URI`
2. PWA captures screen frame every 30s, POSTs to Hermes - [ ] `GET /health` returns `{ ok: true, service: 'podman-backend' }` *(already implemented)*
3. Hermes calls Gemini Vision → extracts `{ currentFile, inferredTask, confidence }` - [ ] MongoDB connection established in `backend/src/db.ts` — export `db` instance
4. Hermes writes to MongoDB (`engineer_states`, `ownership_map`) - [ ] `POST /ingest` stub returns `{ ok: true }` (no logic yet — unblocks parallel work)
5. Hermes runs event detection across all 3 engineers
6. `BLOCKER_DETECTED` or `DEPENDENCY_READY` event fires **Files:** `backend/src/db.ts` (new), `backend/src/index.ts` (add `/ingest` stub)
7. Hermes generates 1–2 sentence nudge via Gemini
8. Hermes speaks nudge into LiveKit room via Gemini Live 2.5
9. Engineers hear it through earbuds
10. Frontend shows live nudge feed (data channel card)
--- ---
## Nice-to-have (only if steps 1–10 done before hour 10) ### 2. PWA frame capture
**Owner:** Zander | **Est:** 1.5h | **Depends on:** task 1 stub
- `DUPLICATE_WORK` event detection - [ ] After joining pod, start capture loop: `setInterval` every 30s
- Ownership map cold-start demo (session 2 is visibly faster) - [ ] `getDisplayMedia` already running — grab frame from existing screen track via `ImageBitmap` → `OffscreenCanvas` → `toBlob('image/jpeg', 0.7)`
- GitHub state fusion (open PRs/branches per file) - [ ] Downscale to max 1280×720 before encoding
- Polish: teammate status cards in UI, confidence indicator - [ ] `POST /ingest` with `{ engineerId, podId, screenshotBase64, capturedAt }`
- [ ] Stop loop on disconnect
**Files:** `frontend/src/lib/capture.ts` (new), `frontend/src/lib/pod.ts` (start capture after connect)
---
### 3. MongoDB state layer
**Owner:** Karti | **Est:** 1.5h | **Depends on:** task 1
- [ ] `engineer_states` upsert: `db.collection('engineer_states').updateOne({ _id: engineerId }, { $set: ctx }, { upsert: true })`
- [ ] `ownership_map` upsert: called after each state write where `currentFile` is non-null
- [ ] `events` insert function
- [ ] `nudges` insert function + cooldown query (find any nudge for same `podId` in last `NUDGE_COOLDOWN_MS`)
- [ ] Load `ownership_map` on Hermes startup → build `Map<file, { primaryOwner, contributors }>`
**Files:** `backend/src/db/states.ts`, `backend/src/db/ownership.ts`, `backend/src/db/events.ts`, `backend/src/db/nudges.ts` (all new)
---
### 4. Gemini Vision pipeline
**Owner:** Ramis | **Est:** 2h | **Depends on:** tasks 1, 3
Wire `POST /ingest` fully:
- [ ] Receive `{ engineerId, podId, screenshotBase64, capturedAt }`
- [ ] Call `gemini-2.0-flash` with vision prompt (see `docs/gemini.md`) — inline image as base64
- [ ] Parse JSON response into `EngineerContext`
- [ ] Apply confidence gate: if `confidence < 0.6` → log and return early, no DB write
- [ ] On pass: call state upsert (task 3) → upsert `engineer_states` + `ownership_map`
- [ ] Trigger event detection (task 5) after every successful write
**Files:** `backend/src/vision/gemini.ts` (implement — currently stubbed), `backend/src/index.ts` (wire `/ingest` fully)
---
### 5. Event detector + nudge generator
**Owner:** Yahya | **Est:** 2h | **Depends on:** tasks 3, 4
- [ ] After each state write, fetch all `engineer_states` for the pod (only docs updated in last 2 min — stale engineers ignored)
- [ ] Call `gemini-2.0-flash` with event detection prompt (see `docs/gemini.md`) — pass all states + ownership map as JSON
- [ ] Parse `{ event, involvedEngineers, file, reason }` — if `event` is null, stop
- [ ] Check cooldown: query `nudges` for any sent in last `NUDGE_COOLDOWN_MS` for this pod — if found, skip
- [ ] Call `gemini-2.0-flash` with nudge generation prompt → get 1–2 sentence message
- [ ] Write event to `events` collection
- [ ] Pass message to voice publisher (task 6)
- [ ] Write nudge to `nudges` collection after sent
- [ ] Also publish data channel message for frontend card
**Files:** `backend/src/event/detector.ts` (new), `backend/src/intervention/engine.ts` (implement — currently stubbed)
---
### 6. Gemini Live 2.5 voice via LiveKit Agents
**Owner:** Everyone | **Est:** 2h | **Depends on:** tasks 1, 5
Highest integration risk — do as a team.
- [ ] Add LiveKit Agents SDK to backend (`@livekit/agents` or `livekit-server-sdk` agent support — confirm package)
- [ ] Hermes joins each active pod room as `podman-hermes` on first `/ingest` for that pod
- [ ] Wire Gemini Live 2.5 as voice provider (confirm model ID: `gemini-live-2.5-flash`)
- [ ] On nudge ready: pass text to Gemini Live → stream audio into room
- [ ] Also call `room.localParticipant.publishData(nudgePayload, { reliable: true })` for frontend card
- [ ] Fallback if Gemini Live fails: `@google/genai` TTS → WAV buffer → publish as audio track manually
**Files:** `backend/src/livekit/agent.ts` (new), `backend/src/intervention/engine.ts` (wire voice out)
---
### 7. PWA active session UI
**Owner:** Zander | **Est:** 1.5h | **Depends on:** tasks 2, 6
- [ ] Active session screen (post-join — replace current "Connected" placeholder)
- [ ] Teammate status cards: name, inferred file, inferred task — polled from backend via `GET /pods/:podId/state` or updated via data channel
- [ ] Data channel listener: on `RoomEvent.DataReceived` from `podman-hermes` → parse nudge → append to feed
- [ ] Nudge feed: last 5 nudges, timestamped, engineer names highlighted
- [ ] "PodMan is watching" indicator + frame capture active badge
**Files:** `frontend/src/components/SessionView.tsx` (new), `frontend/src/App.tsx` (render SessionView post-join)
---
## Cut line — below here only if hours 1–7 done before hour 10
### 8. Ownership warm-start demo
**Owner:** Karti | **Est:** 1h | **Depends on:** task 3
- [ ] On Hermes startup: log "Loading session memory for pod X — N files known"
- [ ] Ownership cache pre-populated before first frame arrives
- [ ] Demo: session 1 cold (3 min), session 2 warm (< 30s) — visible in logs + timing
---
### 9. `DUPLICATE_WORK` event type
**Owner:** Yahya | **Est:** 0.5h | **Depends on:** task 5
- [ ] Add to event detection prompt — already supported, just needs testing + nudge template
---
### 10. Backend state endpoint
**Owner:** Karti | **Est:** 0.5h | **Depends on:** task 3
- [ ] `GET /pods/:podId/state` → returns all `engineer_states` for the pod
- [ ] Used by PWA to populate teammate status cards (alternative to data channel push)
--- ---
## Cut immediately ## Cut immediately
- VS Code extension - VS Code extension
- Mic transcription - Mic transcription or voice input
- Manual task input fields - Manual task input fields on PWA
- Multilingual voice - Multilingual voice
- Full task management - GitHub API integration
- Slack / Linear integrations - Slack / Linear / Jira integrations
- Webcam tracks - Webcam tracks
- Always-on raw screen surveillance (we sample every 30s by design) - Voyage vector embeddings (plain MongoDB lookups sufficient for v1)
- Always-on raw screen surveillance (30s sampling is by design)
- Full task management features
- User auth / accounts
--- ---
## Team assignments ## Team assignments summary
| Person | Owns | Hours | | Person | Primary tasks | Hours |
|---|---|---| |---|---|---|
| **Karti** | MongoDB Atlas wiring, `engineer_states` + `ownership_map` upsert logic, DO deploy, `/health` + env setup | 3–4h | | **Karti** | 1 (env + health), 3 (MongoDB layer), 10 (state endpoint) | ~3–4h |
| **Ramis** | `POST /ingest` endpoint, Gemini Vision pipeline (`frameToContext`), confidence gate, frame compression in PWA | 3–4h | | **Ramis** | 4 (Gemini Vision pipeline), part of 2 (capture help) | ~3h |
| **Yahya** | Event detector (all 3 event types), nudge generator (Gemini text), cooldown logic, event + nudge MongoDB writes | 3–4h | | **Yahya** | 5 (event detector + nudge generator), 9 (duplicate work) | ~3h |
| **Everyone** | Gemini Live 2.5 + LiveKit Agents wiring (Hermes joins room + publishes voice) — highest integration risk, do together | 2h | | **Zander** | 2 (PWA frame capture), 7 (active session UI) | ~3h |
| **Everyone** | 6 (Gemini Live + LiveKit Agents voice) | ~2h |
--- ---
## Build order (strictly sequential by dependency) ## Risk table
### Hour 0–1: Plumbing (Karti)
- [ ] Confirm `.env` vars populated: `LIVEKIT_*`, `GEMINI_API_KEY`, `MONGODB_URI`
- [ ] `GET /health` returns `{ ok: true }`
- [ ] MongoDB connection established, collections initialized
- [ ] `POST /ingest` stub returns `{ ok: true }` (no logic yet)
### Hour 1–3: Frame capture + Vision pipeline (Ramis)
- [ ] PWA: `getDisplayMedia` frame capture every 30s → JPEG base64 (1280×720 max, quality 0.7)
- [ ] PWA: `POST /ingest` with `{ engineerId, podId, screenshotBase64, capturedAt }`
- [ ] Hermes: wire `@google/genai`, call `gemini-2.0-flash` with vision prompt
- [ ] Hermes: parse response, apply confidence gate (< 0.6 → discard)
- [ ] Hermes: upsert `engineer_states` in MongoDB
### Hour 1–3: MongoDB state layer (Karti, parallel with Ramis)
- [ ] `engineer_states` upsert function
- [ ] `ownership_map` upsert function (called after each `engineer_states` write)
- [ ] `events` insert function
- [ ] `nudges` insert function + cooldown query
### Hour 3–5: Event detection + nudge generation (Yahya)
- [ ] Hermes: after each state write, fetch all `engineer_states` for the pod
- [ ] Run Gemini event detection prompt → parse `{ event, involvedEngineers, file, reason }`
- [ ] On non-null event: check cooldown, generate nudge message via Gemini
- [ ] Write event + nudge to MongoDB
- [ ] Log to console (voice wiring comes next)
### Hour 3–5: PWA active session UI (Zander, parallel)
- [ ] Active session screen (post-join): "PodMan is watching" + teammate status cards
- [ ] Data channel listener: append nudge to live feed on receive
- [ ] Nudge feed: last 5 nudges, timestamped, engineer names highlighted
### Hour 5–7: Gemini Live 2.5 + LiveKit Agents voice (everyone)
- [ ] Install LiveKit Agents SDK in backend
- [ ] Hermes joins pod room as `podman-hermes` participant on startup
- [ ] Wire Gemini Live 2.5 as voice provider in LiveKit Agents
- [ ] On nudge ready: publish audio into room
- [ ] Also publish data channel message for frontend card
- [ ] Test: voice audible through browser audio output
### Hour 7–9: Integration + demo rehearsal
- [ ] Full end-to-end test: 3 browser tabs, screen share, Hermes processes frames, nudge fires, voice heard
- [ ] Pre-stage demo laptops: large font, clear file names, single editor window
- [ ] Run demo script 2× — fix any timing issues
- [ ] DO deploy (Karti) — verify `/health` live
### Hour 9–10: Ownership map demo (if time)
- [ ] Load `ownership_map` on Hermes startup
- [ ] Session 1 cold-start (3 min to first nudge) vs session 2 warm-start (< 30s)
- [ ] Add "Session memory loaded" log visible in demo
### Hour 10–12: Polish + backup plan
- [ ] Record a backup video of the demo working end-to-end
- [ ] Rehearse 3× with real audio
- [ ] Fallback: Hermes runs locally if DO deploy is flaky
---
## Open risks
| Risk | Mitigation | | Risk | Mitigation |
|---|---| |---|---|
| Gemini Vision accuracy on screens | Large font, single editor window, file name visible in tab. Confidence gate discards bad frames. | | Gemini Vision accuracy on screens | Large font (18pt+), single editor window, file tab fully visible. Confidence gate drops bad frames. |
| Gemini Live 2.5 + LiveKit Agents wiring is unknown territory | Allocate hour 5–7 as a team. Have fallback: plain HTTP TTS → WAV → LiveKit audio track. | | Gemini Live 2.5 + LiveKit Agents unknown territory | Build together (task 6). Fallback: Gemini TTS → WAV → publish manually as audio track. |
| Frame POST latency | Compress JPEG to quality 0.7, max 1280×720. Target < 500ms round trip. | | Frame POST latency | JPEG quality 0.7, max 1280×720. Target < 500ms round trip. |
| Event detection false positives | Cooldown (3 min between nudges). Pre-stage demo so events fire cleanly. | | Event detection false positives | 3-min cooldown per pod. Demo is pre-staged so events fire cleanly. |
| DO deploy fails on stage | Run Hermes local. PWA already defaults to `localhost:8787`. Zero demo impact. | | DO deploy fails on stage | Run Hermes local. PWA defaults to `localhost:8787` automatically. |
| Multiple events fire at once | Cooldown + event deduplication: same file + same engineers within 1 min → skip |
--- ---
## Demo script (3 min) ## Demo script (3 min)
**(0:00)** Three laptops visible. Alice, Bob, Carol join `demo-pod`. PodMan: *"PodMan online. I see Alice, Bob, and Carol. Let's build."* **(0:00)** Three laptops visible. Alice, Bob, Carol join `demo-pod` via PWA. PodMan: *"PodMan online. I see Alice, Bob, and Carol. Let's build."*
**(0:20)** Alice opens `auth/middleware.ts` (big font, clearly visible). Hermes processes first frame. Ownership map: Alice → auth. **(0:20)** Alice opens `auth/middleware.ts` (18pt font, clearly visible). First frame processed. Ownership map: Alice → auth.
**(0:45)** Bob opens `frontend/login.tsx`. Carol's terminal shows `curl: connection refused`. **(0:45)** Bob opens `frontend/login.tsx`. Carol runs `curl http://localhost:3001/auth` → `connection refused`.
**(1:20) MONEY MOMENT 1 — BLOCKER_DETECTED:** PodMan speaks: *"Carol, looks like you're waiting on auth. Alice is actively building it in middleware.ts — hang tight."* **(1:20) MONEY MOMENT 1 — BLOCKER_DETECTED:**
PodMan: *"Carol, looks like you're waiting on the auth endpoint. Alice is actively building it in middleware.ts — hang tight."*
**(1:50)** Alice's server starts (visible in terminal). Hermes detects transition. **(1:50)** Alice starts her server. Terminal shows `Server running on :3001`.
**(2:00) MONEY MOMENT 2 — DEPENDENCY_READY:** PodMan: *"Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate."* **(2:00) MONEY MOMENT 2 — DEPENDENCY_READY:**
PodMan: *"Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate."*
**(2:20)** Optional: show session 2 cold-start vs warm-start comparison. **(2:20)** Optional: show session 2 — Hermes logs "Session memory loaded — 3 files known." First nudge in 28s vs 3min in session 1.
**(2:45)** Close: *"PodMan — the teammate that sees what Slack can't."* **(2:45)** Close: *"PodMan — the teammate that sees what Slack can't."*
---
## Pre-demo checklist (day of)
See `docs/demo-setup.md` for full laptop setup. Key items:
- [ ] All 3 laptops: font 18pt+, single editor window, file tab visible
- [ ] `demo-pod` created in LiveKit Cloud
- [ ] Hermes `/health` returns OK on deployed URL
- [ ] Earbuds tested — voice audible through browser
- [ ] Backup video recorded and on separate device
- [ ] Demo rehearsed 3×
docs/generated/ARCHITECTURE.md
-208
View File
@@ -1,208 +0,0 @@
# PodMan — Architecture (workflow synthesis)
## Overview
PodMan is an ambient "Jarvis for engineering teams" — a Chrome PWA + Node agent that watches every pod member's shared screen in realtime, understands the work with Gemini 3.5 Flash vision, fuses that live (pre-push) signal with GitHub state, and proactively interrupts by voice + card to prevent merge collisions BEFORE anyone pushes. Track: Continual Learning — PodMan accumulates a persistent team world-model (who owns what, what's in-flight, recurring conflict hotspots) in MongoDB Atlas + Voyage vectors and refines its own intervention policy from outcomes, getting measurably better the longer it runs.
The monorepo is already scaffolded at /Users/karti/Desktop/Podman as a pnpm (v10.32) workspace, ESM, Node 22+, strict TS, with three packages (@podman/frontend, @podman/backend, @podman/shared). Shared types already exist and are good (Pod, Engineer, EngineerContext, Collision, GithubStateSnapshot, Intervention, SuggestedAction) — my design EXTENDS these rather than rewriting them. Everything below aligns with the existing tsconfig.base.json (moduleResolution: Bundler, verbatimModuleSyntax, isolatedModules — so all relative imports MUST use .js extensions and type-only imports MUST use `import type`).
THE ONE LOAD-BEARING ARCHITECTURE DECISION: the backend splits into TWO process entry points, not one. (1) backend/src/server.ts = a publicly-routable Express + ws service that mints LiveKit tokens and relays collision/intervention events to the PWA. (2) backend/src/agent.ts = an OUTBOUND LiveKit worker (no HTTP port) that joins each pod room via @livekit/rtc-node, subscribes to screen-share video tracks, grabs+throttles frames, runs Gemini vision, fuses with GitHub, detects collisions, and speaks back. This split exactly matches DigitalOcean App Platform's service-vs-worker model (research lane 4) and is THE thing that makes the DO deploy succeed instead of hanging on a health check.
The realtime vision path uses @livekit/rtc-node directly (NOT @livekit/agents) because Agents-JS integrated live-video input is Python-only as of June 2026 — this is the single biggest LiveKit gotcha and the design is built around it. gemini-3.5-flash with responseJsonSchema does per-frame screen->structured-context (throttled to ~1 fps); gemini-3.1-flash-live-preview does PodMan's voice (general voice — NOT the translate-only model); Octokit does deterministic git state + the real sync-PR creation (MCP has no compare tool); MongoDB Atlas + Voyage is the continual-learning memory + policy store.
## Realtime data flow
End-to-end, one frame's journey from an engineer's laptop to PodMan speaking:
1) CAPTURE (frontend, livekit-client). Each engineer's PWA calls createLocalScreenTracks({audio:true}) then setMicrophoneEnabled / setCameraEnabled, tagging the screen video with Track.Source.ScreenShare so the agent can distinguish it from the webcam. Tracks publish into ONE LiveKit room per pod. A consent gate + "PodMan is watching" indicator is shown before any track publishes (privacy-by-design, defuses the recording optics risk).
2) TOKEN (backend service, livekit-server-sdk). The PWA fetched a JWT from POST /api/token on server.ts; the engineer token carries canPublish+canSubscribe+canPublishData and metadata={githubLogin} for collision attribution. The agent has its own token with the same grants (it must canPublish to speak).
3) INGEST + FRAME GRAB (backend worker, @livekit/rtc-node — THE critical path). agent.ts joins the room (autoSubscribe:true). On RoomEvent.TrackSubscribed it filters pub.source===TrackSource.SOURCE_SCREENSHARE, wraps the RemoteVideoTrack in a VideoStream, async-iterates frames, THROTTLES to ~1 fps (frames arrive at up to 30fps — sending all of them blows cost/latency), converts each VideoFrame to RGBA (frame.convert(VideoBufferType.RGBA)), and encodes to downscaled JPEG with sharp (Gemini wants encoded bytes, not raw RGBA).
4) VISION -> STRUCTURED CONTEXT (gemini-3.5-flash). The JPEG goes to generateContent with responseMimeType:'application/json' + responseJsonSchema, returning a schema-valid EngineerContext-shaped object: { currentFile, currentSymbol, activity, hasUnpushedChanges, confidence }. mediaResolution 'low' (~280 tok/img) + thinkingConfig minimal keeps it cheap (~$0.05/min for 4 engineers) and fast. The per-engineer context is upserted into an in-memory map keyed by engineerId, and also persisted as an observation to MongoDB.
5) GITHUB FUSION (Octokit). In parallel, a cached GitHub poller keeps listCommits/listBranches and, on demand, compareCommitsWithBasehead (branch-vs-branch diff) and getContent (does the remote already have this file?). This is the deterministic half PodMan can trust.
6) COLLISION DETECTION (collision/detector.ts). The detector builds a map normalize(filePath) -> Set<engineerId> from live vision contexts. A collision fires when >=2 engineers point at the same file/feature AND >=1 has local-but-unpushed work — proven by EITHER Tier-1 vision (hasUnpushedChanges, or the file is absent from the remote branch per Octokit) OR an optional Tier-2 local git-reporter sidecar (git rev-list @{u}..HEAD count over the data channel). The "anyUnpushed" predicate is the crux the GitHub API alone literally cannot answer — the moat.
7) POLICY + MEMORY (continual learning). Before intervening, PodMan queries Atlas Vector Search (Voyage embedding of the collision signature) for "have we seen this pattern before?" and reads the learned policy (per-pattern confidence threshold + acceptance history). This both raises confidence ("I've seen session/webhook conflicts here, 0.93") and tunes whether/how to intervene (lead with the action the team accepted last time).
8) INTERVENTION OUT (voice + card). If policy says fire: (a) the agent composes an Intervention message and publishData(JSON, {reliable:true, topic:'podman.intervention', destination_identities:[...]}) so the PWA renders the card with the would-be diff; (b) gemini-3.1-flash-live-preview generates spoken audio that the agent publishes as a mic-source AudioTrack via AudioSource.captureFrame (engineers hear PodMan in the room). The card offers "Open sync PR" -> server calls Octokit createRef+pulls.create on the PUBLIC repo (real artifact). Optional flourish: re-render the same warning through gemini-3.5-live-translate-preview for a Spanish-speaking remote member.
9) OUTCOME ACK -> LOOP CLOSES. The engineer's accept/dismiss flows back over publishData; server.ts records {collisionId, wasRealCollision, acceptedPR} to MongoDB, which updates both the world-model and the policy weights — closing the continual-learning loop within the demo.
## Continual-learning loop
PodMan has TWO stacked learning loops so judges see "it gets smarter" twice in 3 minutes — this is the entire Continual Learning thesis made concrete and demonstrable.
LOOP A — TEAM WORLD-MODEL (knowledge accumulation, minimal supervision). Every vision observation is persisted to the `observations` collection in MongoDB Atlas: {engineerId, file, symbol, activity, hasUnpushedChanges, confidence, observedAt}. From these PodMan derives and continuously updates an `ownership` model (which engineer touches which files/dirs most -> de-facto owners) and a `conflict_hotspots` model (file-pairs that historically co-occur in collisions, e.g. auth/session.ts <-> auth/middleware.ts). Each collision signature (normalized files + symbols + involved roles) is embedded with Voyage AI (voyage-3 family) and stored in Atlas with a Vector Search index. At detection time PodMan runs $vectorSearch over past signatures: a near-neighbor hit RAISES detection confidence and lets PodMan explain WHY ("I've seen session and webhook handlers conflict in this repo"). The longer a session runs, the denser this memory, the earlier+more confident the catch — the literal "more useful the more it watches" track requirement, with no human labeling.
LOOP B — INTERVENTION POLICY (self-improvement from outcomes — the recursive-self-improvement flourish). Every intervention writes an `interventions` record; the engineer ACK (accepted / dismissed / "false alarm") flows back over the LiveKit data channel and is appended as the outcome. A lightweight policy in `memory/policy.ts` maintains, per collision-pattern cluster: (1) a confidence THRESHOLD that adapts up when interventions are dismissed (nag less) and down when they were real+accepted (catch earlier); (2) an action PREFERENCE (if the team accepted sync-PRs before, lead with that; if they prefer a ping, lead with that); (3) phrasing/verbosity. This is a simple, robust online update (running acceptance rate per cluster + threshold nudge) — deliberately NOT a fragile RL training job, so it's reliable on stage while still being a true outcome-driven self-improving policy. Optionally, the heavier "draft the merged sync patch + run tests" step runs OFF the realtime path in the Antigravity managed-agent sandbox (interactions API), kept off the latency-critical loop.
WHAT'S STORED (Atlas collections): pods, observations (TTL-indexed, high volume), collisions, interventions (with outcome), team_model (ownership + hotspots, one doc per pod, continuously upserted), memory_vectors (Voyage embeddings + Vector Search index for pattern recall), policy (per-pattern thresholds/preferences). The demo's "watch it get smarter" beat literally shows a memory_vectors doc written in beat 1 being retrieved in beat 2, plus the policy record reflecting the prior accepted PR.
## Components
### `frontend/` — React 19 + Vite 6 + TypeScript, installable PWA (vite-plugin-pwa), Tailwind, livekit-client
The engineer-facing Chrome PWA. Pod join (password/QR, mockable), consent gate + 'PodMan is watching' indicator, screen+mic+cam capture and publish via a useScreenPublish hook, receive interventions over the LiveKit data channel and render the proactive PodMan card (warning + would-be diff + 'Open sync PR' button), play PodMan's voice from its published audio track, and send accept/dismiss ACKs back. The pod grid is intentionally a small collapsed corner widget, never the hero.
Key libs: `react`, `react-dom`, `vite`, `@vitejs/plugin-react`, `vite-plugin-pwa`, `livekit-client`, `@podman/shared`
### `backend/` — Node 22 + TypeScript (ESM), split into TWO entry points: an Express+ws SERVICE (token mint + collision relay + PR action) and an OUTBOUND WORKER (LiveKit room subscriber + Gemini vision + collision detector + voice out)
server.ts (service, has http_port): POST /api/token (livekit-server-sdk), ws relay to the PWA, POST /api/sync-pr (Octokit createRef+pulls.create), outcome-ACK ingestion. agent.ts (worker, no port): join each pod room via @livekit/rtc-node, subscribe to SCREENSHARE tracks, throttle+grab frames (VideoStream/VideoFrame.convert -> sharp JPEG), gemini-3.5-flash vision -> EngineerContext, fuse with cached GitHub state, run the collision detector, query memory/policy, publishData the intervention card + publish gemini-3.1-flash-live-preview voice as an audio track. Submodules: vision/gemini.ts, github/client.ts, collision/detector.ts, agent/podman.ts (orchestrator loop), memory/{store,vectors,policy}.ts, voice/live.ts.
Key libs: `@livekit/rtc-node`, `livekit-server-sdk`, `@google/genai`, `octokit`, `sharp`, `express`, `ws`, `mongodb`, `@podman/shared`, `dotenv`
### `shared/` — TypeScript types only, compiled to dist, consumed by both frontend and backend as @podman/shared
Single source of truth for cross-cutting types. ALREADY contains Pod, Engineer, EngineerContext, Collision, CollisionSeverity, GithubStateSnapshot, Intervention, InterventionKind, InterventionStatus, SuggestedAction, SuggestedActionKind. Design EXTENDS this with: data-channel wire messages (DataMessage union: COLLISION / VOICE_CUE / ACK), an InterventionOutcome record, a TeamModel (ownership + hotspots) shape, and a LocalGitReport (optional Tier-2 sidecar payload). Keep it dependency-free.
Key libs: `typescript`
### `database/` — MongoDB Atlas (free Sandbox) + Voyage AI embeddings + Atlas Vector Search
Persistence + the continual-learning memory layer. Collections: pods, observations (TTL), collisions, interventions (with outcome), team_model, memory_vectors (Voyage vectors + $vectorSearch index), policy. Holds an init script that creates indexes including the vector index. This is what makes the Continual Learning claim TRUE, not hand-wavy.
Key libs: `mongodb`, `voyageai (or REST)`
### `infra/` — DigitalOcean App Platform (single app spec: static_site + service + worker), doctl, optional DO Gradient inference for a secondary LLM call
.do/app.yaml deploying all three components from the GitHub repo with deploy_on_push; auto-TLS + wss; secrets wired as RUN_TIME env. The LiveKit agent is a WORKER (no http_port/route) so the deploy doesn't hang on a health check; the token/relay API is a SERVICE; the PWA is a static_site. Optional: route the collision-summary/policy LLM call through DO Gradient (https://inference.do-ai.run/v1/) to legitimately claim DO inference for the Best DigitalOcean prize while keeping load-bearing vision on Gemini.
Key libs: `doctl`, `openai (for DO Gradient, OpenAI-compatible)`
### `docs/` — Markdown
PLAN.md (north star, already present) + this v2 architecture, the 3-minute demo script, the architecture slide, and the privacy/consent note. Read first by all teammates.
## File tree
```
Podman/
├── package.json # pnpm workspace root (EXISTS — add no deps here)
├── pnpm-workspace.yaml # frontend/backend/shared (EXISTS)
├── pnpm-lock.yaml # (EXISTS)
├── tsconfig.base.json # strict, Bundler res, verbatimModuleSyntax (EXISTS)
├── .nvmrc / .npmrc / .editorconfig # (EXIST)
├── eslint.config.mjs / .prettierrc # (EXIST)
├── .env.example # (EXISTS — extend per env-vars section)
├── README.md # (EXISTS)
│
├── shared/ # @podman/shared (types only)
│ ├── package.json tsconfig.json # (EXIST)
│ └── src/
│ ├── index.ts # barrel (EXISTS — extend exports)
│ ├── pod.ts engineer.ts # (EXIST)
│ ├── collision.ts intervention.ts # (EXIST)
│ └── messages.ts # NEW: DataMessage wire union, InterventionOutcome, TeamModel, LocalGitReport
│
├── frontend/ # @podman/frontend (React + Vite PWA)
│ ├── package.json
│ ├── tsconfig.json
│ ├── vite.config.ts # + vite-plugin-pwa
│ ├── index.html
│ └── src/
│ ├── main.tsx # React entry
│ ├── App.tsx # join -> capture -> PodMan card host
│ ├── livekit/
│ │ ├── useRoom.ts # connect + token fetch
│ │ ├── useScreenPublish.ts # screen+mic+cam publish hook (STARTER)
│ │ └── useInterventions.ts # DataReceived -> card state, voice playback
│ ├── components/
│ │ ├── ConsentGate.tsx # opt-in + "PodMan is watching"
│ │ ├── PodGridCorner.tsx # collapsed peripheral-vision widget (NOT hero)
│ │ └── InterventionCard.tsx # warn + diff + Open-sync-PR (the hero)
│ └── lib/api.ts # POST /api/token, /api/sync-pr
│
├── backend/ # @podman/backend (Node agent + service)
│ ├── package.json
│ ├── tsconfig.json
│ └── src/
│ ├── env.ts # typed env loader (zod-lite)
│ ├── server.ts # SERVICE: token + ws relay + sync-PR + ACK (STARTER)
│ ├── agent.ts # WORKER: room subscribe + frame grab loop (STARTER)
│ ├── agent/
│ │ └── podman.ts # orchestrator: vision->fuse->detect->intervene (STARTER)
│ ├── vision/
│ │ └── gemini.ts # gemini-3.5-flash frame -> EngineerContext (STARTER)
│ ├── voice/
│ │ └── live.ts # gemini-3.1-flash-live voice -> PCM -> AudioFrame
│ ├── github/
│ │ └── client.ts # Octokit: state + compare + create sync PR (STARTER)
│ ├── collision/
│ │ └── detector.ts # fusion rule: same file + >=1 unpushed (STARTER)
│ └── memory/
│ ├── store.ts # Mongo collections + writes
│ ├── vectors.ts # Voyage embed + Atlas $vectorSearch
│ └── policy.ts # outcome-driven threshold/preference learning
│
├── database/
│ └── init.ts # create collections + indexes + vector index (STARTER)
│
├── infra/
│ └── .do/app.yaml # DO App Platform: static_site + service + worker (STARTER)
│
├── tools/
│ └── git-reporter.mjs # OPTIONAL Tier-2 local sidecar (git status -> agent)
│
└── docs/
├── PLAN.md # (EXISTS)
├── ARCHITECTURE.md # this v2 design
└── DEMO.md # 3-min script + architecture slide
```
## Env vars
- `LIVEKIT_URL` — wss URL of your LiveKit Cloud project (e.g. wss://podman-xxx.livekit.cloud). Used by both the agent (to join rooms) and returned to the PWA with each token.
- `LIVEKIT_API_KEY` — LiveKit API key for minting JWTs (server) and authing the agent worker.
- `LIVEKIT_API_SECRET` — LiveKit API secret paired with the key. Server-side only; never ship to the PWA.
- `GEMINI_API_KEY` — Google AI Studio key for @google/genai. Backend-only — must NOT be exposed in the Vite frontend. Powers vision (3.5-flash) and voice (3.1-flash-live).
- `GEMINI_VISION_MODEL` — Vision model id for per-frame screen understanding. Default gemini-3.5-flash (GA, supports responseJsonSchema).
- `GEMINI_LIVE_MODEL` — General voice model for PodMan speaking. Default gemini-3.1-flash-live-preview. NOT the translate-only gemini-3.5-live-translate-preview (reserve that for the optional Spanish flourish).
- `GITHUB_TOKEN` — Fine-grained PAT for the PUBLIC demo repo. Scopes: Contents R, Pull requests R+W, Metadata R. Used by Octokit for state, compare, and creating the real sync PR.
- `GITHUB_REPO` — Target repo as owner/name. MUST be public per hackathon rules; show its github.com URL during the demo.
- `MONGODB_URI` — MongoDB Atlas Sandbox connection string. Holds the continual-learning memory: observations, collisions, interventions+outcomes, team_model, memory_vectors, policy.
- `VOYAGE_API_KEY` — Voyage AI key for embedding collision signatures (voyage-3, 1024-dim) used by Atlas Vector Search for pattern recall. Optional but needed for the 'it got smarter' beat.
- `PORT` — Port the backend SERVICE binds (default 8787). On DO App Platform this is http_port and must bind 0.0.0.0, not localhost.
- `POD_ROOM` — Room name the agent worker joins (default demo-pod). One room per pod; for the demo a single fixed room is fine.
- `VITE_BACKEND_URL` — Frontend-only: base URL of the backend service for token/sync-pr/outcome calls. Must be VITE_ prefixed to reach the client bundle.
- `VITE_LIVEKIT_URL` — Frontend-only fallback LiveKit URL (the token response also returns url; this is for display/dev). VITE_ prefixed.
- `MODEL_ACCESS_KEY` — OPTIONAL — DigitalOcean Gradient inference key (OpenAI-compatible, https://inference.do-ai.run/v1/). Route a secondary call (collision summary / policy reasoning) through DO to legitimately claim Best DigitalOcean inference; keep load-bearing vision on Gemini.
## MVP build order
1. **0. Verify plumbing + add deps (30 min, Karti)** — Repo already scaffolds with pnpm. Add deps per package: backend (@livekit/rtc-node, livekit-server-sdk, @google/genai, octokit, sharp, express, ws, mongodb, dotenv, @types/ws); frontend (react, react-dom, vite, @vitejs/plugin-react, vite-plugin-pwa, livekit-client). Add shared/src/messages.ts and wire its exports into index.ts. Confirm `pnpm -r build` and `pnpm -r typecheck` pass with the existing strict/Bundler tsconfig (remember .js import extensions + import type).
2. **1. Token + join skeleton (1h, Karti + Zander)** — Stand up backend/src/server.ts (token endpoint + /health + ws relay) and the frontend join flow with useScreenPublish. Prove an engineer can join a LiveKit room and publish a screen track tagged Source.ScreenShare. Smoke test against LiveKit Cloud locally over https/localhost (getDisplayMedia needs secure context).
3. **2. THE CRITICAL PATH — frame grab (2-3h, Ramis) — DE-RISK FIRST** — Build agent.ts: join the room with @livekit/rtc-node, filter SCREENSHARE, VideoStream -> throttle 1fps -> VideoFrame.convert(RGBA) -> sharp JPEG. Just log that frames arrive and are encoded. This is the single hardest/riskiest piece (Agents-JS video is Python-only) so build it before anything else depends on it.
4. **3. Eyes — Gemini vision (1-2h, Ramis)** — Wire vision/gemini.ts: JPEG -> gemini-3.5-flash responseJsonSchema -> EngineerContext. Log structured context per engineer. Tune mediaResolution low + thinkingBudget 0 for latency. This is the on-screen 'live inference caption' for the demo.
5. **4. Brain + collision (2h, Yahya)** — Implement collision/detector.ts (pure fusion rule) + github/client.ts (state + compare + remoteHasFile). Wire agent/podman.ts to fuse vision contexts with GitHub state and emit a Collision when 2 engineers + >=1 unpushed. Unit-test the detector with two hardcoded contexts so the money moment is reproducible.
6. **5. Intervention out — card + voice + PR (2-3h, Yahya + Zander)** — Agent publishData the COLLISION DataMessage; frontend useInterventions renders the hero card with the would-be diff. Add voice/live.ts (gemini-3.1-flash-live -> PCM -> AudioFrame.captureFrame) so PodMan speaks in-room. Wire the card's 'Open sync PR' to POST /api/sync-pr -> Octokit creates a real PR on the public repo. This is the hero beat end-to-end.
7. **6. Continual learning — memory + policy (2-3h, Karti)** — memory/store.ts (Mongo writes), memory/vectors.ts (Voyage embed + Atlas $vectorSearch recall), memory/policy.ts (outcome-driven threshold/preference). Run database/init.ts. Make the 'it got smarter' beat real: write a memory_vectors doc in collision 1, retrieve it in collision 2 to raise confidence + lead with the previously-accepted action.
8. **7. Deploy on DO + rehearse (2-3h, all)** — doctl apps create --spec infra/.do/app.yaml (static_site + service + worker). Set secrets. Confirm wss + /health. Show the public DO URL + the github.com repo URL on screen. Rehearse the 3-min demo 3x with a recorded fallback of the money moment. Keep a hot local fallback in case venue wifi dies, but run live.
## Team split (max 4)
- **Karti** — Principal/infra: monorepo + deps + shared/messages.ts, backend service (server.ts: token + ws relay + sync-PR + outcome), the continual-learning memory layer (memory/store.ts, vectors.ts, policy.ts + database/init.ts), and the DigitalOcean deploy (infra/.do/app.yaml, doctl, secrets). Owns the 'it got smarter' proof.
- **Ramis** — Realtime vision critical path: agent.ts (LiveKit room subscribe + frame grab via @livekit/rtc-node, throttle, sharp encode) and vision/gemini.ts (gemini-3.5-flash -> EngineerContext). This is the hardest, highest-risk piece — owned start to finish, de-risked first.
- **Yahya** — PodMan brain + action: collision/detector.ts (fusion rule), github/client.ts (Octokit state/compare/sync-PR), agent/podman.ts orchestrator, voice/live.ts (gemini-3.1-flash-live voice into the room). Owns the money-moment logic and the real PR artifact.
- **Zander** — Frontend PWA: join + consent gate + 'PodMan is watching' indicator, useScreenPublish/useInterventions hooks, the hero InterventionCard (warn + diff + Open-sync-PR), voice playback, and the collapsed PodGridCorner (kept off-hero). Drives the demo UX and PWA install.
## Risks
- **LiveKit Agents-JS integrated live-VIDEO input is Python-only — architecting the screen-vision around the Node AgentSession video helper would dead-end the whole project.** → Design already uses @livekit/rtc-node's VideoStream/VideoFrame directly (see agent.ts), which is fully supported in Node. Build and prove this path in step 2 before anything depends on it. Fallback: run ONLY the vision worker in Python (Agents-JS Python has the integrated path) while keeping the rest of the backend in Node.
- **Realtime vision cost/latency blows up if every frame (up to 30fps x 4 engineers) is sent to Gemini.** → Hard throttle to ~1fps per engineer in agent.ts, downscale to 1280px JPEG via sharp, use mediaResolution LOW (~280 tok/img) and thinkingBudget 0. ~$0.05/min for 4 engineers. Buffer latest frame; only re-infer on interval.
- **'Unpushed local changes' is the moat but is fuzzy from vision alone — a flaky read undercuts the hero beat and the Technicality score.** → Two-tier: Tier-1 vision hasUnpushedChanges + remoteHasFile() inference; Tier-2 optional local git-reporter sidecar (tools/git-reporter.mjs) posting `git rev-list @{u}..HEAD` over the data channel for ground truth. Ship Tier-2 for demo reliability; the detector accepts either signal.
- **Dashboard-as-hero DQ: the pod grid of live screen tiles is on the explicit banned list.** → Grid is a small collapsed corner widget (PodGridCorner), shown for ~3-5s then collapsed; the hero is the proactive InterventionCard + voice. Say 'PodMan is an agent, not a dashboard — screens are just its eyes' out loud. Demo opens on a single IDE, not the grid.
- **DO App Platform deploy hangs if the LiveKit agent is configured as a service (health check on a port nothing listens on); or wss fails via ws:// mixed content.** → app.yaml puts the agent as a WORKER (no http_port/route) and the API as a service binding 0.0.0.0:8787; frontend selects wss when location.protocol is https. Both are baked into the starter files.
- **Wrong Gemini model ids: using the translate-only model as general voice, or assuming Computer Use needs a separate model.** → Voice = gemini-3.1-flash-live-preview (general); translate model reserved for the optional Spanish flourish only; vision/PR action use 3.5-flash + Octokit (no Computer Use on the hot path). Model ids are env-driven so they're swappable if previews move.
- **On-stage wifi failure or LiveKit/Gemini hiccup kills the live run.** → Rehearse 3x; keep a recorded backup of the money moment; keep a hot local fallback (localhost room + cached frame -> deterministic detector) but run live primarily. The detector is a pure function, so a scripted-but-real fallback is trivial and honest.
- **Sponsor-padding credibility hit if claiming tools not really integrated (MiniMax/Modular).** → Claim only DO (deploy + optional Gradient secondary call), LiveKit (transport), Gemini (3 surfaces: vision + voice + translate), MongoDB Atlas + Voyage (memory). Drop MiniMax/Modular from the pitch.
- **Consent/recording optics with screen+mic+cam of teammates.** → ConsentGate opt-in + persistent 'PodMan is watching' indicator; observations TTL-expire in 6h; mention privacy-by-design in one breath. Turns a red flag into a maturity signal.
## Canonical starter files (staged in `docs/generated/files/`)
- `shared/src/messages.ts` — NEW shared types: the LiveKit data-channel wire protocol (one discriminated union both sides parse), the InterventionOutcome that closes the learning loop, the TeamModel the world-model loop maintains, and the optional Tier-2 LocalGitReport. Dependency-free; uses existing types via .js relative imports per the repo's verbatimModuleSyntax/Bundler config.
- `backend/src/env.ts` — Typed, fail-fast env loader shared by both backend entry points. Loads .env in dev, throws early if a required var is missing so deploys fail loud, not silently.
- `backend/src/server.ts` — The publicly-routable SERVICE (DO App Platform `service`, has http_port). Mints LiveKit tokens, exposes a ws relay so the PWA gets collision/intervention events, creates the real sync PR via Octokit on demand, and ingests outcome ACKs into memory. Binds 0.0.0.0 (App Platform requirement).
- `backend/src/agent.ts` — The OUTBOUND WORKER (DO App Platform `worker`, NO http_port). Joins a pod's LiveKit room with @livekit/rtc-node, subscribes ONLY to screen-share video, throttles to ~1fps, converts each VideoFrame to a downscaled JPEG with sharp, and hands it to the PodMan orchestrator. This is the moat path. Uses VideoStream directly because Agents-JS live-video is Python-only.
- `backend/src/agent/podman.ts` — The orchestrator that ties the whole loop together: receive a frame -> Gemini vision -> EngineerContext -> fuse with GitHub state -> run the collision detector -> consult memory/policy -> publish the intervention card over the data channel + speak. Holds the live per-engineer context map.
- `backend/src/vision/gemini.ts` — The load-bearing screen-understanding call: one JPEG -> schema-valid EngineerContext via gemini-3.5-flash with responseJsonSchema. Uses minimal thinking + structured output for cheap, fast, reliable ambient watching.
- `backend/src/collision/detector.ts` — The fusion rule that is the heart of the moat: a collision fires when >=2 engineers point at the same normalized file AND >=1 has local-but-unpushed work (proven by vision hasUnpushedChanges, optional Tier-2 report, or absence from remote per GitHub state). Pure function, easy to unit-test and demo deterministically.
- `backend/src/github/client.ts` — Octokit wrapper for the deterministic git half: cached state poll (branches/commits), branch-vs-branch compare (REST has it, MCP does not), remote-file existence check, and the real sync-PR creation that is the demo's clickable artifact.
- `frontend/src/livekit/useScreenPublish.ts` — The frontend capture hook: connect to the pod room with a server-minted token, publish screen-share (+system audio) tagged as Source.ScreenShare so the agent finds it, then mic + cam. This is the engineer side of the realtime spine.
- `frontend/src/livekit/useInterventions.ts` — Receives PodMan's interventions over the LiveKit data channel and exposes them as React state for the hero card; also sends the accept/dismiss ACK that closes the learning loop. PodMan's voice plays automatically because the agent publishes it as a normal audio track the room subscribes to.
- `database/init.ts` — One-shot Atlas setup: create collections and the indexes the continual-learning loop depends on, including the Voyage vector-search index used for pattern recall. Run once after creating the cluster.
- `infra/.do/app.yaml` — DigitalOcean App Platform single-app spec deploying all three components: the PWA (static_site), the token/relay API (service, has http_port + route), and the LiveKit agent (worker, NO port — so the deploy doesn't hang on a health check). This split is the key to a successful DO deploy and the Best DigitalOcean prize story.
docs/generated/CRITIQUE.md
-53
View File
@@ -1,53 +0,0 @@
# PodMan — Red-team critique
**Go/No-Go:** CONDITIONAL GO — but only if you cut to a ruthless MVP and de-risk in a specific order. The architecture is sound (I verified the load-bearing piece: @livekit/rtc-node installs on this machine and exposes exactly the API agent.ts assumes — VideoStream, VideoFrame.convert, VideoBufferType.RGBA, TrackSource.SOURCE_SCREENSHARE all present). The idea is genuinely differentiated and the moat ("unpushed changes are invisible to GitHub, only realtime vision sees them") is real and defensible. BUT the gap between the architecture doc and the actual repo is large: the doc describes ~25 files as "STARTER"/done; on disk there are 5 thin stubs and a single token-only backend entry. Nothing of the agent loop, vision wiring, voice, memory/vectors/policy, frontend LiveKit/PWA, or shared/messages.ts exists yet. You are building the entire load-bearing system in 20 hours across 4 people with 5+ external integrations (LiveKit, Gemini vision, Gemini voice, GitHub, MongoDB/Voyage, DO). That is doable for the core moat demo, NOT for the full doc. Go for the moat + ONE learning loop; treat everything else as optional. The continual-learning angle is the right competitive frame and is achievable in a thin form, but do not let it eat the realtime path's time.
**24h feasibility:** Tight but feasible for a SCOPED demo; NOT feasible for the full architecture as written. Reality check against the repo: backend is a single index.ts (token mint only) — the server/agent two-process split the doc calls 'the one load-bearing decision' is NOT built. Missing & uninstalled deps the design needs: @livekit/rtc-node (the moat; ~2 min native install — I timed it), sharp, ws, mongodb, voyageai, plus frontend vite-plugin-pwa/vite/@vitejs/plugin-react (frontend deps lists only react+livekit-client, no build toolchain). Honest hour budget for the HERO path only (capture -> 1fps frame grab -> Gemini vision -> collision detect -> card+voice -> real PR): ~14-18h of focused work with parallelism, and that assumes the LiveKit Cloud account, Gemini key, public repo, and a working screen-share-to-room round trip are all proven in hour 1-2. The doc's own estimate (~22h of listed steps) already exceeds the 20h budget BEFORE accounting for integration debugging, which always doubles. Continual-learning Loops A+B, Voyage vector recall, the Spanish translate flourish, the Tier-2 git sidecar, and DO Gradient are all over the line — ship them only if the hero path is solid by hour 12. Two unverified assumptions could blow the timeline: (1) the exact Gemini model IDs 'gemini-3.5-flash' / 'gemini-3.1-flash-live-preview' and the responseJsonSchema/mediaResolution API shape on @google/genai@2.10.0 — validate with one real API call in hour 1; (2) publishing a server-side AudioTrack via AudioSource.captureFrame for PodMan's voice is the fiddliest unproven bit — budget a half-day or fall back to browser-side TTS.
**DQ check:** Re-checked against the explicit rules. (1) DASHBOARD-AS-MAIN-FEATURE — REAL RISK. The pod grid of live screens is on the banned list and is the obvious thing to show. Mitigation is structural, not cosmetic: hero must be the proactive agent intervention (voice + card + real PR), grid stays a collapsed corner chip, demo opens on a single IDE. Rehearse the framing line. (2) PUBLIC REPO — NOT YET SATISFIED. Remote is git@github.com:karti-ai/podman over SSH and is almost certainly private; rules require public. Make the PodMan repo AND the target demo repo public before the demo, and verify on github.com. (3) ONLY-WHAT-YOU-BUILT / NEW WORK — currently OK (4 commits, all dated 2026-06-27, scaffold only) but enforce that no pre-existing or non-hackathon code lands, and that the demo repo's PR is opened against fresh hackathon content. (4) MAX 4 MEMBERS — team is exactly 4 (Karti, Ramis, Yahya, Zander), OK. (5) IMAGE-ANALYZER banned type — PodMan ingests screen frames with a vision model, which a hostile judge could caricature as 'image analyzer.' Defuse by framing vision as one sensor of an ambient agent that fuses with git state and acts, not an analysis product. (6) BASIC-RAG banned — the Voyage/Atlas vector recall must be framed as outcome-driven memory for a learning agent, not a RAG Q&A app. (7) Sponsor honesty — drop MiniMax/Modular claims; claim only DO, LiveKit, Gemini, MongoDB/Voyage that are actually integrated.
## Top risks
- **[critical]** SCOPE: the repo is ~5% built vs. what the architecture doc presents as done. The agent loop, vision wiring, voice, memory/vectors/policy, all frontend LiveKit hooks + PWA, and shared/messages.ts do not exist. The team may believe they are 'extending starters' when they are actually writing everything from scratch under a 20h clock.
→ _Fix:_ Reframe the doc as a spec, not a status. Hour 0: write the real file skeletons (empty exported functions) so the 4 people can work in parallel without merge stalls. Freeze scope to the hero path TODAY. Put the full file list on a board and mark only 8 files as P0 (agent.ts, vision/gemini.ts, collision/detector.ts [exists], podman.ts orchestrator, server token+ws, useScreenPublish, useInterventions, InterventionCard). Everything else is P1/P2.
- **[high]** DEPLOY HANG: the actual infra/app.yaml is a SINGLE Docker service with an http /health check — the exact trap the doc warns about. The agent (no HTTP port) cannot run as a health-checked service; it will deploy-hang or crash-loop. The doc's static_site+service+worker split is NOT implemented. Also @livekit/rtc-node is a native binding that took ~2 min to install — DO build timeouts and missing build deps (it pulls platform binaries) are a real risk.
→ _Fix:_ Do NOT block the demo on DO. Run the live demo from localhost (or one laptop + ngrok) as primary; deploy to DO in parallel as the 'it's also in the cloud' talking point. If you do deploy: split into worker (agent, no port, no health check) + service (token/ws, http_port 8787 bound 0.0.0.0) + static_site (PWA) per the doc, and confirm rtc-node's native deps are present in the runtime image. Verify the deploy by hour 14, not hour 20.
- **[high]** MODEL ID / API DRIFT: 'gemini-3.5-flash' and 'gemini-3.1-flash-live-preview' and the responseJsonSchema + mediaResolution:'MEDIA_RESOLUTION_LOW' + thinkingBudget:0 shape are assumptions, unverified against installed @google/genai@2.10.0. If any ID is wrong or the config keys differ, the load-bearing vision call fails and the whole demo is dead.
→ _Fix:_ Hour 1, before anything else: make ONE real generateContent call with a sample screenshot and the structured-output schema. Keep model IDs in env (already done) so they are swappable. If 3.5-flash isn't available on your key, fall back to the latest available flash vision model immediately. Confirm structured JSON actually returns and parses.
- **[high]** VISION FLAKINESS ON THE MONEY BEAT: detecting 'same file + >=1 unpushed' from screen pixels is the entire hero, but vision reading a file path and a dirty-git-gutter reliably under stage lighting/compression is fragile. A wrong read at the climax kills the demo and the Technicality score.
→ _Fix:_ Build the Tier-2 local git-reporter sidecar (git rev-list @{u}..HEAD + git status over the data channel) — it is ~30 lines and gives ground-truth 'unpushed'. The detector already accepts either signal. Use real vision for the 'what file' read (impressive, true) but back the 'unpushed' predicate with the sidecar so the climax is deterministic. Pre-stage the two engineers on the same known file so the read target is large, high-contrast, and unambiguous.
- **[high]** DASHBOARD-AS-HERO DQ: a grid of live teammate screens is on the explicit banned list, and that grid is the most natural thing to put on screen. Judges DQ on this exact pattern.
→ _Fix:_ Open the demo on a SINGLE engineer's IDE, never the grid. Keep PodGridCorner collapsed to a tiny peripheral chip. Say out loud 'PodMan is an agent, not a dashboard — the screens are just its eyes.' The hero on screen is the proactive intervention card + PodMan's voice + the real GitHub PR opening in a browser tab. Lead the 1-min video with the voice interruption, not any UI grid.
- **[medium]** VOICE OUT IS THE FIDDLIEST UNPROVEN PIECE: publishing PodMan's spoken audio as a server-side AudioTrack via AudioSource.captureFrame (Gemini live PCM -> AudioFrame) is the least-trodden path in the stack and easy to lose a day on.
→ _Fix:_ Treat in-room agent voice as P1, not P0. P0 fallback: send the intervention text over the data channel and speak it with browser-side Web Speech API / a pre-generated TTS clip on the PWA. It sounds identical to the audience. Only attempt server-published AudioTrack once the card + PR path works end to end.
- **[medium]** PUBLIC REPO + ONLY-WHAT-YOU-BUILT RULES: remote is git@github.com:karti-ai/podman (SSH, likely private). Rules require PUBLIC repos, max 4 members, and demo only NEW hackathon work. A private demo repo or pre-existing code = DQ.
→ _Fix:_ Make BOTH repos public now: the PodMan repo itself AND the target demo repo the PR is opened against (the doc shows its URL on screen). Confirm the GitHub PAT is fine-grained on the public demo repo with Contents R + PRs R/W. Ensure all committed code is dated within the hackathon window and the 4-member cap holds.
- **[medium]** INTEGRATION COUNT: 5-6 external services on the hot path (LiveKit, Gemini vision, Gemini voice, GitHub, Mongo/Voyage, DO). Each adds auth/latency/failure surface; the realtime chain is only as reliable as its weakest hop and venue wifi.
→ _Fix:_ Drop everything not load-bearing: cut MiniMax and Modular from the pitch entirely (sponsor-padding hurts credibility). Make Mongo/Voyage optional behind a flag so a DB outage can't crash the agent. Cache the latest frame and degrade gracefully. Rehearse on a phone hotspot, not just venue wifi.
- **[medium]** CONTINUAL-LEARNING CLAIM IS THIN IF NOT SHOWN: the winning frame is Continual Learning, but Voyage vector recall + policy adaptation are P1 and at real risk of being cut, leaving the pitch making a claim the demo doesn't show.
→ _Fix:_ Build the CHEAPEST honest version of 'it got smarter': persist collision signatures to Mongo and, on the 2nd identical collision, retrieve the prior record to raise confidence and lead with the previously-accepted action. That single retrieve-and-explain beat ('I've seen this conflict before') satisfies the track without the full vector+policy stack. Voyage embeddings are a bonus, not a requirement, for the beat.
## Demo-day reliability
- RUN LIVE BUT CARRY A RECORDED MONEY-MOMENT. Record a clean take of the full collision->voice->card->real-PR beat early and keep it one keystroke away. If anything hiccups on stage, narrate over the recording without missing a beat.
- PRE-STAGE THE TWO COLLIDING ENGINEERS. Have both laptops already joined to a fixed room (POD_ROOM=demo-pod) with screen-share live and parked on the SAME known file (e.g. src/auth/session.ts) with large high-contrast text, one with uncommitted edits. Don't ask vision to find a needle live.
- MAKE 'UNPUSHED' DETERMINISTIC VIA THE TIER-2 SIDECAR. Use real vision for the 'what file' read, but feed the unpushed predicate from git rev-list @{u}..HEAD over the data channel so the climax can't misfire on a flaky pixel read. The detector already accepts either signal.
- VALIDATE GEMINI IN HOUR 1. One real generateContent call with a sample screenshot + the structured schema, on the actual installed @google/genai@2.10.0. Confirm the model ID resolves and JSON parses before building anything on top.
- DON'T DEPEND ON DO FOR THE LIVE RUN. Demo from localhost or one laptop + ngrok (secure context is required for getDisplayMedia anyway). Keep the DO deployment as a parallel 'also in the cloud' tab, verified by hour 14.
- BRING YOUR OWN NETWORK. Phone hotspot as primary, venue wifi as backup (not the reverse). The realtime chain dies on bad wifi. Test the full path on the hotspot during rehearsal.
- FALL BACK TO BROWSER TTS FOR VOICE. If server-published AudioTrack via AudioSource.captureFrame isn't rock-solid, speak the intervention text with Web Speech API / a pre-rendered clip on the PWA — sounds identical to the audience and removes the riskiest hop.
- OPEN A REAL PR, SHOW THE GITHUB TAB. The clickable PR on a public repo is the credibility anchor. Pre-create the branch base and confirm the PAT scopes so pulls.create can't 403 on stage. Have the github.com URL already on screen.
- PRE-WARM EVERYTHING. Agent already in the room, GitHub state cached, Mongo connected, frames flowing, BEFORE you start talking. Cold-starting @livekit/rtc-node + first Gemini call live wastes 10+ seconds of a 3-min slot.
- REHEARSE 3X END TO END with a stopwatch. The 3-min slot is unforgiving; lead with the voice interruption in the first 20 seconds, the 'I've seen this conflict before' learning beat in the middle, the real PR at the climax.
## Scope cuts if behind (in order)
- FIRST CUT: Spanish live-translate flourish (gemini-3.5-live-translate). Pure bonus, zero impact on the core thesis.
- DigitalOcean Gradient secondary LLM call. The DO deploy alone earns the DO story; Gradient is a nice-to-have you can name without building.
- Voyage AI embeddings + Atlas $vectorSearch. Replace with an exact/normalized-signature match in Mongo to still get the 'I've seen this before' beat. Keep the beat, drop the vector machinery.
- Policy Loop B (outcome-driven threshold/preference adaptation). Keep the ACK plumbing so outcomes are recorded, but don't build the adaptive policy if time is short — the world-model recall (Loop A) carries the Continual Learning claim.
- Server-published agent AudioTrack. Fall back to browser-side TTS for PodMan's voice; identical audience experience, far less risk.
- PWA installability (vite-plugin-pwa, service worker, manifest). Ship a normal Vite web app; 'installable' is a one-line pitch claim, not a demo requirement.
- Tier-2 git sidecar — KEEP unless truly desperate; it's the cheapest reliability insurance for the climax. Only cut if vision's unpushed read is proven solid in rehearsal.
- Multi-pod / password+QR join. Hardcode a single demo-pod room. Join UX is not the hero.
- MongoDB persistence entirely (last resort). Run the learning beat from an in-memory map for the demo; you lose the 'persistent world-model' talking point but keep a working agent. Cut this only if the DB is actively breaking the run.
docs/generated/RESEARCH.md
-655
View File
@@ -1,655 +0,0 @@
# PodMan — Research findings (validated, June 2026)
## Gemini 3.5 realtime/vision APIs for PodMan (screen-understanding, Live voice/translate, Computer Use, Managed Agents)
All three organizer-named models are REAL and callable via the @google/genai JS/TS SDK (Gemini Developer API) as of June 2026. Reconciliation: (1) `gemini-3.5-flash` exists, is GA/stable, and is the right model for the vision-to-structured-JSON screen-understanding layer (1M ctx, supports responseJsonSchema). (2) `gemini-3.5-live-translate-preview` exists for the Live Translate feature, BUT it is translate-ONLY (audio in -> translated audio out, no general conversation, no video). For PodMan's "PodMan speaks" voice the correct general Live model is `gemini-3.1-flash-live-preview` (native audio, accepts text+image+audio+video over one WebSocket). There is no general-purpose `gemini-3.5-flash-live`; use 3.1-flash-live for voice and 3.5-live-translate only for the translation demo flourish. (3) `antigravity-preview-05-2026` is real — the Antigravity managed agent on the Interactions API (hosted Linux sandbox, can write/run code, browse web, manage files). Computer Use: the whats-new FAQ line saying "not supported in 3.5 Flash" is STALE — Google shipped built-in Computer Use INSIDE `gemini-3.5-flash` on 2026-06-24 (3 days ago) as public preview, covering browser/mobile/desktop. So Computer Use needs NO separate model now; it is a tool on 3.5-flash.
ARCHITECTURE RECOMMENDATION for PodMan: (a) Screen understanding = `gemini-3.5-flash` generateContent with image part + responseJsonSchema, run per-engineer at ~1 frame / 2-5s (NOT the Live API — cheaper, structured, no 1fps cap). Use mediaResolution "low" (280 tok/img) for ambient watching. (b) Voice out + barge-in = ai.live.connect with `gemini-3.1-flash-live-preview`, responseModalities:[AUDIO]. (c) Opening the sync PR: do NOT use Computer Use for this — use the GitHub REST API directly (deterministic, instant). Reserve Computer Use as a creative "PodMan drives the screen" demo moment only. (d) Antigravity/Interactions API: worth it ONLY for the self-improvement angle (background agent that analyzes conflict outcomes / generates the PR diff in a sandbox); skip for the realtime hot path (latency + cost). Keep realtime local.
**Model / IDs:** `gemini-3.5-flash — GA, vision + structured JSON (responseJsonSchema), 1M ctx / 65k out; ALSO hosts built-in Computer Use tool (preview, since 2026-06-24). Use for PodMan screen-understanding.`, `gemini-3.1-flash-live-preview — Live API native audio (text+image+audio+video in, audio out over WebSocket). Use for PodMan's voice / barge-in.`, `gemini-3.5-live-translate-preview — Live Translate ONLY (audio in -> translated audio out, 16kHz->24kHz). Optional translation demo, not general voice.`, `antigravity-preview-05-2026 — Antigravity managed agent via Interactions API; hosted Linux sandbox (code exec + web + files), stateful. Use for off-path self-improvement / PR drafting.`, `gemini-3-flash-preview — older preview of the Flash line; appears in some docs. Prefer gemini-3.5-flash.`, `gemini-live-2.5-flash-native-audio / gemini-2.5-flash-native-audio-preview-12-2025 — previous-gen Live native audio (fallback if 3.1-flash-live preview is unstable).`, `gemini-2.5-computer-use-preview-10-2025 — LEGACY standalone Computer Use model; superseded by built-in Computer Use in gemini-3.5-flash. Avoid for new work.`
**Packages**
- `@google/genai` — npm i @google/genai
_Official Google Gen AI JS/TS SDK (context7 id /googleapis/js-genai, latest v2.0.1). Single package covers generateContent (vision+structured), ai.live.connect (Live API WebSocket), and ai.interactions.create (Managed Agents/Antigravity). Use `new GoogleGenAI({ apiKey })`. For the browser PWA, keep the API key server-side (Node/backend) and proxy — do not ship the key in the Vite frontend. Live API can also be hit as a raw WebSocket from the browser if you mint ephemeral tokens, but easiest is backend relay._
- `ws (Node)` — npm i ws
_Only if you hand-roll the Live API WebSocket on the backend instead of using ai.live.connect. The SDK's live module already wraps this; prefer the SDK._
**Critical snippets**
### (a) Screen frame -> structured scene understanding (gemini-3.5-flash)
```typescript
import { GoogleGenAI, Type } from "@google/genai";
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
// frameJpegBase64 = one screenshot grabbed from the LiveKit screen-share track
const res = await ai.models.generateContent({
model: "gemini-3.5-flash",
contents: [{
role: "user",
parts: [
{ text: "You are PodMan watching an engineer's screen. Identify what they are working on. Return JSON only." },
{ inlineData: { mimeType: "image/jpeg", data: frameJpegBase64 } }
]
}],
config: {
responseMimeType: "application/json",
responseJsonSchema: {
type: Type.OBJECT,
properties: {
app: { type: Type.STRING, description: "e.g. VS Code, Chrome, terminal" },
repo: { type: Type.STRING },
filePath: { type: Type.STRING, description: "open file path if visible" },
symbols: { type: Type.ARRAY, items: { type: Type.STRING }, description: "functions/classes on screen" },
activity: { type: Type.STRING, description: "editing | reading | debugging | terminal | PR review" },
feature: { type: Type.STRING, description: "inferred feature/ticket" },
unpushedHint:{ type: Type.BOOLEAN, description: "signs of uncommitted/unpushed local edits (dirty git gutter, modified markers)" }
},
propertyOrdering: ["app","repo","filePath","symbols","activity","feature","unpushedHint"]
}
}
});
const scene = JSON.parse(res.text); // guaranteed schema-valid
```
> This is PodMan's load-bearing 'see unpushed changes' layer. Use generateContent (one-shot per frame), NOT the Live API, so you get structured JSON, no 1fps cap, and ~280 tokens/img at low res. Throttle to one frame every 2-5s per engineer. To control vision cost add per-part `mediaResolution`/`resolution: "low"`.
### (b) PodMan speaks: Live API native audio (gemini-3.1-flash-live-preview)
```typescript
import { GoogleGenAI, Modality } from "@google/genai";
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
const session = await ai.live.connect({
model: "gemini-3.1-flash-live-preview", // general voice; NOT the translate model
config: {
responseModalities: [Modality.AUDIO],
speechConfig: { voiceConfig: { prebuiltVoiceConfig: { voiceName: "Kore" } } },
systemInstruction: "You are PodMan, an ambient assistant for an engineering pod. Speak only to warn about imminent code collisions; be terse."
},
callbacks: {
onopen: () => console.log("PodMan voice live"),
onmessage: (msg) => {
// msg.serverContent.modelTurn.parts[].inlineData = base64 24kHz PCM -> play in browser
},
onerror: (e) => console.error(e),
onclose: () => console.log("closed")
}
});
// trigger PodMan to speak a warning (text in -> audio out):
session.sendClientContent({ turns: "Two engineers are both editing auth/session.ts. Warn them." });
// to feed live frames/mic instead:
// session.sendRealtimeInput({ media: { data: pcm16kBase64, mimeType: "audio/pcm;rate=16000" } });
// session.sendRealtimeInput({ media: { data: jpegBase64, mimeType: "image/jpeg" } });
```
> Raw WS endpoint if not using SDK: wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=KEY . Audio: 16-bit PCM 16kHz in, 24kHz out. sendRealtimeInput uses VAD auto-response. Session limits: audio-only 15 min, audio+video only 2 min, video max 1 fps. For the optional Live Translate demo swap model to gemini-3.5-live-translate-preview and add translationConfig:{targetLanguageCode:'es'} (audio-in only).
### (d) Managed Agent / Antigravity (Interactions API) for self-improvement loop
```typescript
// Hosted sandbox agent: analyze a detected collision, draft the sync PR diff, run tests.
const interaction = await ai.interactions.create({
model: "antigravity-preview-05-2026",
input: "Two unpushed branches both touch auth/session.ts. Here is each diff... produce a merged sync patch and run the test suite.",
tools: [{ type: "code_execution" }],
// background: true // async; resume later via the returned environment id
});
console.log(interaction.outputs);
```
> Runs in a Google-hosted ephemeral Linux container (reason+code+web+files). State persists across calls via the environment id; auto-compacts ctx ~135k tokens. Use OFF the realtime path — e.g. PodMan's 'refine its own intervention policy from outcomes' / generate-PR step. For simple compute you can use model:'gemini-2.5-flash' with the same code_execution tool. Skip for low-latency interventions (network round-trip).
**Gotchas**
- Computer Use FAQ is STALE: an older whats-new page says Computer Use is 'not supported in Gemini 3.5 Flash', but Google shipped built-in Computer Use INSIDE gemini-3.5-flash on 2026-06-24 (public preview, browser+mobile+desktop). No separate model needed. Computer Use requires mediaResolution 'ultra_high' on the screenshot (2,240 tokens/img) — expensive; do not use it on the ambient watch loop.
- Do NOT use Computer Use to open the sync PR in the demo hot path — it is a slow screenshot->action loop and flaky. Open the PR with the GitHub REST API (deterministic). Keep Computer Use as an optional 'PodMan takes the wheel' wow-moment only.
- gemini-3.5-live-translate-preview is TRANSLATE-ONLY: audio in -> translated audio out, no text input, no video, no free conversation. It cannot be PodMan's general voice. Use gemini-3.1-flash-live-preview for PodMan speaking. There is no general-purpose 'gemini-3.5-flash-live'.
- Live API session caps will bite the demo: audio-only 15 min, audio+VIDEO only 2 minutes, and video input is capped at 1 frame/sec. So DON'T push every engineer's screen video through the Live API. Do screen understanding via one-shot generateContent on gemini-3.5-flash (throttled frames), and reserve the Live session for short voice bursts. Use sessionResumption/contextWindowCompression config to survive reconnects.
- Vision cost scales with frames x engineers: image tokens are low=280, medium=560, high=1,120, ultra_high=2,240 per image (Gemini 3 family). 4 engineers x 1 frame/2s x low-res = ~33.6k img-tokens/min input. At $1.50/1M input that is cheap (~$0.05/min) but DON'T run high/ultra_high on the ambient loop. Set per-part resolution:'low' (per-content-item resolution is a Gemini 3 exclusive).
- API key security: @google/genai in a Vite frontend would leak the key. Run generateContent + Live + Interactions calls from the backend; for browser-side Live, mint ephemeral tokens or relay the WebSocket through your server.
- gemini-3.5-flash knowledge cutoff is Jan 2025 and it defaults to 'medium' thinking effort — for the fast ambient classifier set thinkingConfig to minimal/low to cut latency and cost; reserve higher thinking for the collision-reasoning step.
- Pricing snapshot (per 1M tokens, paid tier): gemini-3.5-flash $1.50 in / $9.00 out; gemini-3.1-flash-live-preview audio $3.00 in (or $0.005/min) / $12.00 out (or $0.018/min); gemini-3.5-live-translate-preview audio $3.50 in / $21.00 out. Live Translate output is the priciest — budget it. Free tier exists for the hackathon but is rate-limited.
- SDK field naming: structured output uses responseMimeType:'application/json' + responseJsonSchema (or responseSchema) — you MUST set the mime type or the schema is ignored. Live config uses responseModalities:[Modality.AUDIO]. Interactions API tools use {type:'code_execution'} objects, not the function-calling shape.
**Sources**
- https://ai.google.dev/gemini-api/docs/whats-new-gemini-3.5 (gemini-3.5-flash capabilities; note stale Computer-Use FAQ line)
- https://ai.google.dev/gemini-api/docs/models/gemini-3.5-flash (3.5 Flash specs, 1M/65k token limits, GA)
- https://ai.google.dev/gemini-api/docs/live-api/live-translate (gemini-3.5-live-translate-preview, translationConfig, audio-only)
- https://ai.google.dev/gemini-api/docs/live-api/get-started-websocket (WebSocket endpoint, gemini-3.1-flash-live-preview, responseModalities AUDIO, PCM formats)
- https://ai.google.dev/gemini-api/docs/live-api/capabilities (Live model ids, 1fps video, session 15min/2min limits, speechConfig voices)
- https://ai.google.dev/gemini-api/docs/computer-use (Computer Use loop; gemini-3.5-flash / gemini-3-flash-preview model ids, browser/mobile/desktop)
- https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-computer-use-gemini-3-5-flash/ (built-in Computer Use in 3.5 Flash, 2026-06-24 public preview)
- https://ai.google.dev/gemini-api/docs/antigravity-agent (antigravity-preview-05-2026 managed agent)
- https://ai.google.dev/gemini-api/docs/custom-agents and https://blog.google/innovation-and-ai/technology/developers-tools/managed-agents-gemini-api/ (Interactions API / Managed Agents, hosted Linux sandbox, code_execution, background, environment resume)
- https://ai.google.dev/gemini-api/docs/pricing (per-token pricing for 3.5-flash, 3.1-flash-live, 3.5-live-translate; free tier)
- https://ai.google.dev/gemini-api/docs/media-resolution (image token counts low 280/medium 560/high 1120/ultra_high 2240; per-part resolution field, Gemini 3 exclusive)
- context7 /googleapis/js-genai v2.0.1 (generateContent responseJsonSchema, ai.live.connect callbacks, sendRealtimeInput media Blob, ai.interactions.create code_execution)
## LiveKit realtime stack for PodMan: browser screen-share + mic + cam publishing, server-side token minting, Node agent subscribing to a remote screen-share video track and grabbing raw frames for a vision model (the critical path), data-channel interventions back to engineers, and PodMan TTS voice into the room, plus how it composes with Gemini.
All four pieces are buildable today, but the architecture hinges on one finding: the LiveKit Agents-JS framework's integrated LIVE-VIDEO pipeline is currently Python-only (the vision/video doc marks "Node.js: not available; Python: available"). For a Node backend you drop to the lower-level @livekit/rtc-node SDK, where VideoStream + VideoFrame work fully. So PodMan's critical path (b) = a plain Node process that connects with @livekit/rtc-node (autoSubscribe:true, dynacast:true), listens for RoomEvent.TrackSubscribed, filters for the screen-share publication (publication.source === TrackSource.SOURCE_SCREENSHARE), wraps the RemoteVideoTrack in a VideoStream, async-iterates frame events, calls frame.convert(VideoBufferType.RGBA) to get a raw pixel buffer, encodes to JPEG/PNG (sharp), and ships it to Gemini vision. This sidesteps the framework limitation and gives full control over frame-rate throttling (critical for cost/latency: sample ~1 fps, not 30). Recommended component split: (a) Browser = livekit-client: createLocalScreenTracks({audio:true}) for screen + system audio, then setCameraEnabled/setMicrophoneEnabled for cam+mic; receive interventions via RoomEvent.DataReceived. (b) Token server = livekit-server-sdk: new AccessToken + addGrant({roomJoin, room, canPublish, canSubscribe, canPublishData}). (c)+(d) Vision/voice agent = @livekit/rtc-node: subscribe + frame grab + publishData(JSON, {reliable:true, topic}) back to engineers + publish a TTS audio track via AudioSource/AudioFrame.captureFrame. Two valid voice strategies: (A) full Agents-JS voice.AgentSession with the Google plugin (google.beta.realtime.RealtimeModel = Gemini Live native audio, model gemini-2.5-flash-native-audio-preview, or google.beta.TTS) — cleanest for the VOICE leg; (B) DIY: Gemini TTS bytes -> Int16 PCM -> AudioFrame -> AudioSource.captureFrame on a published mic-source track. Because Agents-JS cannot do the live VIDEO leg in Node, the pragmatic winning setup is a HYBRID: one rtc-node process owns the screen-frame vision loop, and either the same process (DIY audio) or a co-located Agents-JS worker owns the voice. If the team can tolerate Python for just the vision worker, Agents-JS Python gives the fully integrated path with VideoStream + llm.ImageContent — but Node + rtc-node keeps the whole backend in one language and is proven by the SDK's own examples.
**Model / IDs:** `gemini-2.5-flash-native-audio-preview-12-2025 (via @livekit/agents-plugin-google google.beta.realtime.RealtimeModel — confirm exact id with Gemini lane; PROJECT targets Gemini 3.5 Live API)`, `Gemini 2.5/3.5 Flash vision (via @google/genai, for per-frame screen understanding — exact id per Gemini research lane)`, `deepgram/nova-3 (example STT in Agents-JS, only if you add engineer voice input)`, `cartesia/sonic-3 (example TTS in Agents-JS inference; replaceable by Gemini TTS)`
**Packages**
- `livekit-client` — npm i livekit-client
_Browser SDK (frontend, React+Vite). Room, createLocalScreenTracks, LocalParticipant.setScreenShareEnabled/setCameraEnabled/setMicrophoneEnabled, publishData, RoomEvent.DataReceived/TrackSubscribed. This is what each engineer's PWA runs._
- `livekit-server-sdk` — npm i livekit-server-sdk
_Node backend token minting + room admin. AccessToken, VideoGrant fields, TokenVerifier, RoomServiceClient. Runs on your token endpoint (e.g. Express on DigitalOcean)._
- `@livekit/rtc-node` — npm i @livekit/rtc-node
_THE critical-path package for the PodMan agent. Lets a Node process join a room as a participant, subscribe to remote tracks, read raw video frames (VideoStream/VideoFrame.convert), publish audio frames (AudioSource/AudioFrame), and publishData. Native bindings (prebuilt binaries) — works on macOS/Linux; build it into your DigitalOcean container. Call dispose() on shutdown._
- `@livekit/agents` — npm i @livekit/agents
_Optional higher-level agent framework (defineAgent, JobContext, voice.AgentSession, cli.runApp). Great for the VOICE leg (STT/LLM/TTS/VAD orchestration). NOTE: its integrated live-VIDEO input is Python-only right now, so do not rely on it for screen-frame vision in Node._
- `@livekit/agents-plugin-google` — npm i @livekit/agents-plugin-google
_Gemini plugin for Agents-JS. Exposes google.beta.realtime.RealtimeModel (Gemini Live native audio, e.g. gemini-2.5-flash-native-audio-preview) and google.beta.TTS. Use for PodMan's spoken interventions if you adopt the AgentSession path._
- `@livekit/agents-plugin-silero` — npm i @livekit/agents-plugin-silero
_Only needed if you build a turn-taking voice.AgentSession (VAD). Not needed for one-way TTS announcements._
- `sharp` — npm i sharp
_Encode the RGBA/RGB24 buffer from VideoFrame.convert into JPEG/PNG before sending to Gemini (Gemini wants encoded image bytes, not raw RGBA). Fast native encoder; downscale here to cut tokens/latency._
- `@google/genai` — npm i @google/genai
_Google Gen AI SDK to call Gemini 2.5/3.5 Flash vision with the encoded screen frame (inlineData base64 image + prompt) and to get TTS audio bytes if going the DIY voice route. (Confirm exact model id with the Gemini research lane.)_
**Critical snippets**
### (a) BROWSER: publish screen-share (+system audio) then mic + camera (livekit-client)
```typescript
import { Room, RoomEvent, Track, createLocalScreenTracks, VideoPresets } from 'livekit-client';
const room = new Room({ adaptiveStream: true, dynacast: true });
await room.connect(LIVEKIT_WS_URL, token); // token from your server
// SCREEN SHARE + system audio. Triggers the browser picker.
// cursor:'always' keeps the pointer visible (useful for the vision model).
const screenTracks = await createLocalScreenTracks({ audio: true, resolution: VideoPresets.h1080.resolution });
for (const t of screenTracks) {
await room.localParticipant.publishTrack(t.mediaStreamTrack, {
source: t.kind === Track.Kind.Audio ? Track.Source.ScreenShareAudio : Track.Source.ScreenShare,
});
}
// (Equivalent one-liner: await room.localParticipant.setScreenShareEnabled(true, { cursor:'always' }))
// MIC + CAMERA (separate prompts).
await room.localParticipant.setMicrophoneEnabled(true);
await room.localParticipant.setCameraEnabled(true);
```
> createLocalScreenTracks returns up to 2 tracks (video + optional system-audio); you must tag the screen video with Source.ScreenShare so the agent can find it. getDisplayMedia permission is implicit. Screen-share audio is a SEPARATE source from mic.
### (a) SERVER: mint an access token (livekit-server-sdk)
```typescript
import { AccessToken } from 'livekit-server-sdk';
export async function createToken(roomName: string, identity: string, name: string) {
const at = new AccessToken(process.env.LIVEKIT_API_KEY!, process.env.LIVEKIT_API_SECRET!, {
identity, name, ttl: '4h',
metadata: JSON.stringify({ githubLogin: name }), // handy for collision attribution
});
at.addGrant({
roomJoin: true, room: roomName,
canPublish: true, // engineer publishes screen/mic/cam
canSubscribe: true,
canPublishData: true, // allows acking interventions
});
return await at.toJwt(); // async -> returns the JWT string
}
// The PodMan agent itself wants a token too: canSubscribe:true (read screens),
// canPublish:true (publish TTS audio), canPublishData:true (push interventions).
```
> toJwt() is async in current versions. Scope tightly: give engineers canSubscribe only if you also want them to hear PodMan. roomCreate is auto on first join so you usually don't need it.
### (b) CRITICAL PATH: Node agent subscribes to the screen-share track and grabs RGBA frames (@livekit/rtc-node)
```typescript
import {
Room, RoomEvent, TrackKind, TrackSource,
VideoStream, VideoBufferType, dispose,
type RemoteTrack, type RemoteTrackPublication, type RemoteParticipant,
} from '@livekit/rtc-node';
import sharp from 'sharp';
const room = new Room();
await room.connect(LIVEKIT_WS_URL, agentToken, { autoSubscribe: true, dynacast: true });
let lastSentAt = 0;
const SAMPLE_INTERVAL_MS = 1000; // ~1 fps to the vision model -> cheap + low latency
room.on(RoomEvent.TrackSubscribed,
(track: RemoteTrack, pub: RemoteTrackPublication, participant: RemoteParticipant) => {
// Only the SCREEN SHARE video, not webcam, not mic.
if (track.kind !== TrackKind.KIND_VIDEO || pub.source !== TrackSource.SOURCE_SCREENSHARE) return;
const stream = new VideoStream(track);
(async () => {
for await (const event of stream) { // event.frame is a VideoFrame
const now = Date.now();
if (now - lastSentAt < SAMPLE_INTERVAL_MS) continue; // THROTTLE: drop frames
lastSentAt = now;
const frame = event.frame;
const rgba = frame.convert(VideoBufferType.RGBA); // -> { data:Uint8Array, width, height, type }
// Encode to JPEG (Gemini wants encoded bytes, not raw RGBA). Downscale to save tokens.
const jpeg = await sharp(Buffer.from(rgba.data), {
raw: { width: rgba.width, height: rgba.height, channels: 4 },
}).resize({ width: 1280, withoutEnlargement: true }).jpeg({ quality: 70 }).toBuffer();
await onScreenFrame(participant.identity, jpeg); // -> hand to Gemini vision
}
})();
});
// on shutdown: await room.disconnect(); await dispose();
```
> THIS is PodMan's moat (catching unpushed local edits via the screen). Key points: (1) Agents-JS integrated video is Python-only, so use rtc-node directly. (2) VideoFrame.convert(VideoBufferType.RGBA) gives a packed 4-channel buffer; use RGB24 if you want 3 channels. (3) Throttle yourself — frames arrive at the publish FPS (could be 30). (4) Encode with sharp before sending; do not send raw RGBA. (5) Tag screen-share via pub.source so you never confuse it with the webcam.
### (b-alt) Same frame-grab using the Agents-JS framework (VideoStream + llm.ImageContent)
```typescript
// Inside a voice.Agent subclass (Agents-JS). Useful pattern even though integrated
// live-video is officially Python-first; the VideoStream loop itself runs in Node.
import { VideoStream } from '@livekit/rtc-node';
import { llm } from '@livekit/agents';
private latestFrame: VideoFrame | null = null;
private createVideoStream(track: Track): void {
this.videoStream?.cancel();
this.videoStream = new VideoStream(track);
(async () => { for await (const e of this.videoStream!) this.latestFrame = e.frame; })();
}
async onUserTurnCompleted(chatCtx: llm.ChatContext, msg: llm.ChatMessage) {
if (this.latestFrame) {
msg.content.push(llm.createImageContent({ image: this.latestFrame }));
this.latestFrame = null;
}
}
```
> Buffer the latest frame and only attach it on a turn — avoids streaming every frame to the LLM. createImageContent accepts a VideoFrame directly. Good if you later move the vision worker to Python for the fully-supported path.
### (c) DATA CHANNEL: agent broadcasts an intervention; browser receives it
```typescript
// --- AGENT SIDE (@livekit/rtc-node) ---
const payload = new TextEncoder().encode(JSON.stringify({
type: 'COLLISION', file: 'src/auth/session.ts',
withIdentity: 'alice', message: 'Bob has unpushed edits to this file', action: 'OFFER_SYNC_PR',
}));
await room.localParticipant!.publishData(payload, {
reliable: true, topic: 'podman.intervention',
destination_identities: ['bob'], // omit to broadcast to the whole pod
});
// --- BROWSER SIDE (livekit-client) ---
import { RoomEvent } from 'livekit-client';
room.on(RoomEvent.DataReceived, (payload, participant, _kind, topic) => {
if (topic !== 'podman.intervention') return;
const msg = JSON.parse(new TextDecoder().decode(payload));
showInterventionCard(msg); // render the warn/diff/sync-PR card
});
```
> Reliable mode is ordered+retransmitted, cap ~15 KiB per packet (lossy cap ~1300 bytes). Use a topic to multiplex intervention types. destination_identities targets one engineer; omit for pod-wide. For larger payloads (a full diff), use byte/text streams (streamBytes/sendText) instead. Engineer ACKs (accepted PR? dismissed?) flow back via publishData — that closes the self-improvement / continual-learning loop.
### (d) PodMan VOICE: publish a TTS audio track from the agent (@livekit/rtc-node, DIY route)
```typescript
import { AudioSource, AudioFrame, LocalAudioTrack, TrackPublishOptions, TrackSource } from '@livekit/rtc-node';
const SAMPLE_RATE = 24000, CHANNELS = 1; // match your TTS output (Gemini TTS ~24kHz)
const source = new AudioSource(SAMPLE_RATE, CHANNELS);
const track = LocalAudioTrack.createAudioTrack('podman-voice', source);
const opts = new TrackPublishOptions(); opts.source = TrackSource.SOURCE_MICROPHONE;
await room.localParticipant!.publishTrack(track, opts);
// ttsPcm: Int16Array of mono PCM from Gemini TTS (decode the returned audio to raw PCM16 first).
async function speak(ttsPcm: Int16Array) {
const CHUNK = SAMPLE_RATE / 100; // 10ms frames
for (let i = 0; i < ttsPcm.length; i += CHUNK) {
const slice = ttsPcm.subarray(i, i + CHUNK); // NOTE: subarray, NOT slice (slice() is unstable in Node)
await source.captureFrame(new AudioFrame(slice, SAMPLE_RATE, CHANNELS, slice.length));
}
}
```
> Engineers hear this because they subscribe to the agent's mic-source track. CRITICAL gotcha straight from the SDK: when converting Uint8Array->Int16Array use buffer.subarray, never buffer.slice (slice is flagged unstable by Node and injects noise). Match SAMPLE_RATE to your TTS. captureFrame backpressures, so awaiting it paces playback in realtime.
### (d-alt) PodMan VOICE via Agents-JS + Gemini plugin (cleanest voice leg)
```typescript
import { voice } from '@livekit/agents';
import * as google from '@livekit/agents-plugin-google';
const session = new voice.AgentSession({
// Gemini Live native audio (single realtime model does the speaking):
llm: new google.beta.realtime.RealtimeModel({ model: 'gemini-2.5-flash-native-audio-preview' }),
// OR keep your own LLM and just use Gemini TTS: tts: new google.beta.TTS(),
});
await session.start({ agent, room: ctx.room });
await session.generateReply({ instructions: 'Warn Bob: collision on session.ts; offer a sync PR.' });
```
> Lets LiveKit handle audio framing/turn-taking for you. Pair this voice worker with the SEPARATE rtc-node vision worker (snippet b) since the integrated video input is Python-only in Node. Confirm the exact Gemini model id with the Gemini research lane — names move fast.
**Gotchas**
- BIGGEST one: LiveKit Agents-JS integrated LIVE-VIDEO input is currently Python-only (the docs/agents/multimodality/vision/video page literally marks Node.js as not available). Do NOT architect PodMan's screen-vision around the Node AgentSession video helper. Use @livekit/rtc-node's VideoStream/VideoFrame directly (fully supported) OR run just the vision worker in Python. The voice leg is fine in Node.
- Frame-rate throttling is mandatory. A VideoStream yields frames at the publisher's FPS (up to 30). Sending every frame to Gemini will blow up cost and latency. Throttle to ~1 fps (or send only on a turn / on detected change). Buffer the latest frame and sample it.
- VideoFrame.convert() returns a NEW buffer; you must encode it (sharp -> JPEG/PNG) before sending to Gemini — vision models want encoded image bytes, not raw RGBA. Downscale (e.g. width 1280) at encode time to cut tokens.
- VideoBufferType values: ARGB, RGBA, ABGR, BGRA (32-bit), RGB24 (24-bit), I444/I422/I420/I420A/I010 (planar YUV), NV12. Incoming WebRTC frames are typically I420 (YUV); always convert to RGBA/RGB24 before handing to an image encoder.
- Audio gotcha (from the SDK's own README): when converting Uint8Array -> Int16Array, use buffer.subarray, NEVER buffer.slice — slice is marked unstable in Node and can append large bursts of noise to the audio.
- Screen-share AUDIO (system audio) is a DISTINCT source (Track.Source.ScreenShareAudio) from the microphone. createLocalScreenTracks({audio:true}) can return two tracks; publish each with the correct source or the agent will mis-route them.
- Tag and filter by source. On publish, set source: Track.Source.ScreenShare; on the agent, filter pub.source === TrackSource.SOURCE_SCREENSHARE so you never feed the webcam to the collision detector instead of the screen.
- getDisplayMedia (screen share) REQUIRES a user gesture and a secure context (https or localhost). As an installable PWA this is fine, but the demo must serve over HTTPS — plan TLS on the DigitalOcean deploy.
- publishData reliable cap is ~15 KiB; lossy ~1300 bytes (MTU). A full git diff can exceed this — use byte/text streams (localParticipant.streamBytes / sendText with a topic + writer.write/close) for large intervention payloads, and keep publishData for small JSON cards.
- toJwt() is ASYNC in current livekit-server-sdk versions (returns Promise<string>). Forgetting the await yields '[object Promise]' as the token and a confusing auth failure.
- @livekit/rtc-node uses native bindings (prebuilt per-platform binaries). It works on macOS for local dev and Linux in the DigitalOcean container, but make sure the install runs in the deploy image (not copied node_modules from macOS). Call dispose() (and room.disconnect()) on shutdown to free the native runtime.
- The agent's token needs canPublish (for its TTS audio track) AND canSubscribe AND canPublishData. A subscribe-only token cannot speak or intervene.
- For the continual-learning frame: feed engineer ACKs (was the collision real? did they accept the sync PR?) back over a data topic and persist them. That's the outcome signal that lets PodMan refine its intervention policy — it's the difference between 'dashboard' (banned) and 'agent that gets better the more it watches'.
**Sources**
- https://github.com/livekit/client-sdk-js (createLocalScreenTracks, setScreenShareEnabled/setCameraEnabled/setMicrophoneEnabled, Room.connect, RoomEvent.TrackSubscribed)
- https://github.com/livekit/node-sdks/blob/main/packages/livekit-server-sdk/README.md (AccessToken, addGrant VideoGrant fields, toJwt, install)
- https://github.com/livekit/node-sdks/blob/main/packages/livekit-rtc/README.md (Room.connect autoSubscribe/dynacast, AudioSource/AudioFrame publish, subarray-not-slice gotcha)
- https://context7.com/livekit/node-sdks/llms.txt (rtc-node publish audio/video RGBA frames, VideoBufferType.RGBA, publishData with reliable/topic/destination_identities)
- https://docs.livekit.io/agents/multimodality/vision/video (VideoStream frame-buffering pattern in TS + Python; Node.js NOT available for integrated live video; llm.createImageContent / ImageContent)
- https://docs.livekit.io/home/client/tracks/subscribe (VideoStream over a subscribed video track, iterate frames)
- https://docs.livekit.io/home/client/data/messages (publishData Uint8Array, RoomEvent.DataReceived, reliable 15KiB / lossy 1300B limits, topics)
- https://github.com/livekit/node-sdks/blob/main/packages/livekit-rtc/src/video_frame.ts (VideoFrame.data/width/height/type, convert(dstType, flipY?), getPlane; VideoBufferType enum: ARGB/RGBA/ABGR/BGRA/RGB24/I444/I422/I420/I420A/I010/NV12)
- https://github.com/livekit/agents-js (defineAgent, JobContext, voice.AgentSession, cli.runApp, ctx.connect)
- https://github.com/livekit/agents-js/blob/main/plugins/google/README.md (google.beta.realtime.RealtimeModel Gemini native audio, google.beta.TTS)
- https://github.com/livekit/agents-js/blob/main/agents-js/agents/src/voice/remote_session.ts (streamBytes for large reliable binary payloads with topic + destinationIdentities)
## GitHub integration for PodMan — official GitHub MCP server, Node consumption (MCP client SDK vs REST), and detecting unpushed-local collisions
RECOMMENDATION FOR THE HACKATHON: For PodMan's backend agent, call the GitHub REST API directly via Octokit (`octokit@5.0.5`) for the git-state half — it is faster to ship, fully typed, no extra process/transport to babysit, and you only need ~6 endpoints (list commits, list branches, compare refs, get file contents, create ref/branch, create PR). Keep the official GitHub MCP server as the "agent skill" surface IF a Gemini/Antigravity managed agent needs to discover tools dynamically; for that use the REMOTE server at https://api.githubcopilot.com/mcp/ with a PAT Bearer header (zero infra) and the stable MCP TypeScript client `@modelcontextprotocol/sdk@1.29.0`. Do NOT use Octokit AND MCP for the same calls — pick REST for deterministic backend logic, MCP only if you want the LLM to autonomously pick GitHub tools.
(a) OFFICIAL GITHUB MCP SERVER (github/github-mcp-server). Two ways to run: (1) REMOTE hosted by GitHub at `https://api.githubcopilot.com/mcp/` — transport type "http", no install; auth via OAuth 2.0 (recommended) OR a PAT sent as `Authorization: Bearer <PAT>`. (2) LOCAL Docker: `docker run -i --rm -e GITHUB_PERSONAL_ACCESS_TOKEN=<token> ghcr.io/github/github-mcp-server` (stdio transport). Toolsets are grouped and toggled via `GITHUB_TOOLSETS="repos,pull_requests,git,context"` (default on: context, repos, issues, pull_requests, users); read-only safety via `GITHUB_READ_ONLY=1` (also `--read-only` flag / X-MCP-Readonly header on remote). EXACT TOOLS PodMan needs (verified): repos toolset -> `list_commits`, `get_commit` (commit details incl. diff/files), `list_branches`, `create_branch`, `get_file_contents`, `search_code`, `search_commits`; pull_requests toolset -> `list_pull_requests`, `pull_request_read` (one tool with a `method` arg: get/get_diff/get_files/get_reviews/get_status — get_diff returns the unified diff string), `create_pull_request`, `update_pull_request`, `merge_pull_request`; git toolset -> `get_repository_tree`. Note: there is NO dedicated compare-two-refs MCP tool — to diff two branches via MCP you must fall back to commits or the REST compare endpoint, which is another reason to use Octokit directly for collision diffing.
(b) NODE CONSUMPTION — MCP client SDK vs direct API. Direct Octokit (WINNER for hackathon): `npm i octokit` (v5.0.5) gives `new Octokit({auth: PAT})` then `octokit.rest.repos.listCommits/getCommit/listBranches/createRef`, `octokit.rest.pulls.list/create`, `octokit.rest.repos.compareCommitsWithBasehead` for branch-vs-branch diffs, and `octokit.rest.repos.getContent`. Typed, single dependency, no transport lifecycle. MCP client path (only if an LLM should auto-select tools): `npm i @modelcontextprotocol/sdk@1.29.0` — STABLE v1, imports use `.js` subpaths: `import { Client } from '@modelcontextprotocol/sdk/client/index.js'` and `import { StreamableHTTPClientTransport } from '@modelcontextprotocol/sdk/client/streamableHttp.js'`; pass the PAT via `requestInit.headers`. Then `client.listTools()` / `client.callTool({name, arguments})`. CAUTION: Context7/some docs surface `@modelcontextprotocol/client@alpha` with bare subpaths (no `.js`) and an `authProvider` option — that is the v2 ALPHA, do NOT use it for the hackathon; the v1.29.0 shape above is the stable one matching the npm exports map I verified.
(c) CRITICAL ARCHITECTURE — detecting "two engineers on the same feature, not yet pushed". The GitHub API is BLIND to unpushed local commits and uncommitted working-tree edits — confirmed, this is PodMan's moat. Recommended two-tier fusion: TIER 1 (always-on, the demo hero) = realtime SCREEN VISION. Gemini 3.5 Flash (vision) reads each engineer's editor frame from the LiveKit screen-share track and emits a structured "active-edit" signal per person: { engineer, repoGuess, filePath(s) visible in tab/title bar/file tree, symbol/function under cursor, rough change summary }. PodMan maintains a live in-memory map filePath -> {engineers actively editing}. A COLLISION fires when 2+ engineers' vision signals point at the same file/feature AND git shows neither change is pushed yet. TIER 2 (optional, makes it bulletproof) = a tiny LOCAL GIT REPORTER: a ~30-line Node/bash sidecar each engineer runs that posts `git status --porcelain`, `git branch --show-current`, `git rev-list @{u}..HEAD` (count of unpushed commits) and `git diff --name-only` to PodMan over the LiveKit data channel or a WS endpoint every few seconds. This converts the fuzzy vision signal into ground-truth file paths + unpushed-commit counts. FUSION RULE for an intervention: collision = (same normalized file path or same feature/branch) seen across ≥2 engineers, where ≥1 has local-but-unpushed work (detected by Tier 2 unpushed count >0, OR inferred by Tier 1 when that file is NOT yet present in the remote branch via Octokit getContent/compare). Then PodMan: voice-warns + shows a card, pulls the remote file via Octokit getContent (or compareCommitsWithBasehead between their branches) to render the would-be diff, and offers create_branch + create_pull_request to sync. CONTINUAL-LEARNING framing: persist every {filePath, engineer, branch, wasRealCollision, acceptedPR} outcome to MongoDB Atlas + Voyage embeddings to learn code ownership, recurring conflict hotspots, and to tune the intervention threshold over the session.
**Model / IDs:** `Gemini 3.5 Flash (native vision + Computer Use) — reads each engineer's editor frame from the LiveKit screen-share track to emit structured active-edit signals (file paths, symbol under cursor, change summary)`, `Gemini 3.5 Live API — realtime audio for PodMan's voice intervention when a collision fires`, `Voyage AI embeddings (voyage-3 family) — embed collision/ownership outcomes into MongoDB Atlas vector search for the continual-learning memory layer`
**Packages**
- `octokit` — npm i octokit
_v5.0.5. THE recommended path for PodMan's deterministic git-state logic. Bundles @octokit/rest (v22.0.1). new Octokit({auth: process.env.GITHUB_PAT}). Fully typed, Node + browser/edge compatible. Use rest.repos.compareCommitsWithBasehead({owner,repo,basehead:'main...feature'}) for branch-vs-branch diffs since MCP has no compare tool._
- `@modelcontextprotocol/sdk` — npm i @modelcontextprotocol/sdk
_STABLE v1.29.0. Only needed if you want an LLM/managed-agent to auto-discover GitHub tools via MCP. Node-compatible. Imports MUST end in .js: '@modelcontextprotocol/sdk/client/index.js' and '@modelcontextprotocol/sdk/client/streamableHttp.js'. Pass PAT via transport requestInit.headers. Avoid the @modelcontextprotocol/client@alpha (v2) docs — different import shape._
- `github/github-mcp-server (remote, no install)` — connect to https://api.githubcopilot.com/mcp/ with header Authorization: Bearer <PAT>
_Zero-infra remote MCP server. Set GITHUB_READ_ONLY behavior via X-MCP-Readonly:true header and GITHUB_TOOLSETS via X-MCP-Toolsets header on remote, or env vars when local._
- `github/github-mcp-server (local Docker)` — docker run -i --rm -e GITHUB_PERSONAL_ACCESS_TOKEN=<token> -e GITHUB_TOOLSETS=repos,pull_requests,git ghcr.io/github/github-mcp-server
_stdio transport; use only if you cannot reach the remote or want full local control. Add -e GITHUB_READ_ONLY=1 for safety during demos._
**Critical snippets**
### RECOMMENDED: direct Octokit for PodMan git-state + collision diffing
```typescript
import { Octokit } from 'octokit';
const gh = new Octokit({ auth: process.env.GITHUB_PAT });
// who pushed what recently
const { data: commits } = await gh.rest.repos.listCommits({ owner, repo, per_page: 20 });
const { data: branches } = await gh.rest.repos.listBranches({ owner, repo });
// branch-vs-branch diff (the would-be collision view) — MCP has NO compare tool, REST does
const { data: cmp } = await gh.rest.repos.compareCommitsWithBasehead({
owner, repo, basehead: 'main...alice-feature'
});
const changedFiles = cmp.files?.map(f => f.filename) ?? [];
// is this file already on the remote branch? (if not, the edit on someone's screen is unpushed)
const exists = await gh.rest.repos.getContent({ owner, repo, path: 'src/auth.ts', ref: 'main' })
.then(() => true).catch(() => false);
// PodMan's sync intervention: create branch + PR
const { data: mainRef } = await gh.rest.git.getRef({ owner, repo, ref: 'heads/main' });
await gh.rest.git.createRef({ owner, repo, ref: 'refs/heads/podman-sync', sha: mainRef.object.sha });
await gh.rest.pulls.create({ owner, repo, title: 'PodMan: sync auth.ts before collision', head: 'podman-sync', base: 'main', body: 'Alice & Bob both editing src/auth.ts (unpushed). Sync now.' });
```
> Single dependency, typed, no transport to manage. This is the fast hackathon path for the deterministic backend logic.
### OPTIONAL: consume the official GitHub MCP server from Node (stable v1)
```typescript
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StreamableHTTPClientTransport } from '@modelcontextprotocol/sdk/client/streamableHttp.js';
const transport = new StreamableHTTPClientTransport(
new URL('https://api.githubcopilot.com/mcp/'),
{ requestInit: { headers: { Authorization: `Bearer ${process.env.GITHUB_PAT}` } } }
);
const client = new Client({ name: 'podman', version: '1.0.0' });
await client.connect(transport);
const { tools } = await client.listTools(); // discover create_pull_request, list_commits, pull_request_read, ...
const diff = await client.callTool({
name: 'pull_request_read',
arguments: { method: 'get_diff', owner, repo, pullNumber: 42 }
});
await client.callTool({
name: 'create_pull_request',
arguments: { owner, repo, title: 'PodMan sync', head: 'podman-sync', base: 'main', body: 'collision detected' }
});
```
> Use ONLY if you want an LLM/managed agent to auto-select GitHub tools. Note import paths end in .js — that is the stable v1.29.0 shape, NOT the alpha @modelcontextprotocol/client package.
### Tier-2 local git reporter sidecar (each engineer runs this) — fills the API blind spot
```bash
# unpushed commit count (invisible to GitHub API until pushed)
git rev-list --count @{u}..HEAD 2>/dev/null || echo 0
# uncommitted working-tree edits, machine-readable
git status --porcelain
# files touched but not yet committed
git diff --name-only
git branch --show-current
```
> Wrap in a ~30-line Node watcher that POSTs JSON {engineer, branch, unpushedCount, dirtyFiles[]} over the LiveKit data channel every few seconds. This turns the fuzzy vision signal into ground-truth file paths + proof the work is unpushed.
### Fusion rule (pseudo) — when PodMan fires a collision intervention
```typescript
// per-file map built from Tier-1 vision signals + Tier-2 reporter
const editors = activeEditsByFile.get(normalize(path)); // Set<engineer>
if (editors && editors.size >= 2) {
const anyUnpushed = [...editors].some(e =>
reporter[e]?.unpushedCount > 0 || // Tier 2 ground truth
!remoteHasFile(path, branchOf(e)) // Tier 1 inference via Octokit getContent/compare
);
if (anyUnpushed) intervene({ file: path, engineers: [...editors] });
}
```
> Collision = same file/feature across >=2 engineers AND >=1 has local-but-unpushed work. anyUnpushed is the crux the GitHub API alone cannot answer.
**Gotchas**
- GitHub API is BLIND to unpushed local commits and uncommitted edits — this is by design and is exactly PodMan's moat. Never claim collision detection from git state alone; the vision layer (and/or the local reporter) is load-bearing, not decorative.
- The official GitHub MCP server has NO compare-two-refs tool. To diff branch-vs-branch (the core collision view) you MUST use the REST compare endpoint (octokit.rest.repos.compareCommitsWithBasehead, basehead 'main...feature'). Another reason to lead with Octokit.
- pull_request_read is ONE tool with a `method` argument (get / get_diff / get_files / get_reviews / get_status), not separate tools. get_diff returns the full unified diff as a string and has NO pagination — large PRs can return huge payloads and have crashed IDE clients (issue #2122). Cap/scope what you feed Gemini.
- MCP SDK package confusion: stable is @modelcontextprotocol/sdk@1.29.0 with .js subpath imports and PAT via requestInit.headers. Context7/alpha docs show @modelcontextprotocol/client (no .js, authProvider option) — that's the v2 ALPHA. Mixing them yields module-not-found / wrong-API errors. Pin v1 for the hackathon.
- PAT scopes: a fine-grained PAT needs Contents: Read (commits, branches, file contents), Pull requests: Read & Write (list + create PRs), and Metadata: Read (mandatory). A classic PAT needs the `repo` scope. Repos must be PUBLIC per hackathon rules, but a token is still required to CREATE branches/PRs.
- Hackathon rule reminder: do NOT make the pod grid/dashboard the hero (dashboards are a banned hero). Frame the GitHub integration as the proactive create_branch + create_pull_request intervention triggered by the realtime collision, not as a repo dashboard.
- Rate limits: PAT-authenticated REST is 5,000 req/hr. PodMan polling many repos/branches every few seconds will burn this fast — cache git state, use conditional requests (ETag), and prefer webhooks or longer poll intervals for push events.
- Remote MCP server toolset/read-only toggles on the hosted endpoint are set via request HEADERS (e.g. X-MCP-Toolsets, X-MCP-Readonly) since you can't pass env vars to a hosted server; only the local Docker run uses GITHUB_TOOLSETS / GITHUB_READ_ONLY env vars.
- Vision filePath extraction is fuzzy (editor tab text, breadcrumb, file-tree highlight). Normalize paths (strip workspace prefixes, lowercase on case-insensitive FS) before matching across engineers, or the same file from two screens won't match. The Tier-2 local reporter eliminates this ambiguity — strongly recommend shipping it for the demo's reliability.
**Sources**
- https://github.com/github/github-mcp-server
- https://docs.github.com/en/copilot/how-tos/provide-context/use-mcp-in-your-ide/set-up-the-github-mcp-server
- https://github.blog/changelog/2025-06-12-remote-github-mcp-server-is-now-available-in-public-preview/
- https://github.blog/ai-and-ml/generative-ai/a-practical-guide-on-how-to-use-the-github-mcp-server/
- https://github.com/github/github-mcp-server/issues/2122
- https://github.com/modelcontextprotocol/typescript-sdk/blob/main/docs/client.md
- https://www.npmjs.com/package/@modelcontextprotocol/sdk
- https://octokit.github.io/rest.js/
- https://docs.github.com/en/rest/commits/commits?apiVersion=2022-11-28
- https://docs.github.com/en/rest/commits/commits#compare-two-commits
- https://www.npmjs.com/package/octokit
## DigitalOcean deployment for PodMan (realtime Node websocket/LiveKit backend, React Vite PWA, DB) + "Best DigitalOcean" prize strategy
RECOMMENDED ARCHITECTURE for the hackathon (fastest path that still wins "Best DigitalOcean"):
1) BACKEND (PodMan agent + LiveKit agent worker, Node + websockets) -> DigitalOcean App Platform "Web Service" deployed from your GitHub repo. App Platform natively supports websockets/wss, auto-TLS, deploy-on-push, and zero-config Node detection. It is dramatically faster than provisioning a Droplet. ONE important caveat: the LiveKit *agent* (the bot that joins the room to do Gemini vision) is an outbound worker, not an inbound HTTP server — it should be a separate App Platform component of type "Worker" (no http_port, no public route). Your token/REST server (issues LiveKit join tokens, serves the collision API/websocket to the frontend) is the "Web Service" with http_port. Run them as two components in ONE app spec.
2) FRONTEND (React + Vite PWA) -> App Platform "Static Site" component in the SAME app. Free tier (3 static sites free), auto-detects Vite's `dist` output, free TLS + CDN. Putting frontend + backend in one app gives you a single domain (clean for the demo, no CORS) — App Platform routes by path. Alternative: Vercel is also fine for the PWA and arguably faster DX, but keeping it on DO strengthens the "Best DigitalOcean" prize story. Recommendation: keep PWA on DO App Platform static site.
3) DATABASE -> Use MongoDB Atlas Sandbox (free, provided to attendees) for speed, OR DO Managed MongoDB if you want everything on DO. DO DOES offer Managed MongoDB (plus Postgres/MySQL/Valkey/Kafka/OpenSearch), starting ~$15/mo. For the prize, DO Managed MongoDB is a nice stack item but Atlas Sandbox is free and pairs with Voyage AI vector search out of the box. Recommendation: Atlas Sandbox for the memory/vector layer to save the $200 credit; mention DO Managed DB as the production path.
4) "openclaw" IS a real DO product. The user's "openclaw" = OpenClaw (formerly Moltbot/Clawdbot, by Peter Steinberger) — a viral open-source personal-AI-agent framework that DO promotes heavily with 1-Click Droplet deploy AND an App Platform path. It is a FRAMEWORK for always-on proactive agents connected to messaging (Slack/Discord/WhatsApp/Telegram). It is NOT required for PodMan and you should NOT rebuild on it (it's a personal assistant pattern, wrong shape for a team collision detector). BUT name-dropping/optionally wiring a PodMan notification channel through it, or simply deploying your custom agent on the same App Platform that DO markets for OpenClaw, is on-message for judges.
PRIZE STRATEGY ("Best DigitalOcean"): The strongest, simplest winning move is (a) deploy the whole PodMan stack (web service + worker + static site, ideally + Managed DB) on App Platform via a single app spec, AND (b) use DigitalOcean Gradient AI Serverless Inference for at least one model call (it is OpenAI-SDK-compatible, hosts Claude/GPT/Llama/Mistral via one MODEL_ACCESS_KEY at https://inference.do-ai.run/v1/). That way DO touches realtime hosting + inference + data, which is exactly the "built on DO" story judges reward. Note: Gemini is NOT on Gradient, so keep the load-bearing vision on Gemini Live (for the $5000 Gemini prize) and route a secondary call (e.g. the collision-summary/intervention-policy LLM, or embeddings fallback) through DO Gradient to legitimately claim DO usage.
CREDITS: Claim the $200 via the MLH/hackathon DO signup link (mlh.link/digitalocean-signup or the organizer-provided code). New accounts get $200 valid ~60 days after adding a payment method. Apply at signup or in Billing -> add promo code. App Platform free tier covers 3 static sites; the web service + worker + Managed DB draw from credits (well under $200 for a hackathon).
**Model / IDs:** `DO Gradient Serverless Inference: Anthropic Claude (incl. Claude Opus 4.6), OpenAI GPT-4o, Meta Llama 3, Mistral — all via OpenAI-compatible API at https://inference.do-ai.run/v1/ with one MODEL_ACCESS_KEY`, `NOT on DO: Google Gemini (keep Gemini Live vision on Google AI Studio / Gemini API directly)`
**Packages**
- `doctl (DigitalOcean CLI)` — brew install doctl # then: doctl auth init
_Used to create/update the app from your app spec: doctl apps create --spec .do/app.yaml ; doctl apps update <id> --spec .do/app.yaml. Also doctl apps logs <id> for live logs during the demo._
- `openai (Node SDK) — for DO Gradient Serverless Inference` — npm i openai
_DO Gradient is OpenAI-API-compatible. Point baseURL at https://inference.do-ai.run/v1/ and use the MODEL_ACCESS_KEY. Lets you call Claude/GPT/Llama/Mistral on DO for the 'Best DigitalOcean' angle without changing SDKs._
- `livekit-server-sdk (Node)` — npm i livekit-server-sdk
_Token minting on the Web Service component. The LiveKit *agent* itself (joins rooms, does Gemini vision) runs as the Worker component._
- `ws / express` — npm i ws express
_Standard Node websocket stack; works on App Platform as long as the service binds 0.0.0.0:8080 and the service is a Web Service (publicly routable). Confirmed by DO's official sample-websocket repo._
**Critical snippets**
### Single App Platform app spec: static PWA + web (token/ws API) + worker (LiveKit agent)
```yaml
# .do/app.yaml -> deploy: doctl apps create --spec .do/app.yaml
name: podman
region: nyc
static_sites:
- name: web
github:
repo: <org>/Podman
branch: main
deploy_on_push: true
source_dir: frontend
build_command: npm ci && npm run build
output_dir: dist # Vite default; auto-detected if omitted
routes:
- path: / # PWA served at the apex path
services:
- name: api # token mint + collision REST + ws to the PWA
github:
repo: <org>/Podman
branch: main
deploy_on_push: true
source_dir: backend
run_command: node dist/server.js
http_port: 8080 # MUST bind 0.0.0.0:8080, not localhost
instance_size_slug: apps-s-1vcpu-1gb
instance_count: 1
routes:
- path: /api # ws upgrades over wss://<app>.ondigitalocean.app/api
envs:
- { key: LIVEKIT_API_KEY, scope: RUN_TIME, type: SECRET }
- { key: LIVEKIT_API_SECRET, scope: RUN_TIME, type: SECRET }
- { key: GEMINI_API_KEY, scope: RUN_TIME, type: SECRET }
- { key: MODEL_ACCESS_KEY, scope: RUN_TIME, type: SECRET } # DO Gradient
- { key: MONGODB_URI, scope: RUN_TIME, type: SECRET }
workers:
- name: podman-agent # LiveKit agent: joins rooms, runs Gemini vision
github:
repo: <org>/Podman
branch: main
deploy_on_push: true
source_dir: backend
run_command: node dist/agent.js # outbound worker: NO http_port, NO routes
instance_size_slug: apps-s-1vcpu-1gb
instance_count: 1
envs:
- { key: LIVEKIT_URL, scope: RUN_TIME, value: wss://<your>.livekit.cloud }
- { key: LIVEKIT_API_KEY, scope: RUN_TIME, type: SECRET }
- { key: LIVEKIT_API_SECRET, scope: RUN_TIME, type: SECRET }
- { key: GEMINI_API_KEY, scope: RUN_TIME, type: SECRET }
```
> Key design point: inbound HTTP/ws server = `service` (has http_port + route); the LiveKit agent that DIALS OUT to join rooms = `worker` (no port, no route). Both can live in the same repo/source_dir with different run_commands. Frontend talks to the API over wss because App Platform terminates TLS (always HTTPS).
### Frontend wss protocol selection (App Platform is always HTTPS)
```javascript
// On App Platform everything is served over 443/HTTPS, so ws:// will be blocked
// as mixed content. Pick the scheme from the page protocol:
const proto = location.protocol === 'https:' ? 'wss' : 'ws';
const ws = new WebSocket(`${proto}://${location.host}/api/collisions`);
```
> This is the #1 websocket gotcha DO calls out in their official sample-websocket repo. Hardcoding ws:// works locally but fails in prod.
### Call DigitalOcean Gradient Serverless Inference from Node (OpenAI-compatible)
```javascript
import OpenAI from 'openai';
// One MODEL_ACCESS_KEY unlocks Claude / GPT / Llama / Mistral on DO.
const client = new OpenAI({
apiKey: process.env.MODEL_ACCESS_KEY,
baseURL: 'https://inference.do-ai.run/v1/',
});
// e.g. summarize a detected collision / decide the intervention policy on DO
const res = await client.chat.completions.create({
model: 'anthropic-claude-opus-4-6', // Claude Opus 4.6 is live on Gradient
messages: [{ role: 'user', content: collisionContext }],
});
```
> This is the cheap, legitimate way to put DO inference in the loop for the 'Best DigitalOcean' prize WITHOUT giving up the load-bearing Gemini vision (Gemini is NOT on Gradient). Get the key in Control Panel: INFERENCE -> Manage -> Model Access Keys. Verify exact model id strings in the DO inference docs at deploy time.
### Minimal Dockerfile (only if buildpack auto-detect misbehaves)
```dockerfile
# App Platform auto-detects Node via buildpacks, so a Dockerfile is usually
# UNNECESSARY. Add one only if you need a custom build. Reference it with
# dockerfile_path: backend/Dockerfile in the component spec.
FROM node:20-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
EXPOSE 8080
CMD ["node", "dist/server.js"]
```
> Prefer NO Dockerfile for hackathon speed — let App Platform's Node buildpack handle it. Only add this if you hit a build edge case.
**Gotchas**
- Service vs Worker: the LiveKit AGENT must be a `worker` component (no http_port, no public route) — it dials out to join rooms. Only the token/API/ws server is a `service`. Putting the agent as a service with a port will make App Platform wait for a health check on a port nothing listens on and the deploy will fail.
- Bind to 0.0.0.0, not localhost/127.0.0.1. App Platform routes external traffic to 0.0.0.0:http_port (default 8080). Binding to localhost = silent 'no healthy upstream'.
- Websockets MUST use wss:// in production — App Platform always serves over 443/HTTPS, so ws:// triggers mixed-content blocking. Use the protocol-switch snippet. (This is DO's own documented #1 websocket pitfall.)
- Websockets only work on the publicly-routable web service, NOT on internal service-to-service ports. Route ws through the public app domain.
- Gemini is NOT available on DO Gradient inference (only Anthropic/OpenAI/Meta/Mistral). Keep your load-bearing realtime vision on Google's Gemini Live API (also protects the $5000 Gemini prize) and route a SECONDARY model call (collision summary / intervention-policy reasoning / embeddings) through DO Gradient to legitimately claim DO inference usage.
- App Platform free tier covers 3 STATIC sites only; the web service + worker + Managed DB consume credits/billing. Total for a hackathon is tiny (single small instances), well under $200, but it is not $0 — make sure the $200 credit is applied first.
- $200 credits are for NEW DO accounts, valid ~60 days, and require adding a payment method before they apply. Use the MLH/organizer DO link (e.g. mlh.link/digitalocean-signup) or the event promo code — a generic signup may only give the smaller standard trial.
- gVisor sandbox: App Platform runs containers under gVisor; a few exotic syscalls are unsupported. Standard Node + ws + LiveKit SDK are fine, but if a native dependency does something unusual, fall back to a Droplet for that component.
- Demo-safety rule: the hackathon bans 'dashboard-as-hero'. App Platform makes it trivial to expose the pod grid — keep the deployed hero route as the proactive intervention card/voice, not the grid.
- DON'T rebuild PodMan on OpenClaw. OpenClaw is a real DO-promoted product but it's a PERSONAL assistant framework (messaging-channel bot), the wrong shape for a multi-engineer collision detector. At most, wire a PodMan alert channel through it as a bonus, or just note you deploy on the same App Platform DO markets for OpenClaw.
- If you want EVERYTHING on DO for the prize, DO Managed MongoDB exists (~$15/mo) — but for speed and to save credits, the free MongoDB Atlas Sandbox + Voyage AI for the vector/memory layer is the pragmatic choice.
**Sources**
- https://github.com/digitalocean/sample-websocket
- https://www.digitalocean.com/community/questions/websocket-use-in-app-platform-wss
- https://docs.digitalocean.com/products/app-platform/details/limits/
- https://docs.digitalocean.com/products/app-platform/reference/app-spec/
- https://docs.digitalocean.com/products/app-platform/reference/dockerfile/
- https://github.com/digitalocean/sample-vite-react
- https://docs.digitalocean.com/products/app-platform/how-to/manage-static-sites/
- https://www.digitalocean.com/pricing/app-platform
- https://www.digitalocean.com/products/ai-platform
- https://docs.digitalocean.com/products/gradient-ai-platform/
- https://www.digitalocean.com/community/tutorials/serverless-inference-openai-sdk
- https://docs.digitalocean.com/products/inference/how-to/si-endpoints/
- https://docs.digitalocean.com/products/inference/how-to/model-access-keys/
- https://www.digitalocean.com/blog/claude-opus-4-6-gradient-ai-platform
- https://www.digitalocean.com/blog/openclaw-digitalocean-app-platform
- https://docs.digitalocean.com/products/marketplace/catalog/openclaw/
- https://docs.digitalocean.com/products/databases/
- https://www.digitalocean.com/pricing/managed-databases
- https://www.mlh.com/partners/digitalocean
- https://github.com/digitalocean/app_action
docs/generated/STRATEGY.md
-51
View File
@@ -1,51 +0,0 @@
# PodMan — Winning strategy
**Track:** Continual Learning
**One-liner:** PodMan is the Jarvis for engineering teams: it watches every engineer's screen in realtime, fuses that live context with your GitHub state to catch merge collisions the API literally cannot see — because the changes are still unpushed on someone's laptop — and the more it watches, the better it learns who owns what and when to step in.
## Why this track
Continual Learning is the only track where PodMan's core mechanic IS the track definition, not a bolted-on framing. The other two are weaker fits and invite skepticism. (1) "Recursive Intelligence" / agents-building-agents implies PodMan should spawn or rewrite sub-agents — PodMan doesn't; claiming it would be a lie judges can smell, and you'd be compared against literal AutoML/agent-factory demos you can't out-build in 20h. (2) "The Self-Improvement Stack" is about tooling/infra that helps OTHER systems improve (eval harnesses, observability, RL pipelines) — PodMan is an end-user product, not dev infrastructure, so it reads as off-theme. (3) Continual Learning = "a system that gets more useful the longer it runs, accumulating knowledge with minimal human supervision, without catastrophic forgetting." That is exactly PodMan's loadbearing loop: it watches screens + git over a session and builds a persistent team world-model (who owns which files/features, what's in-flight, each engineer's style, recurring conflict patterns) stored in MongoDB Atlas + Voyage embeddings, and it gets measurably better at predicting collisions the more it observes. Critically, it has a SECOND, judge-pleasing learning loop that doubles as the hackathon's recursive-self-improvement theme: PodMan learns its own intervention POLICY from outcomes (was the flagged collision real? did the human accept the sync PR? did they dismiss it?), so its precision/recall improves within the demo. Two stacked learning loops (world-model + policy) let you show "it gets smarter" twice in 3 minutes, which is the single most important thing for this track. Pick Continual Learning; it is defensible, demonstrable live, and uniquely yours.
## DQ risks
- Dashboard-as-hero trap: the pod grid of live screen tiles is visually the most obvious thing on screen and is on the explicit BANNED list ('any project where a dashboard is the main feature'). If the demo opens on, lingers on, or returns to the grid as the centerpiece, a judge can bucket you as a banned dashboard project.
- Image-analyzer mislabel: 'vision model looks at screenshots' can be pattern-matched to the banned 'image analyzers' category if you describe PodMan as analyzing images rather than as a proactive multi-agent that acts on fused screen+git state.
- Private-repo / setup violation: rule requires PUBLIC repos and the demo must show ONLY hackathon-built work. Demoing against a private team repo, or wiring in a pre-existing side project, is an instant DQ.
- Team-size / pre-existing-work violation: max 4 members and NEW work only. Reusing a prior 'screen-watching agent' codebase, or showing more than 4 participants in the pod, breaks the rules.
- Faked-realtime risk: if the collision 'detection' is obviously hardcoded/scripted with no real vision+git inference, judges reading it as a canned animation undercuts both the Technicality score and the legitimacy of the demo (borderline 'show only what you built').
- Consent/recording optics: screen+mic+webcam capture of teammates can read as a privacy red flag to judges; not a formal DQ but it can sour the room if not addressed.
- MiniMax/Modular/unused-sponsor confusion: claiming prizes for tools you didn't meaningfully use (e.g. listing MiniMax or Mojo without real integration) can read as padding and hurt credibility with sponsor judges.
## DQ mitigations
- Reframe the hero as the intervention, not the grid: open the demo on a single engineer's normal coding view (IDE), NOT the pod grid. The pod grid appears only for ~3-5 seconds as 'PodMan's peripheral vision,' then collapses to a corner. The hero shot is the proactive PodMan card + voice interrupting the moment a collision is imminent. Say the words out loud: 'PodMan is a proactive agent, not a dashboard — the screens are just its eyes.'
- Always describe PodMan as an agent that UNDERSTANDS and ACTS on fused screen+git state, never as an 'image analyzer.' Lead with the action (warn / show diff / open sync PR) and the GitHub fusion, so vision is plumbing, not the product.
- Create the demo repo PUBLIC on GitHub up front; show the github.com URL on screen during the demo so judges can see it's public. Keep all code in the new /Users/karti/Desktop/Podman repo (confirmed scaffolded fresh today, empty except placeholders) — nothing pre-existing.
- Cap the visible pod at <=4 members and have exactly the hackathon team in it; state team size in the submission. Keep the git history in the public repo as proof all commits are from the hackathon window.
- Make realtime genuinely load-bearing AND legible: pipe real LiveKit screen tracks to Gemini 3.5 Flash vision, show the live transcript/inference ('I see Alice editing auth/session.ts; Bob has unpushed changes to the same file') as on-screen captions so judges SEE real inference happening, not a scripted card. Have a deterministic fallback path rehearsed in case wifi dies, but the primary run must be live.
- Add a one-line consent gate ('each engineer opts in to share') and a visible 'PodMan is watching' indicator; mention privacy-by-design in one breath. Turns a red flag into a maturity signal.
- Only claim sponsor prizes you actually integrate: DigitalOcean (deploy), LiveKit (transport), Gemini 3.5 (vision+Live API voice+Live Translate), MongoDB Atlas + Voyage (memory). Drop MiniMax/Modular from the pitch unless one is genuinely wired in; do not pad the sponsor list.
## Prize stacking
- **Best Usage of Gemini 3.5 ($5000 cash) — PRIMARY TARGET** — Make Gemini the brain on three distinct surfaces so it's undeniably the core, not a call: (1) Gemini 3.5 Flash native Computer Use + vision interprets each live screen track frame-by-frame into structured 'work intent' (file, feature, function being edited). (2) Gemini Live API drives PodMan's realtime VOICE intervention — it speaks the warning and converses ('want me to open a sync PR?'). (3) Gemini Live Translate makes the voice intervention multilingual for a distributed team — a clean, demo-able wow that almost no one else will show. Optionally use the Managed Agents / Interactions API (antigravity-preview-05-2026) to host PodMan's agent loop. On stage, explicitly enumerate 'three Gemini 3.5 surfaces' so the Gemini judge can check every box.
- **Best LiveKit** — LiveKit is the realtime spine and you should say it's irreplaceable: screen-share + mic + webcam ingest per pod member over LiveKit tracks; PodMan joins each room as an AI participant (LiveKit Agents) that consumes video tracks and publishes its voice + intervention cards back over LiveKit data channels. The MOAT line — 'unpushed changes are invisible to the GitHub API, so realtime capture is the only way' — is literally a LiveKit pitch. Show the agent as a real participant in the room.
- **Best DigitalOcean** — Deploy the PodMan backend agent + the LiveKit egress/worker + the React PWA on DigitalOcean (App Platform or a Droplet) using the $200 credits. Put the public demo URL on a DO-hosted domain and show it in the browser during the demo. Mention the deploy target by name in the submission. Keep MongoDB on Atlas (separate sponsor) and embeddings on Voyage — that's fine, DO hosts the compute.
- **MongoDB Atlas + Voyage AI (memory/continual-learning credibility)** — This is what makes the Continual Learning claim TRUE and technical, not hand-wavy. Store the evolving team world-model + intervention-outcome log in Atlas; embed work-intent snapshots and past conflicts with Voyage AI; use Atlas Vector Search so PodMan retrieves 'have we seen this collision pattern before?' This directly powers the 'watch it get smarter' money moment and earns a second sponsor's attention while reinforcing the chosen track.
## Maximizing each judging criterion
- **Technicality (40%)** — This is 40% — over-index here. Show the hard part explicitly: realtime multi-track video fusion (LiveKit) -> per-frame vision inference (Gemini 3.5 Flash) -> structured work-intent -> JOINED with live git state (local diff via a lightweight per-machine agent + GitHub API) to detect a collision class that is provably undetectable by the API alone. Name the invisible-unpushed-changes insight as the technical moat. Add the vector-memory continual-learning loop (Atlas + Voyage) and the policy-learning-from-outcomes loop. Display live inference captions so judges see real model output, not a script. One architecture slide (LiveKit -> Gemini -> fusion engine -> memory -> intervention) shown for 10s cements it.
- **Creativity / Originality (25%)** — Lead with the counterintuitive insight nobody else will have: 'the most dangerous merge conflicts don't exist in GitHub yet — they're sitting unpushed on someone's laptop, and the only way to see them is to watch the work happen.' That single reframe is the originality hook. 'Jarvis for the whole team' (ambient, proactive, voice) vs the field's solo copilots is a fresh category. The Gemini Live Translate multilingual intervention is an unexpected, memorable flourish.
- **Live Demo (20%)** — Choreograph a single unbroken 'collision caught live' beat (see demo script) where PodMan interrupts BEFORE the push with voice + card. Then the 'it got smarter' beat where a second, similar near-collision is caught faster/more confidently because of what it learned in the first. Rehearse for wifi failure with a hot local fallback, but run live. Keep the banned dashboard off-hero. End on PodMan opening a real sync PR on the public repo — a concrete artifact judges can click.
- **Future Potential & AI Impact (15%)** — Frame as the ambient coordination layer for every engineering org — generalizes beyond merge collisions to duplicated work, onboarding ('PodMan already knows who owns what'), and incident response. Tie to the recursive-self-improvement theme: an agent that compounds team knowledge and refines its own intervention policy is a template for org-scale continual learning. One sentence on the wedge: starts as collision-prevention, becomes the team's shared memory.
## Demo script
1. [0:00-0:20] COLD OPEN ON THE PAIN, NOT THE PRODUCT. Full screen: Alice's IDE, editing auth/session.ts. Voiceover: 'This is Alice. Across the room, Bob is also editing auth/session.ts right now — but he hasn't pushed. GitHub has no idea. In ten minutes, one of them loses an hour to a merge conflict.' (No dashboard on screen yet. Establish the invisible problem.)
2. [0:20-0:40] INTRODUCE PODMAN AS EYES, COLLAPSE THE GRID FAST. Briefly reveal the pod: 4 live tiles. 'PodMan watches every engineer's screen in realtime over LiveKit and understands the work with Gemini 3.5 vision.' Show live caption appearing: 'Alice -> editing auth/session.ts (login flow). Bob -> editing auth/session.ts (token refresh), 14 unpushed lines.' Then COLLAPSE the grid into a small corner widget. Say it: 'PodMan isn't a dashboard — that's just its peripheral vision. Here's what it DOES.'
3. [0:40-1:25] THE MONEY MOMENT — CATCH IT LIVE, BEFORE THE PUSH. As Bob moves to commit, PodMan's voice (Gemini Live API) interrupts in realtime: 'Hold on Bob — Alice is editing the same function in session.ts and hasn't pushed. You'll collide.' An intervention card appears with the live diff of the overlapping region. PodMan offers: 'Want me to open a sync PR so you rebase cleanly?' Click yes -> PodMan opens a REAL PR on the PUBLIC GitHub repo (show the github.com URL). This is the single hero beat; let it breathe.
4. [1:25-2:05] WATCH IT GET SMARTER. Trigger a second, similar near-collision (Carol + Bob on payments/webhook.ts). PodMan catches it faster and with higher confidence, and SHOWS WHY via its memory: 'I've seen this pattern — session and webhook handlers conflict in this repo. Confidence 0.93.' Show the Atlas/Voyage memory entry that was written from beat one being retrieved now. Then the policy loop: 'Last time you accepted my sync PR, so I'll lead with that.' Two learning loops visible in 40 seconds = the Continual Learning proof.
5. [2:05-2:30] ONE FLOURISH + ARCHITECTURE. Carol is remote and Spanish-speaking — PodMan delivers the same intervention via Gemini Live Translate in Spanish voice. Flash the architecture slide for ~10s: LiveKit (ingest) -> Gemini 3.5 (vision + Live voice + translate) -> fusion engine (screen intent x git state) -> Atlas + Voyage memory -> proactive intervention. Name the three Gemini surfaces out loud for the $5k judge.
6. [2:30-3:00] CLOSE ON THE INSIGHT + IMPACT + ASK. 'The most dangerous conflicts aren't in GitHub yet — they're unpushed on someone's laptop. The only way to catch them is to watch the work, and the longer PodMan watches, the better it knows your team. It's the ambient coordination layer for engineering — Jarvis for the whole team.' End on the open PR + the public repo URL + the live DigitalOcean-hosted app. Stop talking; let the caught-collision artifact be the last thing on screen.
docs/generated/files/backend/src/agent.ts
-79
View File
@@ -1,79 +0,0 @@
import {
Room,
RoomEvent,
TrackKind,
TrackSource,
VideoStream,
VideoBufferType,
dispose,
type RemoteTrack,
type RemoteTrackPublication,
type RemoteParticipant,
} from '@livekit/rtc-node';
import sharp from 'sharp';
import { AccessToken } from 'livekit-server-sdk';
import { env } from './env.js';
import { PodMan } from './agent/podman.js';
const POD_ROOM = process.env.POD_ROOM ?? 'demo-pod';
const SAMPLE_INTERVAL_MS = 1000; // ~1 fps to the vision model
async function agentToken(room: string): Promise<string> {
const at = new AccessToken(env.LIVEKIT_API_KEY, env.LIVEKIT_API_SECRET, {
identity: 'podman-agent',
name: 'PodMan',
ttl: '4h',
});
at.addGrant({ roomJoin: true, room, canPublish: true, canSubscribe: true, canPublishData: true });
return at.toJwt();
}
async function main() {
const room = new Room();
const podman = new PodMan(room, POD_ROOM);
await room.connect(env.LIVEKIT_URL, await agentToken(POD_ROOM), {
autoSubscribe: true,
dynacast: true,
});
await podman.start();
console.log(`[agent] PodMan joined room ${POD_ROOM}`);
const lastSent = new Map<string, number>();
room.on(
RoomEvent.TrackSubscribed,
(track: RemoteTrack, pub: RemoteTrackPublication, participant: RemoteParticipant) => {
if (track.kind !== TrackKind.KIND_VIDEO || pub.source !== TrackSource.SOURCE_SCREENSHARE) return;
const id = participant.identity;
const stream = new VideoStream(track);
void (async () => {
for await (const event of stream) {
const now = Date.now();
if (now - (lastSent.get(id) ?? 0) < SAMPLE_INTERVAL_MS) continue; // THROTTLE
lastSent.set(id, now);
const rgba = event.frame.convert(VideoBufferType.RGBA);
const jpeg = await sharp(Buffer.from(rgba.data), {
raw: { width: rgba.width, height: rgba.height, channels: 4 },
})
.resize({ width: 1280, withoutEnlargement: true })
.jpeg({ quality: 70 })
.toBuffer();
await podman.onScreenFrame(id, jpeg);
}
})();
},
);
const shutdown = async () => {
await room.disconnect();
await dispose();
process.exit(0);
};
process.on('SIGINT', shutdown);
process.on('SIGTERM', shutdown);
}
main().catch((e) => {
console.error('[agent] fatal', e);
process.exit(1);
});
docs/generated/files/backend/src/agent/podman.ts
-75
View File
@@ -1,75 +0,0 @@
import { RoomEvent, type Room } from '@livekit/rtc-node';
import type { EngineerContext, Collision, Intervention, DataMessage } from '@podman/shared';
import { DATA_TOPIC } from '@podman/shared';
import { analyzeFrame } from '../vision/gemini.js';
import { detectCollisions } from '../collision/detector.js';
import { getGithubState } from '../github/client.js';
import { recordObservation, recordCollision, recordIntervention } from '../memory/store.js';
import { recallSimilar } from '../memory/vectors.js';
import { shouldIntervene, preferredAction } from '../memory/policy.js';
import { speak } from '../voice/live.js';
export class PodMan {
private contexts = new Map<string, EngineerContext>();
private encoder = new TextEncoder();
constructor(
private room: Room,
private podId: string,
) {}
async start(): Promise<void> {
// Tier-2 optional ground-truth + engineer ACKs arrive over the data channel.
this.room.on(RoomEvent.DataReceived, (payload) => {
try {
const msg = JSON.parse(new TextDecoder().decode(payload)) as DataMessage;
if (msg.type === 'GIT_REPORT') {
const c = this.contexts.get(msg.report.engineerId);
if (c) c.hasUnpushedChanges = msg.report.unpushedCount > 0 || msg.report.dirtyFiles.length > 0;
}
} catch { /* ignore malformed */ }
});
}
async onScreenFrame(engineerId: string, jpeg: Buffer): Promise<void> {
const ctx = await analyzeFrame(engineerId, this.podId, jpeg);
this.contexts.set(engineerId, ctx);
await recordObservation(ctx);
const github = await getGithubState(); // cached
const collisions = detectCollisions([...this.contexts.values()], github);
for (const collision of collisions) await this.handle(collision);
}
private async handle(collision: Collision): Promise<void> {
const prior = await recallSimilar(collision); // Loop A: vector recall raises confidence
if (prior) collision.severity = 'critical';
if (!shouldIntervene(collision, prior)) return; // Loop B: policy gate
await recordCollision(collision);
const action = preferredAction(collision, prior);
const names = collision.engineers.join(' and ');
const message = `${names} are both editing ${collision.file}` +
(collision.githubState?.unpushed ? ' and one has unpushed changes.' : '.') +
(prior ? ` I've seen this conflict pattern before.` : '');
const intervention: Intervention = {
id: `int_${Date.now()}`,
collisionId: collision.id,
podId: this.podId,
kind: 'card',
message,
suggestedAction: action,
status: 'pending',
createdAt: new Date().toISOString(),
};
await recordIntervention(intervention);
const data: DataMessage = { type: 'COLLISION', collision, intervention };
await this.room.localParticipant?.publishData(
this.encoder.encode(JSON.stringify(data)),
{ reliable: true, topic: DATA_TOPIC },
);
await speak(this.room, message); // gemini-3.1-flash-live voice into the room
}
}
docs/generated/files/backend/src/collision/detector.ts
-40
View File
@@ -1,40 +0,0 @@
import type { EngineerContext, Collision, GithubStateSnapshot } from '@podman/shared';
function normalize(path?: string): string | undefined {
if (!path) return undefined;
return path.replace(/^\.?\/?(src\/)?/, 'src/').toLowerCase();
}
export function detectCollisions(
contexts: EngineerContext[],
github: GithubStateSnapshot,
): Collision[] {
const byFile = new Map<string, EngineerContext[]>();
for (const c of contexts) {
const f = normalize(c.currentFile);
if (!f) continue;
(byFile.get(f) ?? byFile.set(f, []).get(f)!).push(c);
}
const out: Collision[] = [];
for (const [file, group] of byFile) {
const engineers = [...new Set(group.map((g) => g.engineerId))];
if (engineers.length < 2) continue;
const anyUnpushed =
group.some((g) => g.hasUnpushedChanges) || github.unpushed === true;
if (!anyUnpushed) continue; // the crux GitHub alone cannot answer
out.push({
id: `col_${file}_${Date.now()}`,
podId: group[0]!.podId,
file,
symbol: group.find((g) => g.currentSymbol)?.currentSymbol,
engineers,
severity: 'warn',
githubState: { ...github, unpushed: anyUnpushed },
detectedAt: new Date().toISOString(),
});
}
return out;
}
docs/generated/files/backend/src/env.ts
-35
View File
@@ -1,35 +0,0 @@
import 'dotenv/config';
function req(name: string): string {
const v = process.env[name];
if (!v) throw new Error(`Missing required env var: ${name}`);
return v;
}
function opt(name: string, fallback = ''): string {
return process.env[name] ?? fallback;
}
export const env = {
// LiveKit
LIVEKIT_URL: req('LIVEKIT_URL'),
LIVEKIT_API_KEY: req('LIVEKIT_API_KEY'),
LIVEKIT_API_SECRET: req('LIVEKIT_API_SECRET'),
// Gemini
GEMINI_API_KEY: req('GEMINI_API_KEY'),
GEMINI_VISION_MODEL: opt('GEMINI_VISION_MODEL', 'gemini-3.5-flash'),
GEMINI_LIVE_MODEL: opt('GEMINI_LIVE_MODEL', 'gemini-3.1-flash-live-preview'),
// GitHub
GITHUB_TOKEN: req('GITHUB_TOKEN'),
GITHUB_REPO: req('GITHUB_REPO'), // owner/name
// Mongo + Voyage
MONGODB_URI: req('MONGODB_URI'),
VOYAGE_API_KEY: opt('VOYAGE_API_KEY'),
// Server
PORT: Number(opt('PORT', '8787')),
} as const;
export function repoParts(): { owner: string; repo: string } {
const [owner, repo] = env.GITHUB_REPO.split('/');
if (!owner || !repo) throw new Error('GITHUB_REPO must be "owner/name"');
return { owner, repo };
}
docs/generated/files/backend/src/github/client.ts
-46
View File
@@ -1,46 +0,0 @@
import { Octokit } from 'octokit';
import type { GithubStateSnapshot } from '@podman/shared';
import { env, repoParts } from '../env.js';
const gh = new Octokit({ auth: env.GITHUB_TOKEN });
let cache: { at: number; state: GithubStateSnapshot } | null = null;
const TTL_MS = 5000;
export async function getGithubState(): Promise<GithubStateSnapshot> {
if (cache && Date.now() - cache.at < TTL_MS) return cache.state;
const { owner, repo } = repoParts();
const [{ data: branches }] = await Promise.all([
gh.rest.repos.listBranches({ owner, repo, per_page: 50 }),
]);
const state: GithubStateSnapshot = {
branches: Object.fromEntries(branches.map((b) => [b.name, b.commit.sha])),
openPrs: [],
unpushed: undefined, // vision/Tier-2 fills this; API cannot know
};
cache = { at: Date.now(), state };
return state;
}
export async function remoteHasFile(path: string, ref = 'main'): Promise<boolean> {
const { owner, repo } = repoParts();
return gh.rest.repos
.getContent({ owner, repo, path, ref })
.then(() => true)
.catch(() => false);
}
export async function createSyncPr(input: { headBranch: string; file: string; summary: string }) {
const { owner, repo } = repoParts();
const { data: mainRef } = await gh.rest.git.getRef({ owner, repo, ref: 'heads/main' });
const branch = `podman-sync-${Date.now()}`;
await gh.rest.git.createRef({ owner, repo, ref: `refs/heads/${branch}`, sha: mainRef.object.sha });
const { data: pr } = await gh.rest.pulls.create({
owner,
repo,
title: `PodMan: sync ${input.file} before collision`,
head: branch,
base: 'main',
body: input.summary,
});
return pr;
}
docs/generated/files/backend/src/server.ts
-59
View File
@@ -1,59 +0,0 @@
import express from 'express';
import { createServer } from 'node:http';
import { WebSocketServer } from 'ws';
import { AccessToken } from 'livekit-server-sdk';
import { env } from './env.js';
import { createSyncPr } from './github/client.js';
import { recordOutcome } from './memory/store.js';
import type { InterventionOutcome } from '@podman/shared';
const app = express();
app.use(express.json());
app.get('/health', (_req, res) => res.json({ ok: true }));
// Mint a LiveKit token for an engineer joining a pod.
app.post('/api/token', async (req, res) => {
const { room, identity, name, githubLogin } = req.body ?? {};
if (!room || !identity) return res.status(400).json({ error: 'room+identity required' });
const at = new AccessToken(env.LIVEKIT_API_KEY, env.LIVEKIT_API_SECRET, {
identity,
name,
ttl: '4h',
metadata: JSON.stringify({ githubLogin: githubLogin ?? name }),
});
at.addGrant({ roomJoin: true, room, canPublish: true, canSubscribe: true, canPublishData: true });
res.json({ token: await at.toJwt(), url: env.LIVEKIT_URL });
});
// PodMan's hero action: open a real sync PR on the PUBLIC repo.
app.post('/api/sync-pr', async (req, res) => {
try {
const { headBranch, file, summary } = req.body ?? {};
const pr = await createSyncPr({ headBranch, file, summary });
res.json({ url: pr.html_url, number: pr.number });
} catch (e) {
res.status(500).json({ error: (e as Error).message });
}
});
// Outcome ACK -> closes the continual-learning policy loop.
app.post('/api/outcome', async (req, res) => {
await recordOutcome(req.body as InterventionOutcome);
res.json({ ok: true });
});
const http = createServer(app);
// ws relay: the agent pushes collision/intervention JSON here; PWAs subscribed by pod receive it.
const wss = new WebSocketServer({ server: http, path: '/api/events' });
const clients = new Set<import('ws').WebSocket>();
wss.on('connection', (ws) => {
clients.add(ws);
ws.on('close', () => clients.delete(ws));
ws.on('message', (buf) => {
// fan out agent->PWA events; (auth/pod-scoping omitted for hackathon brevity)
for (const c of clients) if (c !== ws && c.readyState === 1) c.send(buf.toString());
});
});
http.listen(env.PORT, '0.0.0.0', () => console.log(`[server] :${env.PORT}`));
docs/generated/files/backend/src/vision/gemini.ts
-53
View File
@@ -1,53 +0,0 @@
import { GoogleGenAI, Type } from '@google/genai';
import type { EngineerContext } from '@podman/shared';
import { env } from '../env.js';
const ai = new GoogleGenAI({ apiKey: env.GEMINI_API_KEY });
const SCHEMA = {
type: Type.OBJECT,
properties: {
currentFile: { type: Type.STRING, description: 'open file path if visible, e.g. src/auth/session.ts' },
currentSymbol: { type: Type.STRING, description: 'function/class under the cursor' },
activity: { type: Type.STRING, description: 'editing | reading | debugging | terminal | PR review' },
hasUnpushedChanges: { type: Type.BOOLEAN, description: 'dirty git gutter / modified markers visible' },
confidence: { type: Type.NUMBER, description: '0..1 confidence in this read' },
},
propertyOrdering: ['currentFile', 'currentSymbol', 'activity', 'hasUnpushedChanges', 'confidence'],
} as const;
export async function analyzeFrame(
engineerId: string,
podId: string,
jpeg: Buffer,
): Promise<EngineerContext> {
const res = await ai.models.generateContent({
model: env.GEMINI_VISION_MODEL,
contents: [
{
role: 'user',
parts: [
{ text: "You are PodMan watching an engineer's screen. Identify what file/symbol they are working on and whether there are uncommitted edits. JSON only." },
{ inlineData: { mimeType: 'image/jpeg', data: jpeg.toString('base64') } },
],
},
],
config: {
responseMimeType: 'application/json',
responseJsonSchema: SCHEMA,
thinkingConfig: { thinkingBudget: 0 }, // minimal thinking: low latency/cost for ambient loop
mediaResolution: 'MEDIA_RESOLUTION_LOW',
},
});
const parsed = JSON.parse(res.text ?? '{}') as Partial<EngineerContext>;
return {
engineerId,
podId,
currentFile: parsed.currentFile,
currentSymbol: parsed.currentSymbol,
activity: parsed.activity,
hasUnpushedChanges: parsed.hasUnpushedChanges,
confidence: parsed.confidence ?? 0.5,
observedAt: new Date().toISOString(),
};
}
docs/generated/files/database/init.ts
-41
View File
@@ -1,41 +0,0 @@
import { MongoClient } from 'mongodb';
const uri = process.env.MONGODB_URI!;
const DB = 'podman';
async function main() {
const client = new MongoClient(uri);
await client.connect();
const db = client.db(DB);
await db.collection('pods').createIndex({ id: 1 }, { unique: true });
// High-volume observations expire after 6h to keep the cluster light.
await db.collection('observations').createIndex({ observedAt: 1 }, { expireAfterSeconds: 21600 });
await db.collection('observations').createIndex({ podId: 1, engineerId: 1 });
await db.collection('collisions').createIndex({ podId: 1, detectedAt: -1 });
await db.collection('interventions').createIndex({ id: 1 }, { unique: true });
await db.collection('team_model').createIndex({ podId: 1 }, { unique: true });
await db.collection('policy').createIndex({ pattern: 1 }, { unique: true });
// Atlas Vector Search index for collision-pattern recall (Voyage voyage-3 = 1024 dims).
try {
await db.command({
createSearchIndexes: 'memory_vectors',
indexes: [
{
name: 'vector_index',
type: 'vectorSearch',
definition: {
fields: [{ type: 'vector', path: 'embedding', numDimensions: 1024, similarity: 'cosine' }],
},
},
],
});
} catch (e) {
console.warn('vector index (create in Atlas UI if this errors):', (e as Error).message);
}
console.log('PodMan DB initialized.');
await client.close();
}
main().catch((e) => { console.error(e); process.exit(1); });
docs/generated/files/frontend/src/livekit/useInterventions.ts
-39
View File
@@ -1,39 +0,0 @@
import { useEffect, useState, useCallback } from 'react';
import { RoomEvent, type Room } from 'livekit-client';
import type { DataMessage, Intervention, InterventionStatus } from '@podman/shared';
import { DATA_TOPIC } from '@podman/shared';
import { postOutcome } from '../lib/api';
export function useInterventions(room: Room | null) {
const [active, setActive] = useState<Intervention | null>(null);
useEffect(() => {
if (!room) return;
const onData = (payload: Uint8Array, _p: unknown, _k: unknown, topic?: string) => {
if (topic !== DATA_TOPIC) return;
const msg = JSON.parse(new TextDecoder().decode(payload)) as DataMessage;
if (msg.type === 'COLLISION') setActive(msg.intervention);
};
room.on(RoomEvent.DataReceived, onData);
return () => { room.off(RoomEvent.DataReceived, onData); };
}, [room]);
const respond = useCallback(
async (status: InterventionStatus, accepted: boolean) => {
if (!active) return;
await postOutcome({
interventionId: active.id,
collisionId: active.collisionId,
podId: active.podId,
wasRealCollision: true,
accepted,
recordedAt: new Date().toISOString(),
});
setActive(null);
return status;
},
[active],
);
return { active, respond };
}
docs/generated/files/frontend/src/livekit/useScreenPublish.ts
-38
View File
@@ -1,38 +0,0 @@
import { useCallback, useRef, useState } from 'react';
import { Room, Track, createLocalScreenTracks, VideoPresets } from 'livekit-client';
import { fetchToken } from '../lib/api';
export function useScreenPublish() {
const roomRef = useRef<Room | null>(null);
const [connected, setConnected] = useState(false);
const [sharing, setSharing] = useState(false);
const join = useCallback(async (pod: string, identity: string, name: string, githubLogin?: string) => {
const { token, url } = await fetchToken({ room: pod, identity, name, githubLogin });
const room = new Room({ adaptiveStream: true, dynacast: true });
await room.connect(url, token);
roomRef.current = room;
setConnected(true);
return room;
}, []);
const startSharing = useCallback(async () => {
const room = roomRef.current;
if (!room) throw new Error('join the pod first');
const tracks = await createLocalScreenTracks({
audio: true,
resolution: VideoPresets.h1080.resolution,
});
for (const t of tracks) {
await room.localParticipant.publishTrack(t.mediaStreamTrack, {
source:
t.kind === Track.Kind.Audio ? Track.Source.ScreenShareAudio : Track.Source.ScreenShare,
});
}
await room.localParticipant.setMicrophoneEnabled(true);
await room.localParticipant.setCameraEnabled(true);
setSharing(true);
}, []);
return { join, startSharing, connected, sharing, room: roomRef };
}
docs/generated/files/infra/.do/app.yaml
-60
View File
@@ -1,60 +0,0 @@
name: podman
region: nyc
static_sites:
- name: web
github:
repo: <org>/Podman
branch: main
deploy_on_push: true
source_dir: frontend
build_command: corepack enable && pnpm install --frozen-lockfile && pnpm --filter @podman/shared build && pnpm --filter @podman/frontend build
output_dir: dist
routes:
- path: /
services:
- name: api
github:
repo: <org>/Podman
branch: main
deploy_on_push: true
source_dir: backend
build_command: corepack enable && pnpm install --frozen-lockfile && pnpm --filter @podman/shared build && pnpm --filter @podman/backend build
run_command: node dist/server.js
http_port: 8787
instance_size_slug: apps-s-1vcpu-1gb
instance_count: 1
routes:
- path: /api
envs:
- { key: LIVEKIT_URL, scope: RUN_TIME, type: SECRET }
- { key: LIVEKIT_API_KEY, scope: RUN_TIME, type: SECRET }
- { key: LIVEKIT_API_SECRET, scope: RUN_TIME, type: SECRET }
- { key: GITHUB_TOKEN, scope: RUN_TIME, type: SECRET }
- { key: GITHUB_REPO, scope: RUN_TIME, value: <org>/<public-repo> }
- { key: MONGODB_URI, scope: RUN_TIME, type: SECRET }
workers:
- name: podman-agent
github:
repo: <org>/Podman
branch: main
deploy_on_push: true
source_dir: backend
build_command: corepack enable && pnpm install --frozen-lockfile && pnpm --filter @podman/shared build && pnpm --filter @podman/backend build
run_command: node dist/agent.js
instance_size_slug: apps-s-1vcpu-1gb
instance_count: 1
envs:
- { key: LIVEKIT_URL, scope: RUN_TIME, type: SECRET }
- { key: LIVEKIT_API_KEY, scope: RUN_TIME, type: SECRET }
- { key: LIVEKIT_API_SECRET, scope: RUN_TIME, type: SECRET }
- { key: GEMINI_API_KEY, scope: RUN_TIME, type: SECRET }
- { key: GEMINI_VISION_MODEL, scope: RUN_TIME, value: gemini-3.5-flash }
- { key: GEMINI_LIVE_MODEL, scope: RUN_TIME, value: gemini-3.1-flash-live-preview }
- { key: GITHUB_TOKEN, scope: RUN_TIME, type: SECRET }
- { key: GITHUB_REPO, scope: RUN_TIME, value: <org>/<public-repo> }
- { key: MONGODB_URI, scope: RUN_TIME, type: SECRET }
- { key: VOYAGE_API_KEY, scope: RUN_TIME, type: SECRET }
- { key: POD_ROOM, scope: RUN_TIME, value: demo-pod }
docs/generated/files/shared/src/messages.ts
-45
View File
@@ -1,45 +0,0 @@
import type { Collision } from './collision.js';
import type { Intervention, InterventionStatus } from './intervention.js';
/** Topics multiplexed over the LiveKit data channel. */
export const DATA_TOPIC = 'podman.intervention' as const;
/** Wire messages exchanged between the PodMan agent and engineer PWAs. */
export type DataMessage =
| { type: 'COLLISION'; collision: Collision; intervention: Intervention }
| { type: 'VOICE_CUE'; text: string }
| { type: 'ACK'; interventionId: string; status: InterventionStatus; note?: string }
| { type: 'GIT_REPORT'; report: LocalGitReport };
/** Outcome of an intervention — the supervision signal for policy learning. */
export interface InterventionOutcome {
interventionId: string;
collisionId: string;
podId: string;
/** Did the predicted collision turn out real? (engineer-confirmed or inferred). */
wasRealCollision: boolean;
/** Did the engineer accept the offered action (e.g. sync PR)? */
accepted: boolean;
recordedAt: string;
}
/** The continually-refined per-pod world model (Loop A). */
export interface TeamModel {
podId: string;
/** filePath/dir -> engineerId most associated with it (de-facto owner). */
ownership: Record<string, string>;
/** Pairs of files that historically collide, with a co-occurrence weight. */
hotspots: Array<{ files: [string, string]; weight: number }>;
updatedAt: string;
}
/** OPTIONAL Tier-2 ground-truth from a per-laptop git sidecar. */
export interface LocalGitReport {
engineerId: string;
branch: string;
/** Commits ahead of upstream (invisible to the GitHub API). */
unpushedCount: number;
/** Working-tree files with uncommitted edits. */
dirtyFiles: string[];
reportedAt: string;
}