Kartikeya
da5b2622b2
Promote all 12 staged canonical files from docs/generated/files/ to their
live paths, creating the full PodMan architecture:
Backend:
- server.ts: HTTP service (token mint, sync-PR, outcome recording, /health, WS relay)
- agent.ts: worker joining LiveKit room, grabbing screenshare frames at ~1fps
- agent/podman.ts: orchestrator loop (vision -> collision detection -> intervention)
- env.ts: flat env var accessors replacing nested stub
- vision/gemini.ts: JPEG -> Gemini vision -> EngineerContext (real implementation)
- collision/detector.ts: fused vision+GitHub collision detection (the moat)
- github/client.ts: Octokit wrapper with caching + sync PR creation
- memory/store.ts: extended with recordObservation/recordCollision/recordIntervention/recordOutcome helpers
- memory/vectors.ts: stub for Voyage+Atlas vector recall (Loop A)
- memory/policy.ts: stub for intervention policy gate (Loop B)
- voice/live.ts: stub for Gemini Live TTS voice output
Shared:
- messages.ts: LiveKit data-channel wire protocol (DataMessage, InterventionOutcome, TeamModel, LocalGitReport)
- index.ts: re-exports messages module
Frontend:
- livekit/useScreenPublish.ts: hook for joining pod and publishing screenshare
- livekit/useInterventions.ts: hook for receiving collision cards and responding
- lib/api.ts: fetchToken + postOutcome HTTP helpers
Database:
- database/init.ts: MongoDB Atlas collections + indexes + vector search index
Infra:
- infra/.do/app.yaml: DO App Platform spec (static_site + service + worker)
Retire stubs superseded by canonical decomposition:
- backend/src/index.ts (replaced by server.ts)
- backend/src/intervention/engine.ts (logic now in agent/podman.ts)
- backend/src/livekit/token.ts (token minting now in server.ts)
Install missing dependencies: @livekit/rtc-node, sharp, mongodb, ws, @types/ws
Type error fixes:
- vision/gemini.ts: use MediaResolution.MEDIA_RESOLUTION_LOW enum value (not string literal)
- agent/podman.ts: wrap SuggestedActionKind into { kind: action } SuggestedAction object
All packages pass pnpm -r typecheck and pnpm -r build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
🛰️ PodMan — Jarvis for engineering teams
Ambient AI teammate for the 2026 AI Engineer World's Fair Hackathon (track: Continual Learning).
Engineers join a pod and share their screen + mic. PodMan watches every screen in
realtime, understands what each person is working on, fuses that with the team's
GitHub state, and proactively prevents collisions before code is even pushed —
speaking up like Jarvis: "Karti and Yahya are both editing auth.ts, Yahya has unpushed
changes — here's the diff, want a sync PR?"
The moat: unpushed local code is invisible to the GitHub API. The realtime screen-vision layer is the only way to catch a collision before the push. That is the whole product — and the reason this is an ambient agent, not a dashboard.
Monorepo layout
| Folder | What |
|---|---|
frontend/ |
React + Vite PWA — join pod, screen/mic/cam capture, PodMan voice + intervention cards |
backend/ |
PodMan agent: LiveKit room subscriber, Gemini vision, GitHub fusion, collision detector, voice out |
infra/ |
DigitalOcean deploy (App Platform / Droplet), IaC, Dockerfiles |
database/ |
MongoDB Atlas schema + Voyage vector memory for continual learning |
shared/ |
Shared TypeScript types (pod, engineer context, intervention) |
docs/ |
PLAN.md (the north star), demo script |
Prizes we're stacking
- 🏆 Best Gemini 3.5 — $5,000 cash (realtime vision + Live API voice)
- 🏆 Best LiveKit — Keychron keyboards (realtime transport is the core)
- 🏆 Best DigitalOcean — credits (deploy target)
👉 Read docs/PLAN.md first.