# PodMan **An ambient pair programmer for engineering teams.** PodMan watches the work happening inside a shared LiveKit room, understands what each engineer is doing, remembers which interventions helped, and nudges the team before duplicated work, merge collisions, or missed handoffs slow everyone down. [LiveKit](https://livekit.io/) · [MongoDB](https://www.mongodb.com/) · [Gemini](https://ai.google.dev/) · [Hermes](https://hermes-agent.nousresearch.com/) · [vLLM](https://vllm.ai/) · [DigitalOcean](https://www.digitalocean.com/) ## Read This First | Need | Use this | | ----------------------- | --------------------------------------------------------------------------------------------- | | Open the product | `https://podman.live` | | Check the app | `curl https://podman.live/health` | | Check pods and memory | `curl https://podman.live/api/pods && curl https://podman.live/api/memory/stats` | | Use the LLM externally | Base URL `https://llm.alhinai.dev/v1`, model `gemma-4-31B-it` | | Test Hermes | `hermes -z 'Reply with exactly: working' --provider gemma4-31b-vllm --model gemma-4-31B-it` | | Start local development | API, vision agent, and frontend commands are in [Local Development](#local-development) | | Debug production | Public checks first, then systemd services in [Production Operations](#production-operations) | --- ## Current Live System | Surface | Running now | Purpose | | ------------- | ---------------------------- | -------------------------------------- | | Product | `https://podman.live` | Team room, screen context, cards | | API | `https://podman.live/api/*` | Pods, tokens, outcomes, memory | | Health | `https://podman.live/health` | Backend readiness | | Local API | `127.0.0.1:8787` | Express service behind Caddy | | Reasoning LLM | `https://llm.alhinai.dev/v1` | OpenAI-compatible Gemma/vLLM endpoint | | API key | `not-needed` | Placeholder key for OpenAI clients | | Hermes model | `gemma-4-31B-it` | 262K-context tool-using agent | | Hermes config | `gemma4-31b-vllm` | Custom provider used by Hermes locally | ```mermaid flowchart TB Browser["Engineer Browser
React + Vite PWA"] Room["LiveKit Room
screen share + audio + data messages"] subgraph Droplet["DigitalOcean PodMan Droplet"] Caddy["Caddy
static app + /api proxy"] API["Express API
127.0.0.1:8787"] Vision["Vision Agent
screen frame observer"] Voice["Live Conversation Agent
Python + Gemini Live"] Ops["Hermes Ops Timers
watchdog + sync deploy"] end subgraph Memory["MongoDB Atlas"] Observations["observations"] State["engineer_states"] Outcomes["interventions + outcomes"] end subgraph Google["Google Gemini APIs"] GeminiVision["Vision"] GeminiVoice["Live voice + TTS"] GeminiEmbed["Embeddings fallback"] Lyria["Music"] end subgraph Reasoning["External Reasoning Endpoint"] Tunnel["Cloudflare Tunnel
llm.alhinai.dev"] VLLM["vLLM OpenAI Server
gemma-4-31B-it
262144 context"] Hermes["Hermes Agent
provider: gemma4-31b-vllm"] end Browser --> Caddy --> API Browser <-->|screen, audio, cards| Room API --> Room API --> Observations API --> State API --> Outcomes Vision <-->|screen tracks| Room Vision --> GeminiVision Vision --> Observations Vision --> Outcomes Voice <-->|conversation| Room Voice --> GeminiVoice Vision --> GeminiEmbed Voice --> GeminiEmbed Ops --> Hermes --> Tunnel --> VLLM API --> Hermes Lyria --> Voice classDef user fill:#e8f1ff,stroke:#3366cc,color:#0b1f44; classDef app fill:#eef8ee,stroke:#2f8a3a,color:#123915; classDef data fill:#fff6df,stroke:#c47f00,color:#3d2b00; classDef ai fill:#f4edff,stroke:#805ad5,color:#2d1857; class Browser,Room user; class Caddy,API,Vision,Voice,Ops app; class Observations,State,Outcomes data; class GeminiVision,GeminiVoice,GeminiEmbed,Lyria,Tunnel,VLLM,Hermes ai; ``` ## What It Does ```mermaid flowchart LR A["Engineer shares screen"] --> B["Gemini extracts work context"] B --> C["PodMan detects overlap
files, symbols, research, unpushed work"] C --> D["MongoDB recalls
similar prior events"] D --> E{"Policy gate"} E -->|"seen false alarm"| F["stay quiet"] E -->|"seen real collision"| G["raise urgency"] E -->|"new useful signal"| H["show card or message"] G --> I["Hermes / voice escalation"] H --> J["teammate accepts or dismisses"] I --> J J --> K["outcome becomes future memory"] K --> D ``` PodMan removes the reason to interrupt. It gives the team a live picture of work in progress, then improves from every accepted or dismissed intervention. > GitHub sees pushed work. PodMan sees work while it is still happening. --- ## The Model Stack PodMan uses multiple AI surfaces. They are intentionally split by job. ```mermaid flowchart LR Work["Screen + room activity"] --> VisionRoute["Perception route"] VoiceInput["Engineer speech"] --> ConversationRoute["Conversation route"] OpsNeed["Ops / autonomous task"] --> HermesRoute["Reasoning route"] MemoryNeed["Similarity search"] --> EmbedRoute["Memory route"] Ambient["Session atmosphere"] --> MusicRoute["Audio route"] VisionRoute --> Gemini20["gemini-2.0-flash
screen understanding"] ConversationRoute --> GeminiLive["gemini-3.1-flash-live-preview
live Q&A"] ConversationRoute --> GeminiTTS["gemini-3.1-flash-tts-preview
urgent speech"] HermesRoute --> Gemma["gemma-4-31B-it
Hermes + vLLM + tools"] EmbedRoute --> Voyage["voyage-4-lite
primary embeddings"] EmbedRoute --> GeminiEmbed["gemini-embedding-001
fallback embeddings"] MusicRoute --> Lyria["lyria-3-clip-preview
background music"] classDef signal fill:#e8f1ff,stroke:#3366cc,color:#0b1f44; classDef route fill:#eef8ee,stroke:#2f8a3a,color:#123915; classDef model fill:#f4edff,stroke:#805ad5,color:#2d1857; class Work,VoiceInput,OpsNeed,MemoryNeed,Ambient signal; class VisionRoute,ConversationRoute,HermesRoute,EmbedRoute,MusicRoute route; class Gemini20,GeminiLive,GeminiTTS,Gemma,Voyage,GeminiEmbed,Lyria model; ``` ### Gemma 4 31B via vLLM Hermes uses the external OpenAI-compatible endpoint: ```text Base URL: https://llm.alhinai.dev/v1 API key: not-needed Model: gemma-4-31B-it Context: 262144 tokens ``` The vLLM server is configured for Hermes-style tool use: ```text --max-model-len 262144 --max-num-seqs 1 --max-num-batched-tokens 16384 --gpu-memory-utilization 0.85 --kv-cache-dtype fp8 --enable-auto-tool-choice --tool-call-parser gemma4 --enable-chunked-prefill ``` Hermes should point at that endpoint with this provider shape: ```yaml model: default: gemma-4-31B-it provider: gemma4-31b-vllm providers: gemma4-31b-vllm: name: Gemma 4 31B vLLM (256K) api: https://llm.alhinai.dev/v1 api_key: not-needed transport: chat_completions default_model: gemma-4-31B-it discover_models: true models: gemma-4-31B-it: context_length: 262144 agent: tool_use_enforcement: auto ``` Why this matters: Hermes sends OpenAI tool schemas and `tool_choice: "auto"`. Without `--enable-auto-tool-choice` and `--tool-call-parser gemma4`, vLLM returns HTTP 400 before Hermes can initialize an agent. ### Gemini Surfaces Gemini remains the realtime perception and voice layer inside PodMan. | Use | Model | Code | | -------------------------- | ------------------------------- | ------------------------------------------ | | Screen understanding | `gemini-2.0-flash` | `backend/src/vision/gemini.ts` | | Urgent spoken alerts | `gemini-3.1-flash-tts-preview` | `backend/src/voice/live.ts` | | Live room conversation | `gemini-3.1-flash-live-preview` | `agents/podman-live-conversation/agent.py` | | Memory embeddings fallback | `gemini-embedding-001` | `backend/src/memory/vectors.ts` | | Ambient background music | `lyria-3-clip-preview` | `backend/src/voice/music.ts` | Voyage embeddings can be used first when `VOYAGE_API_KEY` is set. Gemini embeddings remain the fallback. If no embedding provider is available, PodMan falls back to exact-signature matching. --- ## How The Learning Loop Works The learning loop is the product. A teammate only has to accept or dismiss an intervention; the rest is captured automatically. ```text observe -> detect -> recall prior outcomes -> policy gate -> act -> record outcome ^ | +-------------------------- next recall -----------------------------+ ``` | Stage | What happens | Code | | ------- | --------------------------------------------------------------------------- | ----------------------------------- | | Observe | Gemini Vision turns sampled screen frames into structured work context. | `backend/src/vision/gemini.ts` | | Detect | PodMan detects overlapping files, symbols, research, and unpushed work. | `backend/src/collision/detector.ts` | | Recall | MongoDB Atlas recalls similar prior events and outcomes. | `backend/src/memory/vectors.ts` | | Gate | Policy suppresses dismissed false alarms and escalates recurring real ones. | `backend/src/memory/policy.ts` | | Act | PodMan publishes a card, Hermes message, or urgent voice cue. | `backend/src/action/hermes.ts` | | Learn | Accept/dismiss feedback is written back to memory. | `backend/src/memory/store.ts` | --- ## Runtime Components | Layer | Runtime | Responsibility | | ------------------ | ----------------------------------- | -------------------------------------------------------------------- | | Frontend | React + Vite | Pod rooms, screen share, cards, voice controls, member state | | Backend API | Express on `:8787` | LiveKit tokens, pod CRUD, outcomes, memory stats, sync PRs | | Vision agent | Node + `@livekit/rtc-node` | Subscribes to screen tracks, samples frames, publishes interventions | | Live voice agent | Python LiveKit Agents + Gemini Live | Real-time voice Q&A in a pod room | | Memory | MongoDB Atlas | Observations, engineer state, collisions, interventions, outcomes | | Reasoning agent | Hermes + Gemma vLLM | Tool-using autonomous assistant and ops layer | | Realtime transport | LiveKit Cloud | Screen tracks, audio tracks, data messages | | Hosting | DigitalOcean + Caddy + systemd | Static frontend, API, workers, watchdog timers | --- ## Data Flow ```mermaid sequenceDiagram autonumber participant Dev as Engineer PWA participant API as Backend API participant LK as LiveKit Room participant Agent as Vision Agent participant Gemini as Gemini APIs participant Mongo as MongoDB Atlas participant Hermes as Hermes/Gemma Dev->>API: POST /api/token API-->>Dev: LiveKit URL + JWT Dev->>LK: Join pod room + publish screen share Agent->>LK: Subscribe to screen-share video Agent->>Gemini: Sampled frame Gemini-->>Agent: Structured work context Agent->>Mongo: Store observation Agent->>Mongo: Recall similar prior events Mongo-->>Agent: Prior outcome + policy hints Agent->>Hermes: Escalate when autonomous help is useful Agent->>LK: Publish card / message / voice cue Dev->>API: POST /api/outcome API->>Mongo: Store accept/dismiss feedback ``` --- ## Public Interfaces | Interface | Purpose | | ----------------------------------------------------------- | ---------------------------------------------- | | `GET /health` | API health check | | `POST /api/token` | Mint LiveKit room tokens | | `GET /api/pods` | List pods | | `GET/POST/PATCH/DELETE /api/pods` | Pod CRUD | | `POST/DELETE /api/pods/:id/members` | Pod membership | | `GET /api/pods/:id/members/:name/history` | Recent member work history | | `POST /api/outcome` | Store accepted/dismissed intervention outcomes | | `GET /api/memory/stats` | Live memory collection counts | | `POST /api/sync-pr` | Create a visible sync PR artifact | | LiveKit topic `podman.intervention` | Intervention data channel | | Wire messages `COLLISION`, `ACK`, `GIT_REPORT`, `VOICE_CUE` | Agent/PWA contract | --- ## Monorepo Layout | Folder | Purpose | | ----------- | ----------------------------------------------------------------------- | | `frontend/` | React + Vite PWA | | `backend/` | Express API, vision agent, memory, collision detection, Hermes job APIs | | `agents/` | Python LiveKit conversation agent | | `shared/` | Shared TypeScript contracts | | `database/` | MongoDB setup and seed utilities | | `infra/` | Caddy, Docker, DigitalOcean, systemd units | | `scripts/` | Git watcher, deploy doctor, watchdog, verification tooling | | `docs/` | Demo, deployment, learning, graph, and architecture notes | --- ## Local Development Run the core app in three terminals: ```mermaid flowchart LR Env["1. Configure .env"] --> Install["2. pnpm install"] Install --> API["Terminal A
backend API :8787"] Install --> Agent["Terminal B
vision agent"] Install --> UI["Terminal C
frontend :5173"] Install --> Voice["Optional
conversation agent"] API --> Browser["Open local PWA"] Agent --> Browser UI --> Browser Voice --> Browser classDef step fill:#eef8ee,stroke:#2f8a3a,color:#123915; classDef run fill:#e8f1ff,stroke:#3366cc,color:#0b1f44; class Env,Install step; class API,Agent,UI,Voice,Browser run; ``` ```bash cp .env.example .env # Fill LIVEKIT_*, GEMINI_*, GITHUB_*, MONGODB_URI. pnpm install pnpm --filter @podman/backend dev # API on :8787 pnpm --filter @podman/backend dev:agent # LiveKit vision agent pnpm --filter @podman/frontend dev # PWA on :5173 ``` Run the Python live conversation agent: ```bash pnpm livekit:conversation:agent ``` Run the local git watcher on each demo laptop: ```bash node scripts/podman-agent.mjs --name --pod ``` Demo identities: ```bash node scripts/podman-agent.mjs --name alice --pod demo-pod node scripts/podman-agent.mjs --name bob --pod demo-pod node scripts/podman-agent.mjs --name carol --pod demo-pod ``` --- ## Production Operations The production droplet is systemd-supervised. Caddy serves the built frontend and proxies `/api/*` to the backend on `127.0.0.1:8787`. ```mermaid flowchart LR Public["https://podman.live"] --> Caddy["caddy.service"] Caddy --> Static["/var/www/podman"] Caddy --> API["podman-platform-api.service
:8787"] API --> Agent["podman-platform-agent.service"] API --> Voice["podman-live-conversation-agent.service"] Watchdog["podman-hermes-watchdog.timer"] --> Public Sync["podman-hermes-sync-deploy.timer"] --> API classDef public fill:#e8f1ff,stroke:#3366cc,color:#0b1f44; classDef service fill:#eef8ee,stroke:#2f8a3a,color:#123915; classDef timer fill:#fff6df,stroke:#c47f00,color:#3d2b00; class Public public; class Caddy,Static,API,Agent,Voice service; class Watchdog,Sync timer; ``` | Service / timer | Purpose | | ---------------------------------------- | -------------------------------------------------------------- | | `podman-platform-api.service` | Built backend API on port `8787` | | `podman-platform-agent.service` | Node LiveKit vision agent | | `podman-live-conversation-agent.service` | Python Gemini Live conversation agent | | `podman-hermes-watchdog.timer` | Periodic public health and remediation | | `podman-hermes-sync-deploy.timer` | Clean-tree fast-forward deploy loop | | `caddy.service` | Serves `/var/www/podman`, proxies `/api/*` to `127.0.0.1:8787` | Check the app from the outside first: ```bash curl https://podman.live/ curl https://podman.live/health curl https://podman.live/api/pods curl https://podman.live/api/presence curl https://podman.live/api/memory/stats ``` Then check the droplet services: ```bash systemctl is-active podman-platform-api podman-platform-agent systemctl is-active podman-live-conversation-agent systemctl is-active podman-hermes-watchdog.timer podman-hermes-sync-deploy.timer ``` Hermes operations scripts: ```bash pnpm hermes:watchdog pnpm hermes:watchdog:strict pnpm hermes:sync-deploy pnpm deploy:doctor:strict ``` Gemma/vLLM endpoint checks: ```bash curl https://llm.alhinai.dev/v1/models \ -H "Authorization: Bearer not-needed" curl https://llm.alhinai.dev/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer not-needed" \ -d '{ "model": "gemma-4-31B-it", "messages": [{"role": "user", "content": "Reply with exactly: working"}], "temperature": 0, "max_tokens": 512 }' ``` --- ## Required Environment ```bash LIVEKIT_URL=wss://your-livekit-server.livekit.cloud LIVEKIT_API_KEY=... LIVEKIT_API_SECRET=... LIVEKIT_CONVERSATION_AGENT_NAME=podman-live-conversation GEMINI_API_KEY=... GEMINI_VISION_MODEL=gemini-2.0-flash GEMINI_LIVE_MODEL=gemini-3.1-flash-tts-preview GEMINI_CONVERSATION_MODEL=gemini-3.1-flash-live-preview GEMINI_EMBEDDING_MODEL=gemini-embedding-001 GEMINI_TTS_VOICE=Charon GITHUB_TOKEN=... GITHUB_REPO=karti-ai/podman MONGODB_URI=mongodb+srv://... VOYAGE_API_KEY=... VOYAGE_EMBEDDING_MODEL=voyage-4-lite PORT=8787 POD_ROOM=demo-pod ``` Gemma/Hermes provider values live in Hermes config, not PodMan `.env`: ```text provider: gemma4-31b-vllm model: gemma-4-31B-it base_url: https://llm.alhinai.dev/v1 api_key: not-needed ``` --- ## Verification Before calling a deployment healthy: ```bash pnpm verify pnpm verify:infra pnpm deploy:doctor:strict pnpm hermes:watchdog:strict ``` For the public site: ```bash curl https://podman.live/ curl https://podman.live/health curl https://podman.live/api/pods curl https://podman.live/api/presence curl https://podman.live/api/memory/stats ``` For Hermes/Gemma: ```bash hermes -z 'Reply with exactly: working' \ --provider gemma4-31b-vllm \ --model gemma-4-31B-it ``` Expected output: ```text working ``` --- ## Troubleshooting | Symptom | Likely cause | Check | | ---------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------ | | Frontend loads but API fails | Backend or Caddy proxy issue | `systemctl status podman-platform-api caddy` | | `/api/*` returns 502 | API not listening on `8787` | `ss -ltnp`, `curl http://127.0.0.1:8787/health` | | No screen observations | Vision agent not in LiveKit room | `journalctl -u podman-platform-agent -n 80` | | Live voice missing | Python conversation agent down | `journalctl -u podman-live-conversation-agent -n 80` | | Memory empty | MongoDB unavailable or env missing | `curl /api/memory/stats`, backend logs | | Hermes says tool auto-choice is disabled | vLLM missing tool flags | verify `--enable-auto-tool-choice --tool-call-parser gemma4` | | `llm.alhinai.dev` returns 502 | Gemma vLLM still loading or tunnel target down | `curl /v1/models`, vLLM logs on Gemma host | --- ## Why It Gets Better PodMan is not a static alert system. It remembers what actually helped. - A dismissed false alarm lowers future urgency. - An accepted real collision raises future urgency for similar work. - Exact-signature recall catches repeats even without vector search. - MongoDB outcomes become the policy signal for the next session. - Hermes and Gemma give the system a tool-using agent when the coordination problem needs active investigation instead of a passive card. The goal is simple: fewer interruptions, fewer duplicate branches, and a team that can move fast without constantly asking what everyone else is doing.