OSS/podman
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
bc5040658732175a3eb4a6322f7492066d1d45dd
podman/README.md
T
Yahya Alhinai 88abd8efb7 docs: update README with Modular MAX architecture
2026-06-28 18:10:27 +00:00

673 lines
25 KiB
Markdown
Raw Blame History

# PodMan
**An ambient pair programmer for engineering teams.**
PodMan watches the work happening inside a shared LiveKit room, understands what
each engineer is doing, remembers which interventions helped, and nudges the
team before duplicated work, merge collisions, or missed handoffs slow everyone
down.
[LiveKit](https://livekit.io/) · [MongoDB](https://www.mongodb.com/) ·
[Gemini](https://ai.google.dev/) · [Hermes](https://hermes-agent.nousresearch.com/) ·
[Modular MAX](https://www.modular.com/max) ·
[DigitalOcean](https://www.digitalocean.com/)
<p align="center">
<img
src="docs/assets/interruption-flow-recovery.jpg"
alt="A chart showing that a five minute interruption can create a much longer recovery period before a developer returns to flow state."
width="760"
/>
</p>
> PodMan exists to prevent the five-minute interruption from becoming a
> half-hour recovery tax.
Source image: `~/pic.jpg`, committed as
`docs/assets/interruption-flow-recovery.jpg` so it renders on GitHub.
## Read This First
| Need | Use this |
| ----------------------- | --------------------------------------------------------------------------------------------- |
| Open the product | `https://podman.live` |
| Check the app | `curl https://podman.live/health` |
| Check pods and memory | `curl https://podman.live/api/pods && curl https://podman.live/api/memory/stats` |
| Use the LLM externally | Base URL `https://llm.alhinai.dev/v1`, model `gemma-4-31B-it` |
| Test Hermes | `hermes -z 'Reply with exactly: working' --provider gemma4-31b-max --model gemma-4-31B-it` |
| Gemma 4 bring-up | Modular/MAX lab at `/home/alhinai/modular-lab` |
| Start local development | API, vision agent, and frontend commands are in [Local Development](#local-development) |
| Debug production | Public checks first, then systemd services in [Production Operations](#production-operations) |
---
## Current Live System
| Surface | Running now | Purpose |
| ------------- | ---------------------------- | -------------------------------------- |
| Product | `https://podman.live` | Team room, screen context, cards |
| API | `https://podman.live/api/*` | Pods, tokens, outcomes, memory |
| Health | `https://podman.live/health` | Backend readiness |
| Local API | `127.0.0.1:8787` | Express service behind Caddy |
| Reasoning LLM | `https://llm.alhinai.dev/v1` | OpenAI-compatible Modular/MAX endpoint |
| Gemma MAX lab | `/home/alhinai/modular-lab` | Modular/MAX bring-up path for Gemma 4 |
| API key | `not-needed` | Placeholder key for OpenAI clients |
| Hermes model | `gemma-4-31B-it` | 262K-context tool-using agent |
| Hermes config | `gemma4-31b-max` | Custom provider used by Hermes locally |
```mermaid
flowchart TB
Browser["Engineer Browser<br/>React + Vite PWA"]
Room["LiveKit Room<br/>screen share + audio + data messages"]
subgraph Droplet["DigitalOcean PodMan Droplet"]
Caddy["Caddy<br/>static app + /api proxy"]
API["Express API<br/>127.0.0.1:8787"]
Vision["Vision Agent<br/>screen frame observer"]
Voice["Live Conversation Agent<br/>Python + Gemini Live"]
Ops["Hermes Ops Timers<br/>watchdog + sync deploy"]
end
subgraph Memory["MongoDB Atlas"]
Observations["observations"]
State["engineer_states"]
Outcomes["interventions + outcomes"]
end
subgraph Google["Google Gemini APIs"]
GeminiVision["Vision"]
GeminiVoice["Live voice + TTS"]
GeminiEmbed["Embeddings fallback"]
Lyria["Music"]
end
subgraph Reasoning["External Reasoning Endpoint"]
Tunnel["Cloudflare Tunnel<br/>llm.alhinai.dev"]
MaxLab["Modular MAX Lab<br/>/home/alhinai/modular-lab"]
MaxServe["MAX OpenAI Server<br/>gemma-4-31B-it<br/>262144 context"]
MaxTest["MAX test port<br/>127.0.0.1:8001"]
Hermes["Hermes Agent<br/>provider: gemma4-31b-max"]
end
Browser --> Caddy --> API
Browser <-->|screen, audio, cards| Room
API --> Room
API --> Observations
API --> State
API --> Outcomes
Vision <-->|screen tracks| Room
Vision --> GeminiVision
Vision --> Observations
Vision --> Outcomes
Voice <-->|conversation| Room
Voice --> GeminiVoice
Vision --> GeminiEmbed
Voice --> GeminiEmbed
Ops --> Hermes --> Tunnel --> MaxServe
MaxLab --> MaxTest
MaxTest -. "validated Gemma 4<br/>staging path" .-> MaxServe
Tunnel -. "public route<br/>llm.alhinai.dev/v1" .-> MaxServe
API --> Hermes
Lyria --> Voice
classDef user fill:#e8f1ff,stroke:#3366cc,color:#0b1f44;
classDef app fill:#eef8ee,stroke:#2f8a3a,color:#123915;
classDef data fill:#fff6df,stroke:#c47f00,color:#3d2b00;
classDef ai fill:#f4edff,stroke:#805ad5,color:#2d1857;
class Browser,Room user;
class Caddy,API,Vision,Voice,Ops app;
class Observations,State,Outcomes data;
class GeminiVision,GeminiVoice,GeminiEmbed,Lyria,Tunnel,MaxLab,MaxServe,MaxTest,Hermes ai;
```
## What It Does
```mermaid
flowchart LR
A["Engineer shares screen"] --> B["Gemini extracts work context"]
B --> C["PodMan detects overlap<br/>files, symbols, research, unpushed work"]
C --> D["MongoDB recalls<br/>similar prior events"]
D --> E{"Policy gate"}
E -->|"seen false alarm"| F["stay quiet"]
E -->|"seen real collision"| G["raise urgency"]
E -->|"new useful signal"| H["show card or message"]
G --> I["Hermes / voice escalation"]
H --> J["teammate accepts or dismisses"]
I --> J
J --> K["outcome becomes future memory"]
K --> D
```
PodMan removes the reason to interrupt. It gives the team a live picture of work
in progress, then improves from every accepted or dismissed intervention.
> GitHub sees pushed work. PodMan sees work while it is still happening.
---
## The Model Stack
PodMan uses multiple AI surfaces. They are intentionally split by job.
```mermaid
flowchart LR
Work["Screen + room activity"] --> VisionRoute["Perception route"]
VoiceInput["Engineer speech"] --> ConversationRoute["Conversation route"]
OpsNeed["Ops / autonomous task"] --> HermesRoute["Reasoning route"]
MemoryNeed["Similarity search"] --> EmbedRoute["Memory route"]
Ambient["Session atmosphere"] --> MusicRoute["Audio route"]
VisionRoute --> Gemini20["gemini-2.0-flash<br/>screen understanding"]
ConversationRoute --> GeminiLive["gemini-3.1-flash-live-preview<br/>live Q&A"]
ConversationRoute --> GeminiTTS["gemini-3.1-flash-tts-preview<br/>urgent speech"]
HermesRoute --> Gemma["gemma-4-31B-it<br/>public serving via Modular MAX"]
HermesRoute --> Max["Modular MAX lab<br/>GB10 Gemma 4 validation"]
Max -. "test on :8001<br/>promote to public endpoint" .-> Gemma
EmbedRoute --> Voyage["voyage-4-lite<br/>primary embeddings"]
EmbedRoute --> GeminiEmbed["gemini-embedding-001<br/>fallback embeddings"]
MusicRoute --> Lyria["lyria-3-clip-preview<br/>background music"]
classDef signal fill:#e8f1ff,stroke:#3366cc,color:#0b1f44;
classDef route fill:#eef8ee,stroke:#2f8a3a,color:#123915;
classDef model fill:#f4edff,stroke:#805ad5,color:#2d1857;
class Work,VoiceInput,OpsNeed,MemoryNeed,Ambient signal;
class VisionRoute,ConversationRoute,HermesRoute,EmbedRoute,MusicRoute route;
class Gemini20,GeminiLive,GeminiTTS,Gemma,Max,Voyage,GeminiEmbed,Lyria model;
```
### Gemma 4 31B via Modular MAX
Hermes uses the external OpenAI-compatible endpoint:
```text
Base URL: https://llm.alhinai.dev/v1
API key: not-needed
Model: gemma-4-31B-it
Context: 262144 tokens
```
The Modular/MAX server is configured for long-context Gemma 4 serving:
```text
--max-length 262144
--device-memory-utilization 0.85
--kv-cache-format float8_e4m3fn
--enable-prefix-caching
--enable-chunked-prefill
--max-batch-size 1
--max-batch-input-tokens 16384
```
Hermes should point at that endpoint with this provider shape:
```yaml
model:
default: gemma-4-31B-it
provider: gemma4-31b-max
providers:
gemma4-31b-max:
name: Gemma 4 31B Modular MAX (256K)
api: https://llm.alhinai.dev/v1
api_key: not-needed
transport: chat_completions
default_model: gemma-4-31B-it
discover_models: true
models:
gemma-4-31B-it:
context_length: 262144
agent:
tool_use_enforcement: auto
```
Why this matters: Hermes sends OpenAI tool schemas and `tool_choice: "auto"`.
The model endpoint must support automatic tool choice so Hermes can initialize
the agent without a client-side workaround.
### Modular MAX for Gemma 4
The GB10 box uses a working Modular/MAX lab as the Gemma 4 bring-up,
validation, and serving path. Run MAX on `8001` for testing, then promote it to
the public endpoint when ready.
Installed local Codex skills:
```text
/home/alhinai/.codex/skills/new-modular-project
/home/alhinai/.codex/skills/mojo-syntax
/home/alhinai/.codex/skills/mojo-gpu-fundamentals
/home/alhinai/.codex/skills/mojo-python-interop
/home/alhinai/.codex/skills/import-model
/home/alhinai/.codex/skills/debug-model
/home/alhinai/.codex/skills/profile-model
```
Restart Codex after installing or changing those skills so they are picked up.
Working lab:
```bash
cd /home/alhinai/modular-lab
uv run mojo --version
uv run max --version
uv run mojo hello.mojo
```
Verified output includes:
```text
Mojo 1.0.0b3.dev2026062806
MAX 26.5.0.dev2026062806
hello from Mojo on spark
```
Important finding from `max list`:
| MAX support | Value |
| ------------ | ---------------------------------------- |
| Architecture | `Gemma4ForConditionalGeneration` |
| Example | `google/gemma-4-31B-it` |
| Encodings | `float4_e2m1fnx2`, `float16`, `bfloat16` |
That means Modular/MAX recognizes the Gemma 4 31B architecture on this GB10
machine.
Do not run two GPU model servers at once. Stop the existing Gemma server before
starting another MAX serve process:
```bash
docker stop <current-gemma-container>
```
Then test MAX on a separate port:
```bash
cd /home/alhinai/modular-lab
uv run max serve \
--model /home/alhinai/models/gemma-4-31B-it \
--served-model-name gemma-4-31B-it \
--devices gpu:0 \
--port 8001 \
--trust-remote-code \
--quantization-encoding bfloat16 \
--max-length 262144 \
--kv-cache-format float8_e4m3fn \
--device-memory-utilization 0.85 \
--enable-prefix-caching \
--enable-chunked-prefill \
--max-batch-size 1 \
--max-batch-input-tokens 16384
```
Verify MAX:
```bash
curl http://127.0.0.1:8001/v1/models
```
#### Replace the current Gemma server with Modular/MAX
If MAX works on `8001`, keep it there for testing until the public serving path
is intentionally changed.
Official resources:
- [MAX LLM Book](https://llm.modular.com/)
- [Model bring-up workflow](https://docs.modular.com/max/develop/model-bringup-workflow/)
- [Mojo](https://mojolang.org/)
- [Modular skills](https://github.com/modular/skills)
### Gemini Surfaces
Gemini remains the realtime perception and voice layer inside PodMan.
| Use | Model | Code |
| -------------------------- | ------------------------------- | ------------------------------------------ |
| Screen understanding | `gemini-2.0-flash` | `backend/src/vision/gemini.ts` |
| Urgent spoken alerts | `gemini-3.1-flash-tts-preview` | `backend/src/voice/live.ts` |
| Live room conversation | `gemini-3.1-flash-live-preview` | `agents/podman-live-conversation/agent.py` |
| Memory embeddings fallback | `gemini-embedding-001` | `backend/src/memory/vectors.ts` |
| Ambient background music | `lyria-3-clip-preview` | `backend/src/voice/music.ts` |
Voyage embeddings can be used first when `VOYAGE_API_KEY` is set. Gemini
embeddings remain the fallback. If no embedding provider is available, PodMan
falls back to exact-signature matching.
---
## How The Learning Loop Works
The learning loop is the product. A teammate only has to accept or dismiss an
intervention; the rest is captured automatically.
```text
observe -> detect -> recall prior outcomes -> policy gate -> act -> record outcome
^ |
+-------------------------- next recall -----------------------------+
```
| Stage | What happens | Code |
| ------- | --------------------------------------------------------------------------- | ----------------------------------- |
| Observe | Gemini Vision turns sampled screen frames into structured work context. | `backend/src/vision/gemini.ts` |
| Detect | PodMan detects overlapping files, symbols, research, and unpushed work. | `backend/src/collision/detector.ts` |
| Recall | MongoDB Atlas recalls similar prior events and outcomes. | `backend/src/memory/vectors.ts` |
| Gate | Policy suppresses dismissed false alarms and escalates recurring real ones. | `backend/src/memory/policy.ts` |
| Act | PodMan publishes a card, Hermes message, or urgent voice cue. | `backend/src/action/hermes.ts` |
| Learn | Accept/dismiss feedback is written back to memory. | `backend/src/memory/store.ts` |
---
## Runtime Components
| Layer | Runtime | Responsibility |
| ------------------ | ----------------------------------- | -------------------------------------------------------------------- |
| Frontend | React + Vite | Pod rooms, screen share, cards, voice controls, member state |
| Backend API | Express on `:8787` | LiveKit tokens, pod CRUD, outcomes, memory stats, sync PRs |
| Vision agent | Node + `@livekit/rtc-node` | Subscribes to screen tracks, samples frames, publishes interventions |
| Live voice agent | Python LiveKit Agents + Gemini Live | Real-time voice Q&A in a pod room |
| Memory | MongoDB Atlas | Observations, engineer state, collisions, interventions, outcomes |
| Reasoning agent | Hermes + Gemma Modular/MAX | Tool-using autonomous assistant and ops layer |
| Realtime transport | LiveKit Cloud | Screen tracks, audio tracks, data messages |
| Hosting | DigitalOcean + Caddy + systemd | Static frontend, API, workers, watchdog timers |
---
## Data Flow
```mermaid
sequenceDiagram
autonumber
participant Dev as Engineer PWA
participant API as Backend API
participant LK as LiveKit Room
participant Agent as Vision Agent
participant Gemini as Gemini APIs
participant Mongo as MongoDB Atlas
participant Hermes as Hermes/Gemma
Dev->>API: POST /api/token
API-->>Dev: LiveKit URL + JWT
Dev->>LK: Join pod room + publish screen share
Agent->>LK: Subscribe to screen-share video
Agent->>Gemini: Sampled frame
Gemini-->>Agent: Structured work context
Agent->>Mongo: Store observation
Agent->>Mongo: Recall similar prior events
Mongo-->>Agent: Prior outcome + policy hints
Agent->>Hermes: Escalate when autonomous help is useful
Agent->>LK: Publish card / message / voice cue
Dev->>API: POST /api/outcome
API->>Mongo: Store accept/dismiss feedback
```
---
## Public Interfaces
| Interface | Purpose |
| ----------------------------------------------------------- | ---------------------------------------------- |
| `GET /health` | API health check |
| `POST /api/token` | Mint LiveKit room tokens |
| `GET /api/pods` | List pods |
| `GET/POST/PATCH/DELETE /api/pods` | Pod CRUD |
| `POST/DELETE /api/pods/:id/members` | Pod membership |
| `GET /api/pods/:id/members/:name/history` | Recent member work history |
| `POST /api/outcome` | Store accepted/dismissed intervention outcomes |
| `GET /api/memory/stats` | Live memory collection counts |
| `POST /api/sync-pr` | Create a visible sync PR artifact |
| LiveKit topic `podman.intervention` | Intervention data channel |
| Wire messages `COLLISION`, `ACK`, `GIT_REPORT`, `VOICE_CUE` | Agent/PWA contract |
---
## Monorepo Layout
| Folder | Purpose |
| ----------- | ----------------------------------------------------------------------- |
| `frontend/` | React + Vite PWA |
| `backend/` | Express API, vision agent, memory, collision detection, Hermes job APIs |
| `agents/` | Python LiveKit conversation agent |
| `shared/` | Shared TypeScript contracts |
| `database/` | MongoDB setup and seed utilities |
| `infra/` | Caddy, Docker, DigitalOcean, systemd units |
| `scripts/` | Git watcher, deploy doctor, watchdog, verification tooling |
| `docs/` | Demo, deployment, learning, graph, and architecture notes |
---
## Local Development
Run the core app in three terminals:
```mermaid
flowchart LR
Env["1. Configure .env"] --> Install["2. pnpm install"]
Install --> API["Terminal A<br/>backend API :8787"]
Install --> Agent["Terminal B<br/>vision agent"]
Install --> UI["Terminal C<br/>frontend :5173"]
Install --> Voice["Optional<br/>conversation agent"]
API --> Browser["Open local PWA"]
Agent --> Browser
UI --> Browser
Voice --> Browser
classDef step fill:#eef8ee,stroke:#2f8a3a,color:#123915;
classDef run fill:#e8f1ff,stroke:#3366cc,color:#0b1f44;
class Env,Install step;
class API,Agent,UI,Voice,Browser run;
```
```bash
cp .env.example .env
# Fill LIVEKIT_*, GEMINI_*, GITHUB_*, MONGODB_URI.
pnpm install
pnpm --filter @podman/backend dev # API on :8787
pnpm --filter @podman/backend dev:agent # LiveKit vision agent
pnpm --filter @podman/frontend dev # PWA on :5173
```
Run the Python live conversation agent:
```bash
pnpm livekit:conversation:agent
```
Run the local git watcher on each demo laptop:
```bash
node scripts/podman-agent.mjs --name <engineer-name> --pod <pod-id>
```
Demo identities:
```bash
node scripts/podman-agent.mjs --name alice --pod demo-pod
node scripts/podman-agent.mjs --name bob --pod demo-pod
node scripts/podman-agent.mjs --name carol --pod demo-pod
```
---
## Production Operations
The production droplet is systemd-supervised. Caddy serves the built frontend
and proxies `/api/*` to the backend on `127.0.0.1:8787`.
```mermaid
flowchart LR
Public["https://podman.live"] --> Caddy["caddy.service"]
Caddy --> Static["/var/www/podman"]
Caddy --> API["podman-platform-api.service<br/>:8787"]
API --> Agent["podman-platform-agent.service"]
API --> Voice["podman-live-conversation-agent.service"]
Watchdog["podman-hermes-watchdog.timer"] --> Public
Sync["podman-hermes-sync-deploy.timer"] --> API
classDef public fill:#e8f1ff,stroke:#3366cc,color:#0b1f44;
classDef service fill:#eef8ee,stroke:#2f8a3a,color:#123915;
classDef timer fill:#fff6df,stroke:#c47f00,color:#3d2b00;
class Public public;
class Caddy,Static,API,Agent,Voice service;
class Watchdog,Sync timer;
```
| Service / timer | Purpose |
| ---------------------------------------- | -------------------------------------------------------------- |
| `podman-platform-api.service` | Built backend API on port `8787` |
| `podman-platform-agent.service` | Node LiveKit vision agent |
| `podman-live-conversation-agent.service` | Python Gemini Live conversation agent |
| `podman-hermes-watchdog.timer` | Periodic public health and remediation |
| `podman-hermes-sync-deploy.timer` | Clean-tree fast-forward deploy loop |
| `caddy.service` | Serves `/var/www/podman`, proxies `/api/*` to `127.0.0.1:8787` |
Check the app from the outside first:
```bash
curl https://podman.live/
curl https://podman.live/health
curl https://podman.live/api/pods
curl https://podman.live/api/presence
curl https://podman.live/api/memory/stats
```
Then check the droplet services:
```bash
systemctl is-active podman-platform-api podman-platform-agent
systemctl is-active podman-live-conversation-agent
systemctl is-active podman-hermes-watchdog.timer podman-hermes-sync-deploy.timer
```
Hermes operations scripts:
```bash
pnpm hermes:watchdog
pnpm hermes:watchdog:strict
pnpm hermes:sync-deploy
pnpm deploy:doctor:strict
```
Gemma Modular/MAX endpoint checks:
```bash
curl https://llm.alhinai.dev/v1/models \
-H "Authorization: Bearer not-needed"
curl https://llm.alhinai.dev/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer not-needed" \
-d '{
"model": "gemma-4-31B-it",
"messages": [{"role": "user", "content": "Reply with exactly: working"}],
"temperature": 0,
"max_tokens": 512
}'
```
---
## Required Environment
```bash
LIVEKIT_URL=wss://your-livekit-server.livekit.cloud
LIVEKIT_API_KEY=...
LIVEKIT_API_SECRET=...
LIVEKIT_CONVERSATION_AGENT_NAME=podman-live-conversation
GEMINI_API_KEY=...
GEMINI_VISION_MODEL=gemini-2.0-flash
GEMINI_LIVE_MODEL=gemini-3.1-flash-tts-preview
GEMINI_CONVERSATION_MODEL=gemini-3.1-flash-live-preview
GEMINI_EMBEDDING_MODEL=gemini-embedding-001
GEMINI_TTS_VOICE=Charon
GITHUB_TOKEN=...
GITHUB_REPO=karti-ai/podman
MONGODB_URI=mongodb+srv://...
VOYAGE_API_KEY=...
VOYAGE_EMBEDDING_MODEL=voyage-4-lite
PORT=8787
POD_ROOM=demo-pod
```
Gemma/Hermes provider values live in Hermes config, not PodMan `.env`:
```text
provider: gemma4-31b-max
model: gemma-4-31B-it
base_url: https://llm.alhinai.dev/v1
api_key: not-needed
```
---
## Verification
Before calling a deployment healthy:
```bash
pnpm verify
pnpm verify:infra
pnpm deploy:doctor:strict
pnpm hermes:watchdog:strict
```
For the public site:
```bash
curl https://podman.live/
curl https://podman.live/health
curl https://podman.live/api/pods
curl https://podman.live/api/presence
curl https://podman.live/api/memory/stats
```
For Hermes/Gemma:
```bash
hermes -z 'Reply with exactly: working' \
--provider gemma4-31b-max \
--model gemma-4-31B-it
```
Expected output:
```text
working
```
---
## Troubleshooting
| Symptom | Likely cause | Check |
| ----------------------------- | ------------------------------------------------ | ---------------------------------------------------- |
| Frontend loads but API fails | Backend or Caddy proxy issue | `systemctl status podman-platform-api caddy` |
| `/api/*` returns 502 | API not listening on `8787` | `ss -ltnp`, `curl http://127.0.0.1:8787/health` |
| No screen observations | Vision agent not in LiveKit room | `journalctl -u podman-platform-agent -n 80` |
| Live voice missing | Python conversation agent down | `journalctl -u podman-live-conversation-agent -n 80` |
| Memory empty | MongoDB unavailable or env missing | `curl /api/memory/stats`, backend logs |
| Hermes tool calls fail | Modular/MAX endpoint does not accept tool schema | verify Hermes provider and `/v1/chat/completions` |
| `llm.alhinai.dev` returns 502 | Gemma server still loading or tunnel target down | `curl /v1/models`, MAX logs on Gemma host |
---
## Why It Gets Better
PodMan is not a static alert system. It remembers what actually helped.
- A dismissed false alarm lowers future urgency.
- An accepted real collision raises future urgency for similar work.
- Exact-signature recall catches repeats even without vector search.
- MongoDB outcomes become the policy signal for the next session.
- Hermes and Gemma give the system a tool-using agent when the coordination
problem needs active investigation instead of a passive card.
The goal is simple: fewer interruptions, fewer duplicate branches, and a team
that can move fast without constantly asking what everyone else is doing.
Reference in New Issue View Git Blame Copy Permalink