docs: reposition around team-coordination, sync specs to code, prune stale docs
Reframe README around the real bottleneck (human coordination, not engineering ability) with the "five-minute meeting" cost story; rewrite the demo script to match. Update gemini/livekit/mongodb/digitalocean specs to reflect shipped code (Gemini Live agent, TTS, embeddings, Lyria via Interactions API; current API routes; hermes_jobs collections). Add hermes.md. Rename graph.md -> cont_learning.md. Remove outdated/dead docs (agent-learning scaffolding, graph-discovery, handoffs, superpowers, idea/plan/demo-setup) and the bundled LiveKit starter under examples/. Rewire CLAUDE.md doc-first gate off the deleted PLAN.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LuV8W8oNYRsDWKoqK8Mkqc
This commit is contained in:
Ramis
-791
@@ -1,791 +0,0 @@
|
||||
# PodMan - Canonical Master Plan
|
||||
|
||||
> Source of truth for PodMan product intent, current implementation truth, demo
|
||||
> strategy, public interfaces, risks, sponsor story, and next build order.
|
||||
>
|
||||
> If this file conflicts with `README.md`, `docs/idea.md`, `docs/livekit.md`,
|
||||
> `docs/gemini.md`, `docs/mongodb.md`, `docs/continual-learning/`,
|
||||
> `docs/graph-discovery/`, `docs/agent-learning/`, `docs/digitalocean.md`,
|
||||
> `docs/demo-setup.md`, or `docs/superpowers/specs/*`, follow this file and
|
||||
> treat the older docs as reference material to reconcile later.
|
||||
|
||||
---
|
||||
|
||||
## 1. Product thesis
|
||||
|
||||
**PodMan sees active work before it becomes visible to GitHub, remembers how the
|
||||
team works, researches better paths in the background, and coordinates teammates
|
||||
without being intrusive.**
|
||||
|
||||
GitHub knows pushed branches, PRs, issues, and comments. It cannot see the most
|
||||
expensive coordination failures while they are still forming on laptops: two
|
||||
engineers editing the same unpushed file, someone blocked on an endpoint a
|
||||
teammate is nearly done with, duplicated work starting silently, or a team
|
||||
walking into a dead-end implementation path.
|
||||
|
||||
PodMan puts engineers in a consented LiveKit pod, watches live IDE/screen
|
||||
context, fuses that with scheduled local git reports, GitHub state, MongoDB team
|
||||
memory, and background research, then routes only useful interventions through
|
||||
Hermes. The default is a small visual card. Hermes can message teammates when
|
||||
the team needs coordination. Voice is reserved for urgent escalation.
|
||||
|
||||
**One-line product definition:** PodMan is a non-intrusive, continual-learning
|
||||
team assistant for active coding.
|
||||
|
||||
**One-line demo promise:** PodMan notices live work, finds a better path,
|
||||
remembers a previous intervention, and escalates only when the team actually
|
||||
needs it.
|
||||
|
||||
---
|
||||
|
||||
## 2. Product contract
|
||||
|
||||
### Inputs
|
||||
|
||||
- **Live IDE/screen context:** engineers join a LiveKit room and publish screen
|
||||
share so the backend agent can sample real work in progress.
|
||||
- **Scheduled local git state:** each laptop should report dirty files,
|
||||
unpushed commits, branch, and latest commit about every minute. This is the
|
||||
deterministic fallback for facts vision cannot reliably infer.
|
||||
- **MongoDB team memory:** ownership, current tasks, blockers, repeated
|
||||
mistakes, preferred tools, decisions, intervention history, and outcomes.
|
||||
- **GitHub repo state:** public repo metadata, branches, PR artifacts, and
|
||||
issue/PR state when it exists.
|
||||
- **Background research signals:** tool, repo, skill, package, docs, and
|
||||
dead-end evidence discovered while teammates are working.
|
||||
|
||||
### Outputs
|
||||
|
||||
- **Default:** small visual intervention card in the PodMan frontend.
|
||||
- **Coordination:** Hermes message to the right teammate(s) or project channel.
|
||||
- **Urgent escalation:** voice only when timing or risk justifies interruption.
|
||||
- **Action path:** optional sync PR, research recommendation, summary, fix
|
||||
suggestion, or teammate notification.
|
||||
|
||||
### Memory rules
|
||||
|
||||
- Remember team-level work patterns, not raw screen recordings.
|
||||
- Store structured observations, collisions, interventions, outcomes, and pod
|
||||
state.
|
||||
- Add exact-signature recall before vector recall: normalized file, symbol,
|
||||
engineer pair, event type, and accepted/dismissed outcome.
|
||||
- Privacy must stay explicit: engineers consent by joining the pod and sharing
|
||||
screen context; do not store raw screenshots, full recordings, or secrets.
|
||||
|
||||
### Non-goals
|
||||
|
||||
- Not a dashboard as the product center.
|
||||
- Not a screenshot analyzer with no action loop.
|
||||
- Not sponsor-padding; every sponsor technology must be load-bearing or clearly
|
||||
marked as optional polish.
|
||||
- Not a task manager, Slack clone, full auth system, or general surveillance
|
||||
tool.
|
||||
|
||||
---
|
||||
|
||||
## 3. Track fit: Continual Learning
|
||||
|
||||
PodMan fits **Continual Learning** because the system gets more useful from team
|
||||
history and intervention outcomes.
|
||||
|
||||
- **Team model:** observations build ownership, hotspot, blocker, tool, and
|
||||
decision memory per pod.
|
||||
- **Outcome loop:** accepted, dismissed, and confirmed interventions become
|
||||
supervision for future thresholds and routing.
|
||||
- **Session compounding:** a later similar situation should reference prior
|
||||
memory, choose a better action sooner, or lower the noise level.
|
||||
- **Visible demo proof:** the first intervention writes memory; the second
|
||||
similar situation retrieves it and says, in effect, "I have seen this pattern
|
||||
before."
|
||||
|
||||
The learning proof should not depend on Atlas Vector Search being finished.
|
||||
Exact MongoDB recall is enough for the MVP learning beat.
|
||||
|
||||
---
|
||||
|
||||
## 4. Current implementation truth
|
||||
|
||||
Verified on `2026-06-27` from local repo inspection, authenticated `gh`, and
|
||||
the current remote plan commit.
|
||||
|
||||
### GitHub state
|
||||
|
||||
- Repo: <https://github.com/karti-ai/podman>
|
||||
- Visibility: public
|
||||
- Default branch: `main`
|
||||
- Current local branch: `main`
|
||||
- Local branch state during this rewrite: behind `origin/main` by two commits
|
||||
- Issues: none
|
||||
- PRs: none
|
||||
- `origin/main` latest relevant commits:
|
||||
- `8271188 feat(frontend): live room view, beat connectivity test, session resume`
|
||||
- `65a0791 docs(plan): audit server state + mark tasks 1-5 done, reflect actual arch`
|
||||
|
||||
### Working / started
|
||||
|
||||
- Monorepo packages exist: `frontend`, `backend`, `shared`, `database`, and
|
||||
`infra`.
|
||||
- Backend is split into two processes:
|
||||
- API service in `backend/src/server.ts`.
|
||||
- LiveKit agent worker in `backend/src/agent.ts`.
|
||||
- Backend API exposes:
|
||||
- `GET /health`
|
||||
- `POST /api/token`
|
||||
- `POST /api/sync-pr`
|
||||
- `POST /api/outcome`
|
||||
- `GET /api/memory/stats`
|
||||
- `GET /api/pods`
|
||||
- `POST /api/pods`
|
||||
- `GET /api/pods/:id`
|
||||
- `PATCH /api/pods/:id`
|
||||
- `DELETE /api/pods/:id`
|
||||
- `POST /api/pods/:id/members`
|
||||
- `DELETE /api/pods/:id/members/:name`
|
||||
- Remote API health check returned `{"ok":true}` at
|
||||
`http://165.22.129.249:8787/health` during verification.
|
||||
- The LiveKit agent uses `@livekit/rtc-node` to join as `podman-agent`, subscribe
|
||||
to `TrackSource.SOURCE_SCREENSHARE`, sample frames near 1 fps, convert frames
|
||||
to RGBA, and encode downscaled JPEGs with `sharp`.
|
||||
- Gemini vision is wired in `backend/src/vision/gemini.ts` with JSON structured
|
||||
output, response schema, low media resolution, and model ID from env.
|
||||
- Collision detection exists and groups engineer contexts by normalized file,
|
||||
then fires when 2+ engineers touch the same file and at least one unpushed or
|
||||
dirty signal exists.
|
||||
- Shared LiveKit data topic and wire messages exist:
|
||||
- topic: `podman.intervention`
|
||||
- messages: `COLLISION`, `VOICE_CUE`, `ACK`, `GIT_REPORT`
|
||||
- MongoDB persistence groundwork exists for observations, collisions,
|
||||
interventions, outcomes, and pods.
|
||||
- Frontend has pod selection, pod join, post-join pod view, LiveKit join helper,
|
||||
and dev-mode fallback.
|
||||
- `origin/main` adds live room participants, active-speaker state, session
|
||||
resume, a "Play beat" audio connectivity test, and a deliberate "Share my
|
||||
screen" button that publishes with `Track.Source.ScreenShare`. Merge that
|
||||
remote commit before doing more frontend work on the local checkout.
|
||||
- DigitalOcean infra scaffolding exists:
|
||||
- `infra/.do/app.yaml` is the split App Platform direction.
|
||||
- `infra/app.yaml` is an older single-service backend spec and should be
|
||||
treated as legacy until reconciled.
|
||||
|
||||
### Server snapshot
|
||||
|
||||
From the remote plan snapshot and health check on `2026-06-27`:
|
||||
|
||||
- Backend API: running on `http://165.22.129.249:8787` and `/health` returned
|
||||
`{"ok":true}`.
|
||||
- Frontend: reported running on `:81`; port `80` was already taken.
|
||||
- Agent worker: reported not running; it still needs LiveKit credentials and
|
||||
`pnpm --filter @podman/backend dev:agent`.
|
||||
- Treat this as operational evidence, not architecture truth. Reverify before
|
||||
demo.
|
||||
|
||||
### Partial / completed since the original audit
|
||||
|
||||
- `backend/src/voice/live.ts` now publishes a `VOICE_CUE` fallback and attempts
|
||||
Gemini audio publication into LiveKit. The agent only calls it for critical
|
||||
interventions so voice remains an urgent escalation path.
|
||||
- Hermes now has a data-channel teammate message path via `HERMES_MESSAGE` on
|
||||
the existing `podman.intervention` topic. This is the MVP notification bridge,
|
||||
not a Slack/Discord integration.
|
||||
- `backend/src/memory/vectors.ts` implements exact-signature recall first and
|
||||
can use Voyage/Gemini embeddings with Atlas Vector Search when configured.
|
||||
- Exact-signature recall now attaches prior interventions/outcomes and prefers
|
||||
accepted real collisions, giving the learning beat deterministic MongoDB
|
||||
proof before vector search.
|
||||
- `backend/src/memory/policy.ts` now uses severity, per-pod cooldown, and prior
|
||||
outcome history. It is still a simple policy, not a trained threshold model.
|
||||
- `POST /api/sync-pr` now creates a visible Markdown sync artifact commit before
|
||||
opening the PR.
|
||||
- Frontend `PodView` renders intervention cards, Hermes messages, voice cues,
|
||||
and the accepted sync PR artifact link.
|
||||
- Browser screen publishing exists, but the active join path must be proven to
|
||||
tag tracks as screen share so the backend agent can filter them correctly. The
|
||||
`origin/main` screen-share button appears to address this; local code remains
|
||||
behind until that commit is merged.
|
||||
- `GIT_REPORT` exists in shared types and agent handling. `scripts/podman-agent.mjs`
|
||||
is the finished per-laptop git sidecar — polls every 15 s, upserts git fields
|
||||
to `engineer_states` collection. The backend agent now fuses those Mongo
|
||||
git-state fields into live contexts before collision detection; direct
|
||||
LiveKit `GIT_REPORT` publication from the sidecar remains optional.
|
||||
- Background research recommendations are a product requirement and demo goal,
|
||||
not an implemented research agent yet.
|
||||
- Deployment reliability is partial; API health is reachable, but API/static
|
||||
site/worker together must still be reverified before demo.
|
||||
- Env docs now align on `gemini-3.5-flash` for vision and
|
||||
`gemini-3.1-flash-tts-preview` for voice. The backend still preserves a Gemini
|
||||
Live path for future available Live models.
|
||||
|
||||
### Not yet proven
|
||||
|
||||
- Real browser -> LiveKit room -> backend agent screen-frame capture end to end.
|
||||
- Real Gemini inference from a live shared IDE frame using the stage key/model.
|
||||
- Real data-channel intervention card rendering in the active frontend.
|
||||
- Hermes message routing to teammates.
|
||||
- Voice escalation heard by participants through LiveKit, including
|
||||
duration-based track holding so longer Gemini TTS announcements finish.
|
||||
- A meaningful real sync PR flow with correct GitHub scopes and artifact.
|
||||
- Atlas Vector Search / Voyage recall path.
|
||||
- DigitalOcean static site + API service + LiveKit agent worker all running
|
||||
together.
|
||||
- Background research recommendation that is both timely and evidence-backed.
|
||||
|
||||
---
|
||||
|
||||
## 5. Architecture to build toward
|
||||
|
||||
```
|
||||
Engineer browser PWA
|
||||
- joins a pod room
|
||||
- publishes screen share and optional mic
|
||||
- receives intervention cards and voice
|
||||
|
|
||||
v
|
||||
LiveKit room
|
||||
- one room per pod
|
||||
- screen-share tracks are the live work signal
|
||||
- small reliable data packets carry interventions
|
||||
|
|
||||
v
|
||||
PodMan backend agent worker
|
||||
- @livekit/rtc-node room participant
|
||||
- screen-track subscription
|
||||
- frame throttle and JPEG encode
|
||||
- Gemini structured vision
|
||||
- scheduled GIT_REPORT fusion
|
||||
- GitHub state fusion
|
||||
- collision, blocker, duplicate-work, and dead-end detection
|
||||
- MongoDB memory recall and policy
|
||||
|
|
||||
v
|
||||
Hermes action layer
|
||||
- visual card routing
|
||||
- teammate messages
|
||||
- urgent voice escalation
|
||||
- optional research/action/sync PR workflows
|
||||
|
|
||||
v
|
||||
Backend API + MongoDB + GitHub
|
||||
- token minting, pod CRUD, outcomes, memory stats
|
||||
- observations, collisions, interventions, outcomes, pod memory
|
||||
- public repo state and PR artifacts
|
||||
```
|
||||
|
||||
The backend must remain split:
|
||||
|
||||
- **API service:** routable HTTP process with `/api/*` endpoints and health
|
||||
checks.
|
||||
- **Agent worker:** outbound LiveKit participant with no HTTP health-check port
|
||||
requirement.
|
||||
|
||||
This split matters for DigitalOcean App Platform: the LiveKit agent should be a
|
||||
worker, not a web service that App Platform expects to health-check over HTTP.
|
||||
|
||||
---
|
||||
|
||||
## 6. Public interfaces to preserve
|
||||
|
||||
Do not rename or reshape these without updating frontend, backend, docs, and demo
|
||||
scripts together.
|
||||
|
||||
### Backend HTTP
|
||||
|
||||
- `GET /health`
|
||||
- `POST /api/token`
|
||||
- `POST /api/sync-pr`
|
||||
- `POST /api/outcome`
|
||||
- `GET /api/memory/stats`
|
||||
- `GET /api/pods/:id/graph`
|
||||
- `GET /api/pods/:id/graph/reach/:nodeId`
|
||||
- `GET /api/pods`
|
||||
- `POST /api/pods`
|
||||
- `GET /api/pods/:id`
|
||||
- `PATCH /api/pods/:id`
|
||||
- `DELETE /api/pods/:id`
|
||||
- `POST /api/pods/:id/members`
|
||||
- `DELETE /api/pods/:id/members/:name`
|
||||
|
||||
### LiveKit data channel
|
||||
|
||||
- Topic: `podman.intervention`
|
||||
- Core messages:
|
||||
- `COLLISION`: agent -> PWA; contains `collision` and `intervention`.
|
||||
- `ACK`: PWA -> agent/API; intervention response.
|
||||
- `GIT_REPORT`: local git sidecar -> agent; dirty/unpushed ground truth.
|
||||
- `VOICE_CUE`: text cue/fallback for voice escalation.
|
||||
|
||||
### Required environment
|
||||
|
||||
```bash
|
||||
LIVEKIT_URL=
|
||||
LIVEKIT_API_KEY=
|
||||
LIVEKIT_API_SECRET=
|
||||
|
||||
GEMINI_API_KEY=
|
||||
GEMINI_VISION_MODEL=
|
||||
GEMINI_LIVE_MODEL=
|
||||
GEMINI_TTS_VOICE=
|
||||
|
||||
GITHUB_TOKEN=
|
||||
GITHUB_REPO=karti-ai/podman
|
||||
|
||||
MONGODB_URI=
|
||||
VOYAGE_API_KEY=
|
||||
POD_ROOM=demo-pod
|
||||
PORT=8787
|
||||
|
||||
VITE_BACKEND_URL=http://localhost:8787
|
||||
VITE_LIVEKIT_URL=
|
||||
```
|
||||
|
||||
Keep all non-`VITE_` secrets server-side.
|
||||
|
||||
---
|
||||
|
||||
## 7. Critical implementation callouts
|
||||
|
||||
### LiveKit
|
||||
|
||||
- Screen share is a video track. The backend agent should consume raw screen
|
||||
frames through `@livekit/rtc-node`.
|
||||
- The agent must filter screen share, not webcam:
|
||||
`pub.source === TrackSource.SOURCE_SCREENSHARE`.
|
||||
- Frontend publishing must tag the track as screen share; otherwise the agent can
|
||||
miss it.
|
||||
- Throttle aggressively. Screens can arrive near video frame rate; Gemini should
|
||||
receive sampled frames only.
|
||||
- Keep reliable data packets small. Use them for intervention metadata, not
|
||||
screenshots, large diffs, or research dumps. Treat reliable payloads as
|
||||
roughly 15 KiB max.
|
||||
- A historical closed `livekit/node-sdks` issue reported high memory use when
|
||||
consuming video; run memory checks during agent frame tests and stop if the
|
||||
loop leaks.
|
||||
|
||||
### Gemini
|
||||
|
||||
- Use structured output for vision: JSON mime type plus response schema.
|
||||
- Use low media resolution for ambient screen watching; reserve higher
|
||||
resolution for debugging or targeted inspection.
|
||||
- Never expose `GEMINI_API_KEY` to the browser.
|
||||
- Use card + Hermes message first. For urgent stage audio, default to Gemini TTS
|
||||
published through LiveKit; keep browser TTS only as an explicit fallback flag.
|
||||
- Keep model IDs in env so preview/availability changes do not require code
|
||||
changes.
|
||||
|
||||
### MongoDB
|
||||
|
||||
- Local MongoDB is fine for dev CRUD and memory counts.
|
||||
- Atlas or Atlas Local is needed for the sponsor-grade Vector Search story.
|
||||
- Build exact-signature recall first:
|
||||
normalized file + symbol + engineer pair + event type + outcome.
|
||||
- Writes from the agent should be best-effort. Mongo hiccups should degrade
|
||||
memory, not kill live detection.
|
||||
- Do not store raw screenshots or recordings.
|
||||
|
||||
### GitHub
|
||||
|
||||
- The repo is public and currently has no issue/PR backlog, so do not make the
|
||||
plan issue-driven yet.
|
||||
- GitHub cannot see local dirty files or unpushed commits. That is still a core
|
||||
product moat.
|
||||
- Sync PRs should use deterministic GitHub REST/Octokit flows, not browser
|
||||
automation.
|
||||
- Verify token scopes and demo repo permissions before stage time.
|
||||
|
||||
### DigitalOcean
|
||||
|
||||
- Use App Platform as:
|
||||
- static site for frontend,
|
||||
- HTTP service for API,
|
||||
- worker for the LiveKit agent.
|
||||
- Do not model the agent worker as a health-checked HTTP service.
|
||||
- Keep a local and recorded fallback even if deployment works; venue network is a
|
||||
stage risk.
|
||||
|
||||
### Hermes
|
||||
|
||||
- Treat Hermes as the action and messaging layer, not as a replacement for the
|
||||
current implemented backend agent until code changes make that real.
|
||||
- Hermes should choose the least intrusive channel:
|
||||
card -> message -> voice.
|
||||
- Hermes can own research summaries, teammate notification, sync PR initiation,
|
||||
and urgent escalation once those workflows exist.
|
||||
|
||||
---
|
||||
|
||||
## 8. Build ladder
|
||||
|
||||
Do not mark a rung done until it is proven in logs, UI, or a visible external
|
||||
artifact.
|
||||
|
||||
### P0 - make the live loop undeniable
|
||||
|
||||
1. **Preserve and reconcile the plan**
|
||||
- Merge local `docs/PLAN.md` with `origin/main:docs/PLAN.md`.
|
||||
- Keep both the broad product thesis and concrete server/current-state facts.
|
||||
- After the docs are safe, merge or rebase the two newer `origin/main` commits
|
||||
before implementing frontend work.
|
||||
|
||||
2. **Browser publish proof**
|
||||
- Start backend API and frontend.
|
||||
- Join a real LiveKit room from the browser.
|
||||
- Confirm the browser publishes a screen-share track with the correct source.
|
||||
|
||||
3. **Agent frame proof**
|
||||
- Start `pnpm --filter @podman/backend dev:agent`.
|
||||
- Confirm room join, screen-track subscription, frame sampling, and JPEG
|
||||
encode logs.
|
||||
- Watch process memory while consuming frames.
|
||||
|
||||
4. **Gemini vision proof**
|
||||
- Send one live sampled IDE frame to Gemini.
|
||||
- Log parsed JSON with `currentFile`, `currentSymbol`, `activity`,
|
||||
`hasUnpushedChanges`, and `confidence`.
|
||||
- Add a confidence/logging gate if noisy frames cause bad reads.
|
||||
|
||||
5. **Scheduled git truth** ✅ partial
|
||||
- `scripts/podman-agent.mjs` polls every 15 s: `git status --short`,
|
||||
`git diff --stat HEAD`, `git log --oneline -1`, `git branch --show-current`.
|
||||
- Upserts `changedFiles`, `diffStat`, `recentCommit`, `branch`, `gitUpdatedAt`
|
||||
to `engineer_states` collection in MongoDB (upsert by `podId::name` key).
|
||||
- **Still needed:** fuse `engineer_states` git fields into the collision
|
||||
detector, and/or publish `GIT_REPORT` data channel messages so the agent
|
||||
worker can incorporate git truth into vision-based decisions.
|
||||
|
||||
6. **Intervention card + Hermes notification**
|
||||
- Publish a real intervention on `podman.intervention`.
|
||||
- Render it as a small card in the frontend.
|
||||
- Route a Hermes message to the affected teammate(s) or project channel once
|
||||
the bridge exists.
|
||||
|
||||
7. **Background research recommendation**
|
||||
- When the team is heading into a poor tool/repo/skill choice or dead end,
|
||||
produce a recommendation card with short evidence.
|
||||
- Minimum evidence: why it matters, what to use instead, and who should act.
|
||||
|
||||
8. **Learning proof**
|
||||
- First intervention writes observation/collision/recommendation/outcome
|
||||
memory.
|
||||
- Second similar situation retrieves exact prior memory and changes the
|
||||
message: "I have seen this pattern before."
|
||||
|
||||
9. **Urgency routing**
|
||||
- Default to card.
|
||||
- Escalate to Hermes message when coordination involves other teammates.
|
||||
- Escalate to voice only when urgent.
|
||||
|
||||
10. **Action artifact**
|
||||
- If demo uses same-file collision, click the card to open a real sync PR
|
||||
artifact or visible GitHub artifact.
|
||||
- If demo uses research recommendation, show the accepted recommendation and
|
||||
memory outcome instead.
|
||||
|
||||
11. **Deployment or fallback proof**
|
||||
- Prove API/static/worker deployment together, or explicitly run local with a
|
||||
recorded backup.
|
||||
- Keep backup video on a separate device.
|
||||
|
||||
### P0.5 - RSI negative-feedback activation (continual-learning)
|
||||
|
||||
The continual-learning loop records outcomes but never feeds the negative
|
||||
signal back. Live Atlas (2026-06-28): `outcomes` = 22 accepted / 85 dismissed,
|
||||
yet `wasRealCollision` is `true` in 107/107 (hardcoded), so the suppression
|
||||
gate is dead and dismissals are unused. These two rungs activate the loop with
|
||||
no schema change. Owner: RSI track. Independent of the MongoDB-cleanup handoff.
|
||||
|
||||
1. **Step 1 - suppress on prior dismissal alone** ✅
|
||||
- `backend/src/memory/policy.ts` `shouldIntervene`: remove the dead
|
||||
`&& !priorOutcome.wasRealCollision` term so a prior `accepted === false`
|
||||
suppresses the next identical-signature nudge.
|
||||
- Spec: `docs/continual-learning/policy.md:41` (dismissed = negative signal),
|
||||
`spec.md:163` (dismissals adapt suppression).
|
||||
- Caveat: recall is single-shot most-recent (`memory/vectors.ts`), so this is
|
||||
"last-outcome-wins" until Step 3 (derive `wasRealCollision`) lands.
|
||||
|
||||
2. **Step 2 - gate the recall severity escalation** ✅
|
||||
- `backend/src/agent/podman.ts` `handle`: only force `severity = 'critical'`
|
||||
when the recalled prior was an accepted *real* collision, instead of
|
||||
blanket-escalating every recall. Surfaces the learned routing in
|
||||
`preferredAction`; stops dismissed/false priors over-escalating to voice.
|
||||
- Spec: `docs/continual-learning/policy.md:62-63` (prefer prior accepted
|
||||
kind), `plan.md:66` (second similar event behaves differently).
|
||||
|
||||
3. **Step 3 - derive `wasRealCollision` from git overlap (backend-authoritative)** ✅
|
||||
- Overlap is captured AT detection time as `Collision.gitOverlap`
|
||||
(`backend/src/agent/podman.ts`), while `engineer_states` are still fresh —
|
||||
true only if ALL involved engineers have the collided file in their git
|
||||
`changedFiles`, matched on case/whitespace-canonical names.
|
||||
- `backend/src/memory/store.ts` `recordOutcome` overrides the client value
|
||||
with `deriveWasRealCollision()`, which prefers the stored `gitOverlap`
|
||||
(immune to late clicks / stale sidecars / the 120s TTL) and only falls back
|
||||
to a live canonical-name re-derivation for pre-existing collisions.
|
||||
`frontend/.../useInterventions.ts` stops sending hardcoded `true`.
|
||||
- Restores the (accepted × wasReal) 2×2 the spec assumes; keeps `learned_from`
|
||||
edges (`graph/live.ts:413`) from being silently zeroed on stage.
|
||||
- Spec: `docs/continual-learning/spec.md:98-108`, `policy.md:35-42`.
|
||||
- Hardened per Codex review (name canonicalization + detection-time capture).
|
||||
|
||||
Follow-ups (separate rungs, not in this change): Step 4-5 `strategy_versions` +
|
||||
Gemini-proposed `LearningProposal` slice; Step 6 durable `owns` write; seed a
|
||||
clean demo pod with a repeated dismissed signature (the historic dismissals are
|
||||
orphaned — `collisionId` resolves to no collision — so they cannot drive the
|
||||
demo verifier).
|
||||
|
||||
### P1 - polish the money moment
|
||||
|
||||
- Add visible live inference captions in the PWA.
|
||||
- Add a small memory stats panel backed by `/api/memory/stats`.
|
||||
- Keep browser-side TTS as an explicit demo fallback only; Gemini TTS over
|
||||
LiveKit is the default urgent-voice path.
|
||||
- Add Hermes notification bridge once the target channel is chosen.
|
||||
- Improve research cards with compatibility, install effort, docs quality, repo
|
||||
health, and security/trust signals.
|
||||
|
||||
### P2 - sponsor and scale polish
|
||||
|
||||
- Implement Voyage embedding + Atlas Vector Search recall.
|
||||
- Improve policy learning from outcomes.
|
||||
- Deploy DigitalOcean static site + API service + worker as the submission path.
|
||||
- Add optional GitHub issue/PR backlog integration after issues/PRs actually
|
||||
exist.
|
||||
|
||||
### Cut if behind
|
||||
|
||||
- Webcam grid.
|
||||
- Mic transcription.
|
||||
- Full auth/accounts.
|
||||
- Slack/Linear/Jira integrations unless Hermes requires one immediately.
|
||||
- Complex dashboards.
|
||||
- Live voice polish beyond the Gemini TTS urgent-alert path.
|
||||
- Vector Search if exact Mongo recall demonstrates the learning beat.
|
||||
|
||||
---
|
||||
|
||||
## 9. Critical 3-minute demo script
|
||||
|
||||
**Rule:** open on one active IDE, not a grid. PodMan is an agent, not a
|
||||
dashboard.
|
||||
|
||||
1. **0:00 - Set the scene**
|
||||
- One engineer is actively coding in the IDE.
|
||||
- The presenter says: "This work is not pushed yet. GitHub cannot see it."
|
||||
|
||||
2. **0:20 - Show the live signal**
|
||||
- Show a compact caption: current file, inferred task, git dirty/unpushed
|
||||
state.
|
||||
- Show that PodMan is watching consented screen context, not stored
|
||||
recordings.
|
||||
|
||||
3. **0:40 - Introduce the better-tool moment**
|
||||
- A teammate starts down a weak path: wrong package, dead repo, bad API,
|
||||
duplicated effort, or risky implementation.
|
||||
- PodMan has been researching in the background.
|
||||
|
||||
4. **1:05 - Money moment**
|
||||
- PodMan shows a small card:
|
||||
"This path is likely a dead end. Use X instead; it matches our stack and is
|
||||
actively maintained."
|
||||
- The card names the affected teammate and the suggested action.
|
||||
|
||||
5. **1:25 - Hermes coordination**
|
||||
- Hermes notifies the right teammate(s), not the whole room.
|
||||
- No voice yet unless the situation is urgent.
|
||||
|
||||
6. **1:50 - Learning beat**
|
||||
- A similar issue appears.
|
||||
- PodMan references memory:
|
||||
"I have seen this pattern before. Last time the team accepted the X
|
||||
recommendation."
|
||||
- Show `/api/memory/stats` or the visible memory indicator.
|
||||
|
||||
7. **2:20 - Urgency escalation**
|
||||
- Raise the severity with a same-file collision, blocking dependency, failing
|
||||
test, or imminent bad push.
|
||||
- Hermes escalates to voice only now.
|
||||
|
||||
8. **2:40 - Close**
|
||||
- Show the public repo, deployed/local URL, and memory stats.
|
||||
- Closing line: "PodMan coordinates work while it is still happening."
|
||||
|
||||
### Reliable fallback demo
|
||||
|
||||
If the research recommendation is not reliable by stage time, use the same-file
|
||||
collision fallback:
|
||||
|
||||
1. Two engineers open the same visible file.
|
||||
2. `GIT_REPORT` or vision marks one as dirty/unpushed.
|
||||
3. Agent publishes `COLLISION` on `podman.intervention`.
|
||||
4. Frontend renders the card.
|
||||
5. The card opens a sync PR artifact.
|
||||
6. A second similar collision retrieves prior memory.
|
||||
|
||||
---
|
||||
|
||||
## 10. Sponsor strategy
|
||||
|
||||
### Gemini
|
||||
|
||||
Gemini must be load-bearing for the vision loop:
|
||||
|
||||
- live IDE/screen frame -> structured work context,
|
||||
- optional message/recommendation generation,
|
||||
- optional Live voice only after card/Hermes routing is stable.
|
||||
|
||||
Do not overclaim voice if it is using browser/pre-generated TTS. Say plainly that
|
||||
it is the reliability fallback.
|
||||
|
||||
### LiveKit
|
||||
|
||||
LiveKit is the real-time spine:
|
||||
|
||||
- engineers join one pod room,
|
||||
- screen-share tracks carry active work context,
|
||||
- PodMan joins as a participant,
|
||||
- data packets carry interventions,
|
||||
- voice can be added as urgent escalation.
|
||||
|
||||
Pitch line: "Unpushed work is invisible to GitHub, so real-time presence is the
|
||||
only way to coordinate before the push."
|
||||
|
||||
### MongoDB + Voyage
|
||||
|
||||
MongoDB is the learning proof:
|
||||
|
||||
- observations, collisions, recommendations, interventions, and outcomes persist,
|
||||
- prior memory changes a later intervention,
|
||||
- exact recall is the MVP,
|
||||
- Voyage + Atlas Vector Search is the stronger sponsor-grade version after exact
|
||||
recall works.
|
||||
|
||||
Canonical docs:
|
||||
|
||||
- [`docs/continual-learning/`](continual-learning/) owns team memory and
|
||||
outcome-backed recall.
|
||||
- [`docs/graph-discovery/`](graph-discovery/) owns graph materialization,
|
||||
hygiene, and `$graphLookup` reachability.
|
||||
- [`docs/agent-learning/`](agent-learning/) owns the planned narrow
|
||||
strategy-version layer. Full autonomous promotion is not implemented unless
|
||||
backed by records.
|
||||
|
||||
### DigitalOcean
|
||||
|
||||
DigitalOcean earns its place when:
|
||||
|
||||
- frontend runs as a static site,
|
||||
- API runs as an HTTP service,
|
||||
- LiveKit agent runs as a worker,
|
||||
- public URL is shown in submission or demo.
|
||||
|
||||
Local fallback is acceptable for stage reliability, but the submission should
|
||||
include the deployment URL if possible.
|
||||
|
||||
---
|
||||
|
||||
## 11. Risks and mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Looks like a dashboard | Keep the UI quiet. Hero is card/message/action, not a grid. |
|
||||
| Looks like a screenshot analyzer | Always show screen signal + git truth + memory + action. |
|
||||
| Interrupts too much | Default to cards, escalate to Hermes messages, reserve voice for urgency. |
|
||||
| Overclaims implemented features | Mark voice, Hermes bridge, vectors, adaptive policy, research agent, real sync PR, and DO worker deploy incomplete until proven. |
|
||||
| Vision misses unpushed state | Use scheduled `GIT_REPORT` for deterministic dirty/unpushed truth. |
|
||||
| Research recommendation lacks evidence | Show only concise evidence: stack fit, repo/tool health, install effort, docs/trust signal. |
|
||||
| No visible learning | Build exact Mongo recall before vector search. |
|
||||
| LiveKit frame loop leaks memory | Monitor agent memory during video consumption; throttle hard. |
|
||||
| GitHub issue/PR backlog absent | Do not invent issue-driven backlog; repo currently has no issues or PRs. |
|
||||
| Venue network failure | Rehearse on hotspot and keep recorded backup. |
|
||||
| DO worker deploy hangs | Deploy agent as worker, not health-checked service. |
|
||||
|
||||
---
|
||||
|
||||
## 12. Documentation reconciliation tasks
|
||||
|
||||
After this plan is accepted, update the supporting docs so they stop conflicting
|
||||
with this file:
|
||||
|
||||
- `README.md`: replace POST-screenshot-first language with LiveKit screen-track
|
||||
agent architecture and Hermes action-layer wording.
|
||||
- `docs/idea.md`: broaden from blocker/dependency voice demo to card/message/
|
||||
urgent-voice coordination plus research and memory.
|
||||
- `docs/livekit.md`: remove "Hermes does NOT subscribe to engineer screen
|
||||
tracks"; current architecture uses backend agent screen subscription.
|
||||
- `docs/gemini.md`: keep structured vision, but mark Gemini Live as P1 and avoid
|
||||
claiming voice is implemented.
|
||||
- `docs/mongodb.md`: align collection names with current code
|
||||
(`observations`, `collisions`, `interventions`, `outcomes`, `pods`) and add
|
||||
exact-signature recall.
|
||||
- `docs/digitalocean.md`: split API service and agent worker; do not deploy the
|
||||
worker as a health-checked HTTP service; mark `infra/app.yaml` legacy or
|
||||
reconcile it with `infra/.do/app.yaml`.
|
||||
- `docs/demo-setup.md`: update the script to include better-tool research,
|
||||
learning recall, Hermes notification, and urgency-based voice.
|
||||
|
||||
---
|
||||
|
||||
## 13. Acceptance checklist
|
||||
|
||||
Before saying PodMan is demo-ready:
|
||||
|
||||
- [ ] `pnpm format:check` passes or all failures are documented as unrelated.
|
||||
- [ ] `pnpm typecheck` passes.
|
||||
- [ ] Browser joins a real LiveKit room.
|
||||
- [ ] Browser publishes a screen-share track with the correct source.
|
||||
- [ ] Backend agent subscribes to the screen-share track.
|
||||
- [ ] Agent logs at least one parsed Gemini context from a real IDE screen.
|
||||
- [x] Local git report supplies dirty/unpushed truth on a schedule (`scripts/podman-agent.mjs` — 15 s poll → MongoDB `engineer_states`). Agent fusion still needed.
|
||||
- [x] Frontend renders a real intervention card.
|
||||
- [x] Hermes notification path works for teammate messages over the LiveKit data
|
||||
channel.
|
||||
- [ ] Voice is heard only for urgent escalation or a fallback is declared.
|
||||
- [x] Outcome ACK writes to MongoDB and updates intervention status.
|
||||
- [x] `/api/memory/stats` shows counts increasing.
|
||||
- [x] Second similar situation uses prior exact memory in the message.
|
||||
- [ ] Research recommendation card is evidence-backed, or fallback collision demo
|
||||
is used.
|
||||
- [x] Sync PR action creates a visible GitHub artifact if used in demo.
|
||||
- [ ] DigitalOcean deployment or local fallback is rehearsed.
|
||||
- [ ] Backup recording is ready on a separate device.
|
||||
|
||||
---
|
||||
|
||||
## 14. Evidence appendix
|
||||
|
||||
### Repo and GitHub state
|
||||
|
||||
- Public repo: <https://github.com/karti-ai/podman>
|
||||
- Verified with authenticated `gh` on `2026-06-27`.
|
||||
- Default branch: `main`.
|
||||
- No GitHub issues or PRs existed at verification time.
|
||||
|
||||
### Hackathon / event
|
||||
|
||||
- AI Engineer World's Fair: <https://www.ai.engineer/worldsfair/2026>
|
||||
- Cerebral Valley hackathon page:
|
||||
<https://cerebralvalley.ai/e/aiewf-hackathon-2026>
|
||||
|
||||
### LiveKit
|
||||
|
||||
- Screen share docs: <https://docs.livekit.io/transport/media/screenshare/>
|
||||
- Data packets docs: <https://docs.livekit.io/transport/data/packets/>
|
||||
- Node SDK reference: <https://docs.livekit.io/reference/client-sdk-node/>
|
||||
- Node SDK releases: <https://github.com/livekit/node-sdks/releases>
|
||||
- Node SDK issue risk: <https://github.com/livekit/node-sdks/issues/444>
|
||||
|
||||
### Gemini
|
||||
|
||||
- Structured output:
|
||||
<https://ai.google.dev/gemini-api/docs/structured-output>
|
||||
- Media resolution: <https://ai.google.dev/gemini-api/docs/media-resolution>
|
||||
- Live API: <https://ai.google.dev/gemini-api/docs/live-api>
|
||||
|
||||
### DigitalOcean
|
||||
|
||||
- App Platform app spec:
|
||||
<https://docs.digitalocean.com/products/app-platform/reference/app-spec/>
|
||||
|
||||
### MongoDB
|
||||
|
||||
- Vector Search index type:
|
||||
<https://www.mongodb.com/docs/vector-search/index/vector-search-type/>
|
||||
- Node driver Atlas Vector Search:
|
||||
<https://www.mongodb.com/docs/drivers/node/current/atlas-vector-search/>
|
||||
@@ -1,43 +0,0 @@
|
||||
# Agent Learning
|
||||
|
||||
Status: planned / narrow v1
|
||||
|
||||
Agent learning owns how PodMan can improve its own prompts, detector rules,
|
||||
policies, verifier choices, and routing strategies. This is deliberately
|
||||
narrower than team memory: it is a versioned strategy layer, not autonomous code
|
||||
rewriting.
|
||||
|
||||
The constraints in [`../../CLAUDE.md`](../../CLAUDE.md) still govern this track:
|
||||
one visible self-improving loop, demo stability, no broad platform rewrite, no
|
||||
dashboard-first product, and no overclaiming.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| [`spec.md`](spec.md) | Read-only data contract for runs, traces, strategies, and proposals |
|
||||
| [`policy.md`](policy.md) | Promotion, rejection, evidence, and safety rules |
|
||||
| [`prompt.md`](prompt.md) | Evaluator prompt for narrow strategy improvements |
|
||||
| [`plan.md`](plan.md) | v1 implementation order if this track is added |
|
||||
|
||||
## What Is Implemented Now
|
||||
|
||||
- Shared TypeScript contracts for `AgentRun`, `AgentTraceEvent`,
|
||||
`StrategyVersion`, and `LearningProposal`.
|
||||
- Exact signature recall and accepted/dismissed outcomes that can later feed
|
||||
strategy decisions.
|
||||
- Documentation of future collections and indexes.
|
||||
|
||||
## What Is Intentionally Cut
|
||||
|
||||
- Full autonomous strategy promotion.
|
||||
- Autonomous code rewriting.
|
||||
- Multi-agent strategy debates.
|
||||
- Claims that PodMan trains or rewrites itself from live usage today.
|
||||
|
||||
## Demo Proof Path
|
||||
|
||||
Observe screen/git state -> detect collision -> send intervention -> accept or
|
||||
dismiss outcome -> recall similar event -> show changed graph or changed
|
||||
behavior. In the current demo, this proof is team-memory learning; agent
|
||||
strategy promotion remains planned unless records are added.
|
||||
@@ -1,88 +0,0 @@
|
||||
# Agent Learning Plan
|
||||
|
||||
Status: planned / narrow v1
|
||||
Goal: ship a visible recursive self-improvement loop without overbuilding
|
||||
|
||||
## Must-Have
|
||||
|
||||
1. Store agent runs.
|
||||
2. Store trace summaries.
|
||||
3. Store active and candidate strategy versions.
|
||||
4. Attach verifier or outcome evidence.
|
||||
5. Show one strategy improvement in the demo narrative.
|
||||
|
||||
## Build Order
|
||||
|
||||
### R1: Trace the run
|
||||
|
||||
Write one `agent_runs` record for an important coordination decision and append
|
||||
trace events for:
|
||||
|
||||
- observation
|
||||
- recall
|
||||
- prediction
|
||||
- intervention
|
||||
- outcome
|
||||
- adaptation
|
||||
|
||||
### R2: Version the strategy
|
||||
|
||||
Create an active strategy version for one of:
|
||||
|
||||
- collision detector threshold
|
||||
- intervention routing
|
||||
- graph discovery filter
|
||||
- card wording prompt
|
||||
|
||||
### R3: Score the outcome
|
||||
|
||||
Use the simplest verifier:
|
||||
|
||||
- accepted real collision = useful
|
||||
- dismissed = noisy
|
||||
- no response after cooldown = uncertain
|
||||
|
||||
### R4: Propose a narrow change
|
||||
|
||||
Examples:
|
||||
|
||||
- "For this exact signature, prefer sync PR card."
|
||||
- "For dismissed docs-only overlaps, suppress voice escalation."
|
||||
- "For repeated auth.ts collisions, raise severity."
|
||||
|
||||
### R5: Promote or reject
|
||||
|
||||
Promote only when evidence is strong enough. Otherwise keep the candidate as
|
||||
rejected or open.
|
||||
|
||||
## Demo Path
|
||||
|
||||
1. Show baseline strategy.
|
||||
2. Trigger a collision.
|
||||
3. Accept or dismiss the intervention.
|
||||
4. Store outcome.
|
||||
5. Show a candidate strategy update.
|
||||
6. Promote it.
|
||||
7. Trigger a similar event.
|
||||
8. Show changed behavior.
|
||||
|
||||
## Nice-to-Have
|
||||
|
||||
- Strategy comparison panel.
|
||||
- Model-generated prompt patch with verifier.
|
||||
- Vector recall over strategy history.
|
||||
- Rollback UI.
|
||||
|
||||
## Cut
|
||||
|
||||
- Full autonomous code rewriting.
|
||||
- Multi-agent strategy debates.
|
||||
- Long-term benchmark suite.
|
||||
- Training a model.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- The demo can point to a MongoDB record proving the agent changed behavior.
|
||||
- The changed behavior is visible.
|
||||
- The strategy has a parent and evidence.
|
||||
- Rejected or failed changes are not deleted.
|
||||
@@ -1,83 +0,0 @@
|
||||
# Agent Learning Policy
|
||||
|
||||
Status: planned / narrow v1
|
||||
Scope: guardrails for recursive self-improvement
|
||||
|
||||
## Prime Rule
|
||||
|
||||
PodMan may improve its agent behavior only when the improvement is narrow,
|
||||
evidence-backed, versioned, and reversible.
|
||||
|
||||
## Allowed Learning
|
||||
|
||||
PodMan may learn:
|
||||
|
||||
- Which prompt version produces clearer interventions.
|
||||
- Which detector threshold reduces false positives.
|
||||
- Which routing channel gets accepted without being intrusive.
|
||||
- Which verifier best predicts user acceptance.
|
||||
- Which graph-discovery rule produces cleaner risk paths.
|
||||
|
||||
## Disallowed Learning
|
||||
|
||||
PodMan must not:
|
||||
|
||||
- Promote a strategy because the model says it is better.
|
||||
- Rewrite broad system behavior from one example.
|
||||
- Hide failures, dismissals, or rejected candidates.
|
||||
- Learn from raw screenshots, secrets, or private terminal content.
|
||||
- Turn voice into the default route.
|
||||
- Create irreversible actions without human approval.
|
||||
|
||||
## Promotion Rules
|
||||
|
||||
A candidate strategy can become active only when all are true:
|
||||
|
||||
1. It has a parent strategy version.
|
||||
2. It describes one concrete behavior change.
|
||||
3. It has a verifier plan.
|
||||
4. It has evidence from a run, outcome, or test.
|
||||
5. It improves or fixes the target metric.
|
||||
6. It does not increase user interruption without payoff.
|
||||
|
||||
## Rejection Rules
|
||||
|
||||
Reject and retain the candidate when:
|
||||
|
||||
- The verifier regresses.
|
||||
- The change is too broad.
|
||||
- The evidence is missing.
|
||||
- The candidate conflicts with privacy rules.
|
||||
- The candidate makes the demo less stable.
|
||||
|
||||
## Evidence Strength
|
||||
|
||||
| Evidence | Strength | Use |
|
||||
| --- | --- | --- |
|
||||
| Model opinion | Weak | Proposal only |
|
||||
| Trace observation | Medium | Candidate rationale |
|
||||
| Human accepted outcome | Strong | Promotion candidate |
|
||||
| Human dismissed outcome | Strong | Suppression or rejection |
|
||||
| Automated verifier | Strong | Promotion or rejection |
|
||||
| Repeated accepted exact signature | Strong | Policy confidence increase |
|
||||
|
||||
## Versioning Rules
|
||||
|
||||
- Strategy versions are immutable after promotion or rejection.
|
||||
- There is one active version per `podId + kind`.
|
||||
- A rollback activates the previous version; it does not edit history.
|
||||
- Parent-child lineage must be preserved.
|
||||
|
||||
## Safety Rules
|
||||
|
||||
- Store summaries, not raw sensitive content.
|
||||
- Prefer deterministic checks over model judgment.
|
||||
- Use exact MongoDB recall before vector recall.
|
||||
- Ask for approval before changing code or data with external effects.
|
||||
- Treat hackathon demo stability as a hard constraint.
|
||||
|
||||
## Demo Honesty
|
||||
|
||||
Seeded strategy versions are acceptable when labeled as demo-backed. Do not claim
|
||||
a strategy was learned live unless a run and outcome actually created the
|
||||
promotion evidence.
|
||||
@@ -1,74 +0,0 @@
|
||||
# Agent Learning Prompt
|
||||
|
||||
Use this prompt for an agent responsible for improving PodMan's own behavior.
|
||||
|
||||
## Prompt
|
||||
|
||||
You are PodMan's agent-learning evaluator.
|
||||
|
||||
Your job is to inspect a completed agent run, identify one narrow improvement,
|
||||
define how to verify it, and decide whether to propose, promote, or reject a
|
||||
strategy change.
|
||||
|
||||
You must not claim improvement without evidence. You must not propose broad
|
||||
rewrites. Keep every change small, reversible, and tied to a run or outcome.
|
||||
|
||||
## Inputs
|
||||
|
||||
- Current active strategy version.
|
||||
- Agent run summary.
|
||||
- Trace events.
|
||||
- Intervention outcome.
|
||||
- Verifier result.
|
||||
- Recent false positives or accepted events.
|
||||
- Current demo constraints.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. Identify the target behavior.
|
||||
2. Identify the failure or success evidence.
|
||||
3. Decide whether a strategy change is warranted.
|
||||
4. Propose one narrow change.
|
||||
5. Define the verifier.
|
||||
6. Decide status: no change, candidate, promote, reject.
|
||||
7. Write a short explanation suitable for the Team memory activity stream.
|
||||
|
||||
## Output Format
|
||||
|
||||
```text
|
||||
Target
|
||||
- Strategy kind:
|
||||
- Active version:
|
||||
- Behavior under review:
|
||||
|
||||
Evidence
|
||||
- Run:
|
||||
- Outcome:
|
||||
- Verifier:
|
||||
- Confidence:
|
||||
|
||||
Decision
|
||||
- Status:
|
||||
- Proposed change:
|
||||
- Why this is narrow:
|
||||
- Risk:
|
||||
|
||||
Verifier
|
||||
- Metric:
|
||||
- Passing condition:
|
||||
- Failing condition:
|
||||
|
||||
Memory Write
|
||||
- Collection:
|
||||
- Record summary:
|
||||
- Graph/activity summary:
|
||||
```
|
||||
|
||||
## Hard Rules
|
||||
|
||||
- Exact outcomes beat model opinion.
|
||||
- Rejected candidates stay in memory.
|
||||
- No raw screenshots or secrets.
|
||||
- No broad policy change from one weak signal.
|
||||
- No voice-first behavior.
|
||||
|
||||
@@ -1,198 +0,0 @@
|
||||
# Agent Learning Spec
|
||||
|
||||
Status: planned / narrow v1
|
||||
Scope: how PodMan agents improve their own prompts, policies, detectors, and routing behavior
|
||||
Owner: agent learning / recursive self-improvement
|
||||
|
||||
## Purpose
|
||||
|
||||
Agent learning is the recursive self-improvement layer. It is not the same as
|
||||
team memory. Team memory learns about engineers and work. Agent learning learns
|
||||
which agent strategies produce better outcomes.
|
||||
|
||||
The demo claim:
|
||||
|
||||
1. PodMan tries a coordination strategy.
|
||||
2. The run is traced in MongoDB.
|
||||
3. A verifier or human outcome scores it.
|
||||
4. Gemini or another agent proposes a narrow strategy change.
|
||||
5. The new strategy is versioned.
|
||||
6. A later run uses the improved strategy and shows a better result.
|
||||
|
||||
## What Is Implemented Now
|
||||
|
||||
- Shared TypeScript record shapes exist for the core objects below.
|
||||
- Exact signature recall and accepted/dismissed outcomes exist in the team
|
||||
memory loop.
|
||||
- No write path currently promotes autonomous strategy changes.
|
||||
|
||||
## What Is Intentionally Cut
|
||||
|
||||
- Autonomous code rewriting.
|
||||
- Full autonomous strategy promotion.
|
||||
- Multi-agent strategy debates.
|
||||
- Claims that model self-evaluation alone can promote a strategy.
|
||||
|
||||
## Core Objects
|
||||
|
||||
### Agent run
|
||||
|
||||
One attempt to execute a goal.
|
||||
|
||||
```text
|
||||
agent_runs
|
||||
runId
|
||||
podId
|
||||
goal
|
||||
trigger
|
||||
strategyVersionId
|
||||
status
|
||||
startedAt
|
||||
completedAt
|
||||
score
|
||||
verifierSummary
|
||||
inputRefs
|
||||
outputRefs
|
||||
```
|
||||
|
||||
Allowed `status` values:
|
||||
|
||||
```text
|
||||
running, succeeded, failed, improved, regressed, abandoned
|
||||
```
|
||||
|
||||
### Trace event
|
||||
|
||||
Append-only event log for a run.
|
||||
|
||||
```text
|
||||
agent_trace_events
|
||||
runId
|
||||
podId
|
||||
step
|
||||
phase
|
||||
eventType
|
||||
inputSummary
|
||||
outputSummary
|
||||
toolName
|
||||
error
|
||||
metrics
|
||||
createdAt
|
||||
```
|
||||
|
||||
### Strategy version
|
||||
|
||||
Versioned prompt, detector rule, policy, verifier, or routing strategy.
|
||||
|
||||
```text
|
||||
strategy_versions
|
||||
strategyVersionId
|
||||
podId
|
||||
kind
|
||||
name
|
||||
parentVersionId
|
||||
status
|
||||
summary
|
||||
promptText
|
||||
policy
|
||||
verifier
|
||||
metrics
|
||||
createdAt
|
||||
promotedAt
|
||||
```
|
||||
|
||||
Allowed `kind` values:
|
||||
|
||||
```text
|
||||
prompt, policy, detector, verifier, routing
|
||||
```
|
||||
|
||||
Allowed `status` values:
|
||||
|
||||
```text
|
||||
candidate, active, retired, rejected
|
||||
```
|
||||
|
||||
### Learning proposal
|
||||
|
||||
A candidate change before promotion.
|
||||
|
||||
```text
|
||||
learning_proposals
|
||||
proposalId
|
||||
podId
|
||||
sourceRunId
|
||||
targetKind
|
||||
parentVersionId
|
||||
proposedChange
|
||||
rationale
|
||||
verifierPlan
|
||||
status
|
||||
createdAt
|
||||
resolvedAt
|
||||
```
|
||||
|
||||
Allowed `status` values:
|
||||
|
||||
```text
|
||||
open, accepted, rejected, superseded
|
||||
```
|
||||
|
||||
## MongoDB Indexes
|
||||
|
||||
| Collection | Index | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `agent_runs` | `{ podId: 1, startedAt: -1 }` | Recent run history |
|
||||
| `agent_runs` | `{ podId: 1, strategyVersionId: 1 }` | Compare strategy performance |
|
||||
| `agent_trace_events` | `{ runId: 1, step: 1 }` | Reconstruct run |
|
||||
| `strategy_versions` | `{ podId: 1, kind: 1, status: 1 }` | Find active strategy |
|
||||
| `strategy_versions` | `{ podId: 1, createdAt: -1 }` | Version history |
|
||||
| `learning_proposals` | `{ podId: 1, status: 1 }` | Open candidate changes |
|
||||
|
||||
## Learning Loop
|
||||
|
||||
```text
|
||||
observe run -> score run -> propose change -> test candidate -> promote or reject
|
||||
```
|
||||
|
||||
Agent learning must always connect these records:
|
||||
|
||||
```text
|
||||
agent_run -> trace_events -> verifier result -> learning_proposal -> strategy_version
|
||||
```
|
||||
|
||||
## Verifier Contract
|
||||
|
||||
Every promoted strategy needs a verifier signal.
|
||||
|
||||
Allowed verifier types:
|
||||
|
||||
- Human accepted or dismissed outcome.
|
||||
- Test pass or fail result.
|
||||
- Reduced false positive rate.
|
||||
- Reduced intervention count with same or better accepted outcomes.
|
||||
- Faster successful run.
|
||||
- Better graph discovery precision.
|
||||
- Explicit demo operator approval.
|
||||
|
||||
Self-evaluation alone is not enough to promote a strategy.
|
||||
|
||||
## Relationship to Team Graph
|
||||
|
||||
Agent learning can appear in the Team memory graph as activity and loop status,
|
||||
but it should not clutter the main risk graph by default.
|
||||
|
||||
Graph discovery may show:
|
||||
|
||||
- `agent_run` activity in the stream.
|
||||
- `strategy_versions` count in the learning loop.
|
||||
- A selected-node detail saying a policy changed because a prior outcome was
|
||||
dismissed or accepted.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Every strategy change has a parent.
|
||||
- Every promoted strategy cites evidence.
|
||||
- Rejected strategies are retained with a reason.
|
||||
- Agent traces are append-only.
|
||||
- The system can answer: "What changed, why, and did it help?"
|
||||
@@ -2,9 +2,8 @@
|
||||
|
||||
> Owner: graph data + visualization. Status: demo-backed / active.
|
||||
> Satisfies the documentation-first gate for the `backend/src/graph/*` and
|
||||
> `frontend/src/components/GraphView.tsx` files.
|
||||
>
|
||||
> Canonical module docs live in [`docs/graph-discovery/`](graph-discovery/).
|
||||
> `frontend/src/components/GraphView.tsx` files. This file is the canonical
|
||||
> graph spec.
|
||||
|
||||
## What this is (and is NOT)
|
||||
|
||||
@@ -1,37 +0,0 @@
|
||||
# Continual Learning
|
||||
|
||||
Status: demo-backed / active
|
||||
|
||||
PodMan's continual-learning track owns team memory: what the system learns about
|
||||
files, collisions, interventions, outcomes, and future routing for a pod.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| [`spec.md`](spec.md) | Data model and observe/store/predict/outcome/adapt loop |
|
||||
| [`policy.md`](policy.md) | What PodMan may and may not remember |
|
||||
| [`prompt.md`](prompt.md) | Memory-agent prompt for outcome-backed learning |
|
||||
| [`plan.md`](plan.md) | Demo build order and acceptance criteria |
|
||||
|
||||
## What Is Implemented Now
|
||||
|
||||
- MongoDB-backed `observations`, `collisions`, `interventions`, `outcomes`,
|
||||
`engineer_states`, and `team_model` records.
|
||||
- Exact signature recall for prior accepted and dismissed outcomes.
|
||||
- Outcome writes through `POST /api/outcome`.
|
||||
- Team memory graph edges from accepted real outcomes.
|
||||
- No raw screenshots or recordings are stored.
|
||||
|
||||
## What Is Intentionally Cut
|
||||
|
||||
- Autonomous model training.
|
||||
- Broad cross-pod generalization.
|
||||
- Raw screen capture retention.
|
||||
- Vector recall as a dependency for the demo proof.
|
||||
|
||||
## Demo Proof Path
|
||||
|
||||
Observe screen/git state -> detect collision -> send intervention -> accept or
|
||||
dismiss outcome -> recall similar event -> show changed graph or changed
|
||||
behavior.
|
||||
@@ -1,68 +0,0 @@
|
||||
# Continual Learning Plan
|
||||
|
||||
Status: demo-backed / active
|
||||
Goal: prove PodMan learns from outcomes in the hackathon demo
|
||||
|
||||
## Must-Have Demo Loop
|
||||
|
||||
1. Observe two engineers touching the same file.
|
||||
2. Store the observation and git state in MongoDB.
|
||||
3. Predict a collision.
|
||||
4. Send a card or Hermes message.
|
||||
5. Record accept or dismiss outcome.
|
||||
6. Adapt `team_model`.
|
||||
7. Show the learned graph edge or changed future behavior.
|
||||
|
||||
## Build Order
|
||||
|
||||
### R1: Make exact recall reliable
|
||||
|
||||
- Normalize file paths.
|
||||
- Build stable memory signatures.
|
||||
- Look up prior accepted and dismissed outcomes.
|
||||
- Prefer exact recall over vector recall.
|
||||
|
||||
### R2: Make outcomes update memory
|
||||
|
||||
- Accepted real collision creates or strengthens ownership.
|
||||
- Accepted real collision creates `learned_from`.
|
||||
- Dismissed outcome lowers confidence or suppresses route.
|
||||
|
||||
### R3: Expose loop data to the graph
|
||||
|
||||
- Add optional loop snapshot.
|
||||
- Add optional activity stream.
|
||||
- Keep existing `PodGraph` fields stable.
|
||||
|
||||
### R4: Show the observatory
|
||||
|
||||
- Render observe/store/predict/outcome/adapt.
|
||||
- Show recent activity.
|
||||
- Make selected-node detail explain why memory changed.
|
||||
|
||||
### R5: Prepare a clean demo chain
|
||||
|
||||
- Ensure one collision -> intervention -> accepted outcome exists.
|
||||
- Ensure repeated signature recalls prior memory.
|
||||
- Verify graph shows learned ownership.
|
||||
|
||||
## Nice-to-Have
|
||||
|
||||
- Atlas Vector Search over memory summaries.
|
||||
- Confidence scoring per ownership edge.
|
||||
- Per-file memory timeline.
|
||||
- Strategy promotion tied to outcomes.
|
||||
|
||||
## Cut
|
||||
|
||||
- Raw screenshot storage.
|
||||
- Full autonomous training.
|
||||
- Broad dashboard metrics.
|
||||
- Multi-pod learning generalization.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A judge can see what changed in memory.
|
||||
- The second similar event behaves differently.
|
||||
- Exact MongoDB records prove the loop.
|
||||
- The graph remains legible with real data.
|
||||
@@ -1,96 +0,0 @@
|
||||
# Continual Learning Policy
|
||||
|
||||
Status: demo-backed / active
|
||||
Scope: what PodMan may learn about a team
|
||||
|
||||
## Prime Rule
|
||||
|
||||
PodMan learns coordination patterns, not personal surveillance profiles.
|
||||
|
||||
## Allowed Memory
|
||||
|
||||
PodMan may store:
|
||||
|
||||
- File and symbol ownership.
|
||||
- Active file overlap.
|
||||
- Repeated collision signatures.
|
||||
- Intervention history.
|
||||
- Accepted and dismissed outcomes.
|
||||
- Routing preferences by event type and severity.
|
||||
- Summaries of decisions relevant to future coordination.
|
||||
|
||||
## Forbidden Memory
|
||||
|
||||
PodMan must not store:
|
||||
|
||||
- Raw screenshots.
|
||||
- Screen recordings.
|
||||
- Secrets or credentials.
|
||||
- Full terminal logs.
|
||||
- Personal performance judgments.
|
||||
- Private content unrelated to the coding task.
|
||||
|
||||
## Evidence Policy
|
||||
|
||||
| Evidence | Can predict? | Can adapt memory? |
|
||||
| --- | --- | --- |
|
||||
| Vision only | Yes, low confidence | No |
|
||||
| Git watcher | Yes | No, unless repeated |
|
||||
| GitHub state | Yes | No, unless verified |
|
||||
| Accepted real outcome | Yes | Yes |
|
||||
| Dismissed outcome | Yes, for suppression | Yes, as negative signal |
|
||||
| Verifier result | Yes | Yes |
|
||||
|
||||
## Intervention Policy
|
||||
|
||||
Use the least intrusive channel:
|
||||
|
||||
1. Watch quietly.
|
||||
2. Card.
|
||||
3. Hermes message.
|
||||
4. Voice.
|
||||
|
||||
Voice is only for urgent, high-confidence, time-sensitive risks.
|
||||
|
||||
## Adaptation Policy
|
||||
|
||||
Allowed adaptations:
|
||||
|
||||
- Add learned ownership after accepted real outcome.
|
||||
- Raise confidence for repeated accepted signatures.
|
||||
- Lower confidence for dismissed signatures.
|
||||
- Prefer the previously accepted intervention kind.
|
||||
- Suppress repeated low-value warnings.
|
||||
|
||||
Disallowed adaptations:
|
||||
|
||||
- Broad threshold changes from one example.
|
||||
- Treating vector similarity as proof.
|
||||
- Hiding dismissals.
|
||||
- Making interruption more aggressive without evidence.
|
||||
|
||||
## Retention Policy
|
||||
|
||||
Keep:
|
||||
|
||||
- Outcomes.
|
||||
- Signatures.
|
||||
- Team model memory.
|
||||
- Strategy metrics.
|
||||
|
||||
Summarize or expire:
|
||||
|
||||
- Old observations.
|
||||
- Low-confidence vision-only events.
|
||||
- Detailed trace text.
|
||||
|
||||
Delete immediately:
|
||||
|
||||
- Secrets.
|
||||
- Accidental raw sensitive captures.
|
||||
|
||||
## Demo Policy
|
||||
|
||||
Seeded data is acceptable only if the demo script is honest about it. Live
|
||||
learning requires a live or staged outcome write that visibly updates the graph
|
||||
or future decision.
|
||||
@@ -1,87 +0,0 @@
|
||||
# Continual Learning Prompt
|
||||
|
||||
Use this prompt for the agent that decides what PodMan should remember from a
|
||||
coordination event.
|
||||
|
||||
## Prompt
|
||||
|
||||
You are PodMan's continual-learning memory agent.
|
||||
|
||||
Your job is to inspect observations, collisions, interventions, and outcomes,
|
||||
then decide what team memory should be updated. You must separate observed
|
||||
facts, inferred risks, human outcomes, and durable learned memory.
|
||||
|
||||
Do not claim something was learned unless an accepted real outcome, verifier, or
|
||||
human label supports it.
|
||||
|
||||
## Inputs
|
||||
|
||||
- Pod id.
|
||||
- Recent engineer states.
|
||||
- Recent observations.
|
||||
- Candidate collision.
|
||||
- Prior exact-signature memory.
|
||||
- Intervention record.
|
||||
- Outcome record.
|
||||
- Current team model.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. Normalize file and symbol.
|
||||
2. Build exact signature.
|
||||
3. Check prior accepted and dismissed outcomes.
|
||||
4. Classify the current event.
|
||||
5. Decide whether memory should change.
|
||||
6. Emit the graph impact.
|
||||
7. Write a short explanation.
|
||||
|
||||
## Output Format
|
||||
|
||||
```text
|
||||
Event
|
||||
- Signature:
|
||||
- Engineers:
|
||||
- File:
|
||||
- Symbol:
|
||||
- Evidence:
|
||||
|
||||
Prior Memory
|
||||
- Accepted matches:
|
||||
- Dismissed matches:
|
||||
- Ownership:
|
||||
|
||||
Decision
|
||||
- Memory action:
|
||||
- Confidence:
|
||||
- Reason:
|
||||
|
||||
Graph Impact
|
||||
- Nodes:
|
||||
- Edges:
|
||||
- Activity text:
|
||||
|
||||
Safety
|
||||
- Sensitive data present:
|
||||
- Redaction needed:
|
||||
```
|
||||
|
||||
## Memory Actions
|
||||
|
||||
Allowed actions:
|
||||
|
||||
- no_change
|
||||
- strengthen_signature
|
||||
- weaken_signature
|
||||
- create_learned_owner
|
||||
- update_route_preference
|
||||
- suppress_signature
|
||||
- request_human_label
|
||||
|
||||
## Hard Rules
|
||||
|
||||
- Exact recall before vector recall.
|
||||
- Dismissals are learning signals.
|
||||
- `learned_from` requires accepted real outcome.
|
||||
- Store summaries, not raw screen content.
|
||||
- Prefer less intrusive future behavior when uncertain.
|
||||
|
||||
@@ -1,232 +0,0 @@
|
||||
# Continual Learning Spec
|
||||
|
||||
Status: demo-backed / active
|
||||
Scope: how PodMan learns team memory from live work and outcomes
|
||||
Owner: continual learning / Team memory
|
||||
|
||||
## Purpose
|
||||
|
||||
Continual learning is the product proof that PodMan gets more useful from use.
|
||||
It learns team-level coordination memory: ownership, repeated collisions,
|
||||
accepted interventions, dismissed noise, and preferred routing.
|
||||
|
||||
The visible loop:
|
||||
|
||||
```text
|
||||
observe -> store -> predict -> outcome -> adapt
|
||||
```
|
||||
|
||||
## What Is Implemented Now
|
||||
|
||||
- `observations`, `collisions`, `interventions`, `outcomes`,
|
||||
`engineer_states`, `team_model`, `graph_nodes`, and `graph_edges` are the
|
||||
current memory truth.
|
||||
- Exact signature recall and accepted/dismissed outcomes exist.
|
||||
- Accepted real outcomes can produce `learned_from` graph edges and ownership
|
||||
memory.
|
||||
- Raw screenshots and recordings are not stored.
|
||||
|
||||
## What Is Intentionally Cut
|
||||
|
||||
- Full autonomous training.
|
||||
- Broad threshold changes from one example.
|
||||
- Making vector search required for the demo learning proof.
|
||||
|
||||
## Source Collections
|
||||
|
||||
### `engineer_states`
|
||||
|
||||
Latest per-engineer state from vision and local git.
|
||||
|
||||
Key fields:
|
||||
|
||||
- `podId`
|
||||
- `name`
|
||||
- `currentFile`
|
||||
- `changedFiles`
|
||||
- `branch`
|
||||
- `confidence`
|
||||
- `visionUpdatedAt`
|
||||
- `gitUpdatedAt`
|
||||
- `updatedAt`
|
||||
|
||||
### `observations`
|
||||
|
||||
Structured perception events.
|
||||
|
||||
Key fields:
|
||||
|
||||
- `podId`
|
||||
- `engineerId`
|
||||
- `currentFile`
|
||||
- `symbol`
|
||||
- `activity`
|
||||
- `confidence`
|
||||
- `observedAt`
|
||||
|
||||
### `collisions`
|
||||
|
||||
Predicted risk events.
|
||||
|
||||
Key fields:
|
||||
|
||||
- `id`
|
||||
- `podId`
|
||||
- `file`
|
||||
- `symbol`
|
||||
- `engineers`
|
||||
- `severity`
|
||||
- `status`
|
||||
- `memorySignature`
|
||||
- `detectedAt`
|
||||
|
||||
### `interventions`
|
||||
|
||||
Actions PodMan sent or suggested.
|
||||
|
||||
Key fields:
|
||||
|
||||
- `id`
|
||||
- `podId`
|
||||
- `collisionId`
|
||||
- `kind`
|
||||
- `channel`
|
||||
- `message`
|
||||
- `suggestedAction`
|
||||
- `createdAt`
|
||||
|
||||
### `outcomes`
|
||||
|
||||
Human or verifier supervision.
|
||||
|
||||
Key fields:
|
||||
|
||||
- `id`
|
||||
- `podId`
|
||||
- `interventionId`
|
||||
- `collisionId`
|
||||
- `accepted`
|
||||
- `wasRealCollision`
|
||||
- `learnedOwner`
|
||||
- `recordedAt`
|
||||
|
||||
### `team_model`
|
||||
|
||||
Durable pod memory.
|
||||
|
||||
Key fields:
|
||||
|
||||
- `podId`
|
||||
- `graph`
|
||||
- `ownership`
|
||||
- `collisionSignatures`
|
||||
- `interventionPolicy`
|
||||
- `updatedAt`
|
||||
|
||||
### `memory_vectors`
|
||||
|
||||
Optional semantic recall. Exact recall comes first.
|
||||
|
||||
Key fields:
|
||||
|
||||
- `podId`
|
||||
- `sourceKind`
|
||||
- `sourceId`
|
||||
- `text`
|
||||
- `embedding`
|
||||
- `embeddingModel`
|
||||
- `tags`
|
||||
|
||||
## Learning Rules
|
||||
|
||||
### Observe
|
||||
|
||||
Write structured evidence from vision, git, GitHub, and agent traces.
|
||||
|
||||
### Store
|
||||
|
||||
Persist source records and materialized summaries. Do not store raw screenshots
|
||||
or recordings.
|
||||
|
||||
### Predict
|
||||
|
||||
Create a collision when multiple engineers converge on the same normalized file
|
||||
or symbol and at least one signal shows active or unpushed work.
|
||||
|
||||
### Outcome
|
||||
|
||||
Record whether the intervention was accepted, dismissed, real, or false.
|
||||
|
||||
### Adapt
|
||||
|
||||
Only accepted real outcomes can create `learned_from` graph edges. Dismissals
|
||||
adapt suppression, routing, or confidence.
|
||||
|
||||
## Exact Signature
|
||||
|
||||
Use deterministic signatures:
|
||||
|
||||
```text
|
||||
podId:eventType:normalizedFile:symbol:sortedEngineers
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Sort engineer names.
|
||||
- Normalize file paths.
|
||||
- Use `*` for missing symbol.
|
||||
- Never include timestamps.
|
||||
|
||||
## UI-Facing Loop Snapshot
|
||||
|
||||
The graph response may include:
|
||||
|
||||
```text
|
||||
loop
|
||||
activeStep
|
||||
steps[]
|
||||
key
|
||||
label
|
||||
value
|
||||
detail
|
||||
status
|
||||
```
|
||||
|
||||
Step mapping:
|
||||
|
||||
| Step | Source |
|
||||
| --- | --- |
|
||||
| Observe | recent observations and git updates |
|
||||
| Store | team model, graph records, memory vectors |
|
||||
| Predict | open collisions |
|
||||
| Outcome | accepted and dismissed outcomes |
|
||||
| Adapt | learned owners, learned edges, strategy changes |
|
||||
|
||||
## Activity Stream
|
||||
|
||||
The graph response may include:
|
||||
|
||||
```text
|
||||
activity[]
|
||||
id
|
||||
at
|
||||
kind
|
||||
title
|
||||
detail
|
||||
nodeId
|
||||
edgeId
|
||||
```
|
||||
|
||||
Allowed `kind` values:
|
||||
|
||||
```text
|
||||
editing, collision, intervention, outcome, learned, agent
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- The system can show one accepted outcome changing future memory.
|
||||
- Exact recall works without vector search.
|
||||
- The Team memory graph can explain the learning loop.
|
||||
- Dismissals and false positives are retained.
|
||||
- The demo does not rely on raw screenshots or hidden state.
|
||||
@@ -1,108 +0,0 @@
|
||||
# Demo Setup
|
||||
|
||||
Pre-stage checklist for the 3-minute live demo. Do this on all 3 laptops before walking on stage.
|
||||
|
||||
---
|
||||
|
||||
## Before demo day
|
||||
|
||||
- [ ] `demo-pod` room created in LiveKit Cloud dashboard
|
||||
- [ ] Hermes deployed on DO (or confirmed running locally as fallback)
|
||||
- [ ] MongoDB Atlas cluster running, `MONGODB_URI` set in Hermes env
|
||||
- [ ] All `.env` vars populated and verified via `GET /health` returning `{ ok: true }`
|
||||
- [ ] Record a backup video of the full demo working end-to-end
|
||||
- [ ] Rehearse the demo script 3× with real audio
|
||||
|
||||
---
|
||||
|
||||
## Laptop setup (all 3 machines)
|
||||
|
||||
### Editor settings
|
||||
|
||||
- Font size: **18pt or larger** — Gemini Vision must read file names and code
|
||||
- Single editor window — no split panes, no overlapping terminals
|
||||
- File tab visible with full file name shown (not truncated)
|
||||
- Light or dark theme is fine — avoid low-contrast themes
|
||||
|
||||
### Browser
|
||||
|
||||
- Chrome (best `getDisplayMedia` support)
|
||||
- PWA tab open and joined to `demo-pod`
|
||||
- Earbuds / headphones plugged in and tested
|
||||
- Volume: medium — PodMan voice should be clearly audible but not startle
|
||||
|
||||
### Screen layout
|
||||
|
||||
- Editor takes 2/3 of screen
|
||||
- Terminal takes bottom 1/3 (always visible)
|
||||
- No other windows on top
|
||||
|
||||
---
|
||||
|
||||
## Demo file setup
|
||||
|
||||
Pre-create these files in the demo repo before the demo:
|
||||
|
||||
**Alice's machine:**
|
||||
|
||||
- Open `auth/middleware.ts` — has visible function stubs
|
||||
- Terminal shows nothing running initially, then `Server running on :3001` at the right moment
|
||||
|
||||
**Bob's machine:**
|
||||
|
||||
- Open `frontend/login.tsx` — has visible form component code
|
||||
- Terminal idle
|
||||
|
||||
**Carol's machine:**
|
||||
|
||||
- Open `frontend/integration.ts` or similar
|
||||
- Terminal shows: `curl http://localhost:3001/auth` → `curl: (7) Failed to connect`
|
||||
|
||||
---
|
||||
|
||||
## Demo script timing
|
||||
|
||||
| Time | Action | Who |
|
||||
| ----- | ----------------------------------------------- | ----------- |
|
||||
| 0:00 | All three join `demo-pod` | All |
|
||||
| 0:05 | PodMan greets by voice | Hermes auto |
|
||||
| 0:20 | Alice opens `auth/middleware.ts`, starts typing | Alice |
|
||||
| 0:45 | Bob opens `frontend/login.tsx` | Bob |
|
||||
| 0:50 | Carol runs `curl` command, sees error | Carol |
|
||||
| ~1:20 | BLOCKER_DETECTED intervention fires | Hermes auto |
|
||||
| 1:50 | Alice starts her server (`node server.js`) | Alice |
|
||||
| ~2:00 | DEPENDENCY_READY intervention fires | Hermes auto |
|
||||
| 2:20 | Optional: show session 2 ownership warm-start | Presenter |
|
||||
| 2:45 | Close | Presenter |
|
||||
|
||||
---
|
||||
|
||||
## Gemini Vision reliability tips
|
||||
|
||||
- Keep font at 18pt+ throughout the demo — do not zoom out
|
||||
- Avoid opening file picker dialogs or overlapping modals during the demo
|
||||
- File names in editor tabs must be fully visible (not `auth/middle...`)
|
||||
- If Hermes logs show `confidence < 0.6` frames: bump font size, ensure file tab is clear
|
||||
- Terminal output must be on a single line — avoid long stack traces during demo
|
||||
|
||||
---
|
||||
|
||||
## Cooldown note
|
||||
|
||||
Hermes has a 3-minute cooldown between urgent voice cues per pod. For the demo, if you need to trigger a second urgent voice event quickly:
|
||||
|
||||
Option 1: restart Hermes between the two demo scenarios (resets cooldown state)
|
||||
Option 2: set `NUDGE_COOLDOWN_MS=0` via env var during demo (add this override to Hermes)
|
||||
|
||||
---
|
||||
|
||||
## Fallback plan
|
||||
|
||||
If any system fails on stage:
|
||||
|
||||
1. **Hermes unreachable:** switch to local (`pnpm --filter backend dev`) — PWA auto-falls back to `localhost:8787`
|
||||
2. **Gemini Vision low confidence:** presenter narrates what PodMan "saw" while playing the backup video
|
||||
3. **LiveKit audio not working:** play backup video — show the intervention cards on screen instead
|
||||
4. **Full system failure:** play the backup recording, narrate the demo live
|
||||
|
||||
Always have the backup video on a separate device, not the same laptop running Hermes.
|
||||
+141
@@ -0,0 +1,141 @@
|
||||
# PodMan — 4-Minute Demo Script
|
||||
|
||||
**Theme:** Continual Learning. **Hard limit:** 4:00. Practice to land at 3:45.
|
||||
|
||||
**The one-line story:** writing code isn't the bottleneck anymore — *coordinating
|
||||
who's writing what* is. PodMan is a pair programmer for the whole team: it watches
|
||||
every member's work in real time, gives everyone live status without anyone having
|
||||
to interrupt anyone, and learns your team's dynamics so it nudges less and helps
|
||||
more over time.
|
||||
|
||||
**The hook to land:** a "quick five-minute question" actually costs ~25 minutes of
|
||||
lost focus — for two people. PodMan removes the reason to ask. Multiply the saved
|
||||
recovery time across every teammate, every day, and that is the value.
|
||||
|
||||
---
|
||||
|
||||
## The script (4:00)
|
||||
|
||||
### 0:00–0:30 — The problem + hook
|
||||
|
||||
> "AI made writing code easy. The thing still slowing teams down is coordination
|
||||
> — checking each other's work, re-planning collisions, and the constant 'what
|
||||
> are you working on?' A five-minute question really costs both people 25 minutes
|
||||
> of lost focus. PodMan is a pair programmer for the whole team: it watches
|
||||
> everyone's work live, so anyone can see another's status without interrupting
|
||||
> them — and it learns your team as it goes."
|
||||
|
||||
*On screen:* the pod view, two teammates joined, screen-share tiles live.
|
||||
|
||||
### 0:30–1:05 — Real-time team awareness (LiveKit + Gemini Vision)
|
||||
|
||||
- Point at the two live screen tiles. "These are real screen shares over
|
||||
**LiveKit**. Our agent subscribes to the tracks and samples frames."
|
||||
- "Each frame goes to **Gemini Vision**, which returns structured context — file,
|
||||
symbol, activity — not a chatbot, a perception layer."
|
||||
- Show the live activity stream filling in (Signals vs Reasoning sections).
|
||||
- Land the value: "This is the part that replaces 'what are you working on?' —
|
||||
every teammate's current work is just *visible*, in real time. Nobody had to
|
||||
ask."
|
||||
|
||||
*Built-by-us callout:* `backend/src/vision/gemini.ts`, the LiveKit agent worker.
|
||||
|
||||
### 1:05–1:50 — The catch (detection + first intervention)
|
||||
|
||||
- Have alice and bob both edit the **same file** with unpushed changes.
|
||||
- "Normally nobody notices until merge time. GitHub can't see this — nothing's
|
||||
pushed. Our detector fuses live screen context with **local git truth** from a
|
||||
watcher on each laptop."
|
||||
- A collision card appears: *"alice + bob both on detector.ts (unpushed)."*
|
||||
- Let the **Gemini TTS** urgent voice fire once over LiveKit: *"alice and bob are
|
||||
both editing detector.ts. Please sync before pushing."*
|
||||
- Land the value: "That's a merge conflict and a wasted afternoon caught before it
|
||||
happened — and neither of them had to be tracking the other."
|
||||
|
||||
*Built-by-us callout:* `collision/detector.ts`, `action/hermes.ts`,
|
||||
`voice/live.ts`.
|
||||
|
||||
### 1:50–2:50 — Continual learning (the theme — the money shot)
|
||||
|
||||
This is the differentiator. Two beats, both from pre-seeded memory:
|
||||
|
||||
1. **It learned to stay quiet.** Trigger a pattern that was dismissed as a false
|
||||
alarm earlier. "Last session a teammate marked this kind of alert as not a
|
||||
real conflict. Watch — PodMan stays silent. No nagging." (No card fires.)
|
||||
2. **It learned to escalate.** Trigger the real-conflict pattern that was
|
||||
accepted before. The card now says **"Seen before."** and goes straight to
|
||||
the spoken urgent cue.
|
||||
|
||||
- "The only input was one accept/dismiss tap. No retraining, no labeling. This is
|
||||
**MongoDB Atlas vector search** recalling similar past events plus a policy
|
||||
that adapts on the recalled outcome."
|
||||
- Optional: show `/api/memory/stats` counts climbing — accumulated experience.
|
||||
|
||||
*Built-by-us callout:* `memory/vectors.ts` ($vectorSearch), `memory/policy.ts`
|
||||
(outcome-conditioned gate), `memory/store.ts`.
|
||||
|
||||
### 2:50–3:30 — The five-minute meeting, killed (Gemini Live API)
|
||||
|
||||
- Frame it: "Instead of breaking a teammate's focus to ask what they're up to,
|
||||
you ask PodMan."
|
||||
- Open the live voice conversation. Ask out loud: *"PodMan, what is everyone
|
||||
working on, and where is the collision detector implemented?"*
|
||||
- It answers with **real tool calls** — `search_repo`, git history, current
|
||||
collisions — not guesses.
|
||||
- "This is the **Gemini Live API**, streaming speech-to-speech over LiveKit, with
|
||||
custom function tools we wrote so it grounds every answer in the actual repo
|
||||
and live state. That's the status sync, answered in seconds, with zero recovery
|
||||
tax on anyone else."
|
||||
|
||||
*Built-by-us callout:* `agents/podman-live-conversation/agent.py`.
|
||||
|
||||
### 3:30–3:50 — Stack + close
|
||||
|
||||
- "All on **DigitalOcean** — static frontend, API, and agent workers, supervised
|
||||
by systemd. The ambient score is **Gemini Lyria** generated per pod through the
|
||||
Interactions API."
|
||||
- Close: "Engineering ability stopped being the bottleneck — coordination is.
|
||||
PodMan gives a whole team real-time awareness without the interruptions, catches
|
||||
collisions before they cost an afternoon, and learns each team's dynamics so it
|
||||
helps more over time. Saved focus, multiplied across every teammate. That's
|
||||
continual learning, shipped."
|
||||
|
||||
### 3:50–4:00 — Buffer / Q&A handoff
|
||||
|
||||
---
|
||||
|
||||
## Sponsor-prize coverage (say each at least once)
|
||||
|
||||
| Prize | Spoken moment | Segment |
|
||||
| --- | --- | --- |
|
||||
| **Gemini** | Vision perception, Live API agent w/ tools, TTS voice, Lyria score | 0:30, 1:05, 2:50, 3:30 |
|
||||
| **LiveKit** | "real screen shares over LiveKit", agent subscribes, TTS audio track, live voice | 0:30, 1:05, 3:30 |
|
||||
| **MongoDB** | "Atlas vector search recalling past events" | 1:50 |
|
||||
| **DigitalOcean** | "all on DigitalOcean, systemd-supervised workers" | 3:30 |
|
||||
|
||||
---
|
||||
|
||||
## If something breaks (live recovery)
|
||||
|
||||
| Failure | Recovery |
|
||||
| --- | --- |
|
||||
| Voice doesn't fire | Cut to the card; say the line aloud; cards are the default path anyway. |
|
||||
| Live conversation drops | Skip 2:50–3:30; lean longer on the learning beat. |
|
||||
| Collision won't trigger | Use the backup recording for that beat; keep narrating. |
|
||||
| Agent flapping | Pre-checked — but if so, `systemctl restart podman-platform-agent`. |
|
||||
|
||||
**Rule:** never debug on stage. Narrate, fall back to recording, keep moving.
|
||||
|
||||
---
|
||||
|
||||
## Tight timing summary
|
||||
|
||||
| Time | Beat |
|
||||
| --- | --- |
|
||||
| 0:00 | Problem (coordination cost) + hook + original-work line |
|
||||
| 0:30 | Real-time team awareness — LiveKit + Gemini Vision |
|
||||
| 1:05 | The catch — collision caught before merge |
|
||||
| 1:50 | **Continual learning — quiet + escalate** |
|
||||
| 2:50 | The five-minute meeting, killed — Gemini Live conversation |
|
||||
| 3:30 | DigitalOcean + Lyria + close |
|
||||
| 3:50 | Buffer |
|
||||
+17
-4
@@ -58,6 +58,13 @@ The mirror at `infra/.do/app.yaml` is kept identical for DO UI/import workflows.
|
||||
- No HTTP route and no HTTP health check
|
||||
- Default room: `POD_ROOM=demo-pod`
|
||||
|
||||
### Worker: live conversation agent (Python)
|
||||
|
||||
- Source: `agents/podman-live-conversation/`
|
||||
- The Gemini Live voice agent (`gemini-3.1-flash-live-preview`), run with `uv`.
|
||||
- On the droplet it runs as `podman-live-conversation-agent.service`
|
||||
(`infra/systemd/`). It is separate from the Node services and the TS agent.
|
||||
|
||||
---
|
||||
|
||||
## Required Runtime Environment
|
||||
@@ -66,11 +73,15 @@ The mirror at `infra/.do/app.yaml` is kept identical for DO UI/import workflows.
|
||||
LIVEKIT_URL=wss://your-livekit-server.livekit.cloud
|
||||
LIVEKIT_API_KEY=...
|
||||
LIVEKIT_API_SECRET=...
|
||||
LIVEKIT_CONVERSATION_AGENT_NAME=podman-live-conversation
|
||||
|
||||
GEMINI_API_KEY=...
|
||||
GEMINI_API_KEY=... # GOOGLE_API_KEY also accepted
|
||||
GEMINI_VISION_MODEL=gemini-2.0-flash
|
||||
GEMINI_LIVE_MODEL=gemini-3.1-flash-tts-preview
|
||||
GEMINI_LIVE_MODEL=gemini-3.1-flash-tts-preview # TTS voice
|
||||
GEMINI_CONVERSATION_MODEL=gemini-3.1-flash-live-preview
|
||||
GEMINI_EMBEDDING_MODEL=gemini-embedding-001
|
||||
GEMINI_TTS_VOICE=Charon
|
||||
# GEMINI_MUSIC_MODEL=lyria-3-clip-preview # optional override
|
||||
|
||||
GITHUB_TOKEN=...
|
||||
GITHUB_REPO=karti-ai/podman
|
||||
@@ -83,8 +94,10 @@ PORT=8787
|
||||
POD_ROOM=demo-pod
|
||||
```
|
||||
|
||||
`VOYAGE_API_KEY` is optional for local/demo fallback. Without it, Mongo exact
|
||||
signature recall still works; Atlas Vector Search recall is skipped.
|
||||
`VOYAGE_API_KEY` is optional. Without it, Gemini embeddings provide vector
|
||||
recall; without any embedding provider, recall degrades to exact signature
|
||||
matching. The Lyria background score uses the Gemini Interactions API and the
|
||||
same `GEMINI_API_KEY`.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+77
-97
@@ -1,138 +1,118 @@
|
||||
# Gemini Integration Spec
|
||||
|
||||
PodMan uses Gemini for two distinct jobs: **vision** (understanding screens) and **voice** (urgent voice cues).
|
||||
Status: active / matches code.
|
||||
|
||||
PodMan uses Gemini for five jobs, all through the `@google/genai` SDK
|
||||
(`GoogleGenAI`) with a single `GEMINI_API_KEY` (`GOOGLE_API_KEY` /
|
||||
`GOOGLE_GENERATIVE_AI_API_KEY` also accepted):
|
||||
|
||||
1. **Vision** — turn screen frames into structured work context.
|
||||
2. **Embeddings** — vector recall over past coordination events.
|
||||
3. **TTS voice** — spoken urgent escalations over LiveKit.
|
||||
4. **Live conversation** — a real-time voice agent teammates talk to.
|
||||
5. **GenMedia (Lyria)** — a per-pod background score.
|
||||
|
||||
Collision detection and intervention text are **deterministic in code**, not
|
||||
Gemini calls. PodMan does not ask Gemini "is this a conflict?" — that is decided
|
||||
by `backend/src/collision/detector.ts` from fused vision + git truth. This is a
|
||||
deliberate reliability choice for the live demo.
|
||||
|
||||
---
|
||||
|
||||
## 1. Vision — Screen Understanding
|
||||
## 1. Vision — screen understanding
|
||||
|
||||
**Model:** `gemini-2.0-flash` (fast, cheap, strong multimodal)
|
||||
**Model:** `GEMINI_VISION_MODEL` (default `gemini-2.0-flash`)
|
||||
**Code:** `backend/src/vision/gemini.ts` → `analyzeFrame()`
|
||||
|
||||
**Trigger:** every 30s per active engineer, when Hermes receives a `POST /ingest` frame
|
||||
**Trigger:** the LiveKit agent samples a JPEG frame from each engineer's
|
||||
screen-share track (not an HTTP upload — frames arrive over LiveKit).
|
||||
|
||||
**Input:** base64-encoded JPEG, max 1280×720, ~50–80KB after compression
|
||||
**Input:** a single base64 JPEG, sampled at low media resolution.
|
||||
|
||||
**Prompt:**
|
||||
|
||||
```
|
||||
You are analyzing a software engineer's screen during a coding session.
|
||||
Extract the following JSON. If you cannot determine a field with confidence above 0.7, set it to null.
|
||||
**Output:** structured JSON via `responseJsonSchema` (no markdown parsing):
|
||||
|
||||
```ts
|
||||
{
|
||||
"currentFile": "string | null", // active file visible in editor tab or title bar
|
||||
"inferredTask": "string | null", // 1 sentence: what the engineer appears to be doing
|
||||
"terminalVisible": true | false, // is a terminal or CLI panel visible
|
||||
"recentTerminalOutput": "string | null", // last meaningful line of terminal output if visible
|
||||
"confidence": 0.0–1.0 // your overall confidence in this extraction
|
||||
currentFile: string, // open file path, e.g. src/auth/session.ts
|
||||
currentSymbol: string, // function/class under the cursor
|
||||
activity: string, // editing | reading | debugging | terminal | PR review
|
||||
hasUnpushedChanges: boolean, // dirty git gutter / modified markers visible
|
||||
confidence: number // 0..1
|
||||
}
|
||||
|
||||
Respond with valid JSON only. No explanation. No markdown.
|
||||
```
|
||||
|
||||
**Confidence gate:** if `confidence < 0.6`, Hermes discards the frame — no state update, no event detection triggered.
|
||||
**Latency/cost levers (in code):**
|
||||
|
||||
**Rate limit:** 1 call per engineer per 30s. With 3 engineers = 6 calls/min ≈ $0.002/min at Flash pricing.
|
||||
- `thinkingConfig: { thinkingBudget: 0 }` — minimal thinking for the ambient loop.
|
||||
- `mediaResolution: MEDIA_RESOLUTION_LOW` — smaller image tokens.
|
||||
- Missing `confidence` defaults to `0.5`.
|
||||
|
||||
**Demo setup requirement:** editors must have large font (18pt+), single window, file name clearly visible in tab. This is the primary reliability lever.
|
||||
**Demo reliability:** large editor font, single window, visible file tab. This is
|
||||
the primary lever for clean reads.
|
||||
|
||||
---
|
||||
|
||||
## 2. Event Detection — Coordination Awareness
|
||||
## 2. Embeddings — semantic recall
|
||||
|
||||
**Model:** `gemini-2.0-flash` (text only, fast)
|
||||
**Model:** `GEMINI_EMBEDDING_MODEL` (default `gemini-embedding-001`, 768 dims)
|
||||
**Code:** `backend/src/memory/vectors.ts`
|
||||
|
||||
**Trigger:** after every successful state write to MongoDB, Hermes runs event detection over all active engineer contexts.
|
||||
Each collision is embedded into a short memory text (`file`, `symbol`,
|
||||
`engineers`, `severity`, unpushed flag) and stored on the `collisions` document.
|
||||
On a new collision PodMan embeds the query and runs MongoDB Atlas `$vectorSearch`
|
||||
(index `collision_embedding`) to recall similar past events and their outcomes.
|
||||
|
||||
**Input:** JSON snapshot of all engineers' current states + ownership map
|
||||
|
||||
**Prompt:**
|
||||
|
||||
```
|
||||
You are a team coordination agent. Below is the current state of each engineer on the team.
|
||||
|
||||
Engineer states:
|
||||
{{engineerStates}}
|
||||
|
||||
Ownership map (who owns which files):
|
||||
{{ownershipMap}}
|
||||
|
||||
Detect if any of these coordination events are occurring:
|
||||
- DEPENDENCY_READY: an engineer who was blocked or waiting now has what they need because another engineer completed relevant work
|
||||
- BLOCKER_DETECTED: an engineer appears stuck (same file, error in terminal, no progress) and another teammate could help
|
||||
- DUPLICATE_WORK: two or more engineers are working on the same file simultaneously
|
||||
|
||||
If an event is detected, respond with:
|
||||
{
|
||||
"event": "DEPENDENCY_READY" | "BLOCKER_DETECTED" | "DUPLICATE_WORK" | null,
|
||||
"involvedEngineers": ["engineerId", ...],
|
||||
"file": "string | null",
|
||||
"reason": "1 sentence explanation"
|
||||
}
|
||||
|
||||
If no event, respond with { "event": null }.
|
||||
Respond with valid JSON only.
|
||||
```
|
||||
**Provider order:** Voyage (`VOYAGE_API_KEY`, `voyage-4-lite`) is tried first when
|
||||
present; Gemini embeddings are the fallback. Without either, recall degrades to
|
||||
exact signature/file matching — the demo still works.
|
||||
|
||||
---
|
||||
|
||||
## 3. Intervention Text Generation
|
||||
## 3. TTS voice — urgent escalation over LiveKit
|
||||
|
||||
**Model:** `gemini-2.0-flash` (text only)
|
||||
**Model:** `GEMINI_LIVE_MODEL` (default `gemini-3.1-flash-tts-preview`)
|
||||
**Default voice:** `GEMINI_TTS_VOICE` (default `Charon`)
|
||||
**Code:** `backend/src/voice/live.ts` → `speak()` / `speakInRoom()`
|
||||
|
||||
**Trigger:** when event detection returns a non-null event
|
||||
|
||||
**Input:** event type + engineer names + file + reason
|
||||
|
||||
**Prompt:**
|
||||
|
||||
```
|
||||
You are PodMan, a friendly AI teammate. Generate a short spoken message (1–2 sentences max) to notify the team about this coordination event.
|
||||
|
||||
Event: {{eventType}}
|
||||
Engineers involved: {{engineerNames}}
|
||||
File: {{file}}
|
||||
Context: {{reason}}
|
||||
|
||||
Rules:
|
||||
- Use first names only
|
||||
- Be direct and specific
|
||||
- Do not use filler words
|
||||
- Sound natural when spoken aloud
|
||||
- Do not start with "Hey" or "Attention"
|
||||
|
||||
Respond with the message text only.
|
||||
```
|
||||
|
||||
**Example output:**
|
||||
|
||||
> "Carol — Alice just got the auth endpoint running. You're clear to integrate."
|
||||
Flow: a short, natural voice line is generated for a critical collision, returned
|
||||
as audio, and published as a LiveKit microphone-source audio track. The track is
|
||||
held for the audio duration plus tail/hold so browsers do not cut playout short.
|
||||
Browser audio must be unlocked by a user gesture first. The frontend always
|
||||
renders the `VOICE_CUE` text as a fallback. See `docs/livekit.md` for delivery.
|
||||
|
||||
---
|
||||
|
||||
## 4. Voice Output — Gemini TTS via LiveKit
|
||||
## 4. Live conversation — real-time voice agent
|
||||
|
||||
**Model:** `gemini-3.1-flash-tts-preview`
|
||||
**Default voice:** `Charon`
|
||||
**Model:** `GEMINI_CONVERSATION_MODEL` (default `gemini-3.1-flash-live-preview`)
|
||||
**Code:** `agents/podman-live-conversation/agent.py` (Python LiveKit Agents,
|
||||
`google.realtime.RealtimeModel`)
|
||||
|
||||
**Integration:** Hermes asks Gemini TTS for short PCM audio, then publishes that audio into the room as a short LiveKit audio track. The code still preserves a Gemini Live path for future available Live models.
|
||||
A teammate can start a live, streaming speech-to-speech session with PodMan. The
|
||||
agent answers using **function tools** rather than guessing, including:
|
||||
|
||||
**Flow:**
|
||||
- `get_active_pod_context`, `get_recent_changes`, `search_team_memory`
|
||||
- `search_repo`, `repo_recent_commits`, `repo_find_commits` (repo + git history)
|
||||
- `record_conversation_note`
|
||||
- `delegate_to_hermes`, `abort_active_hermes_job` (hands work to the async Hermes
|
||||
job runner — see `docs/hermes.md`)
|
||||
|
||||
1. Intervention message text generated (step 3)
|
||||
2. Hermes wraps it in a natural-speaking prompt for Gemini TTS
|
||||
3. Gemini returns audio with the configured prebuilt voice
|
||||
4. Hermes publishes the audio into the LiveKit room
|
||||
5. The frontend still renders the `VOICE_CUE` text, but browser TTS is off unless explicitly enabled
|
||||
Started/stopped via `POST /api/pods/:id/live-conversation/start` and `.../stop`.
|
||||
|
||||
**Why Gemini TTS first:**
|
||||
---
|
||||
|
||||
- Natural voice quality is better than browser `speechSynthesis`
|
||||
- Tone and pacing can be steered directly in the prompt
|
||||
- The voice name is configurable with `GEMINI_TTS_VOICE`
|
||||
- LiveKit remains the delivery layer, so teammates hear the same room audio
|
||||
## 5. GenMedia — Lyria background score
|
||||
|
||||
**Model:** `lyria-3-clip-preview` (override with `GEMINI_MUSIC_MODEL`)
|
||||
**Endpoint:** Gemini **Interactions API** (`/v1beta/interactions`)
|
||||
**Code:** `backend/src/voice/music.ts`
|
||||
|
||||
A pod-specific ~30s clip is generated through the Interactions API, cached in
|
||||
MongoDB, and served via `GET /api/pods/:id/music` to play as ambient room audio.
|
||||
|
||||
---
|
||||
|
||||
## Cooldown
|
||||
|
||||
Per-pod cooldown of **3 minutes** between urgent voice cues. Prevents spam if multiple risks fire simultaneously. Implemented in Hermes, not in Gemini.
|
||||
Per-pod cooldown (`NUDGE_COOLDOWN_MS`, default 180000 ms / 3 min) gates repeated
|
||||
interventions. Implemented in `backend/src/memory/policy.ts`, not in Gemini.
|
||||
|
||||
@@ -1,38 +0,0 @@
|
||||
# Graph Discovery
|
||||
|
||||
Status: demo-backed / active
|
||||
|
||||
Graph discovery owns how MongoDB records become the Team memory graph. It
|
||||
materializes a sparse, auditable graph from real records first, seeded graph
|
||||
second, and demo fallback third.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| [`spec.md`](spec.md) | Source data, graph contract, and discovery rules |
|
||||
| [`policy.md`](policy.md) | Graph hygiene, evidence thresholds, and truthfulness |
|
||||
| [`prompt.md`](prompt.md) | Graph materialization and review prompt |
|
||||
| [`plan.md`](plan.md) | Risk-path and observatory build plan |
|
||||
|
||||
## What Is Implemented Now
|
||||
|
||||
- `GET /api/pods/:podId/graph`.
|
||||
- `GET /api/pods/:podId/graph/reach/:id` backed by MongoDB `$graphLookup`.
|
||||
- Live graph materialization from `pods`, `engineer_states`, `observations`,
|
||||
`collisions`, `interventions`, and `outcomes`.
|
||||
- Seeded graph in `team_model.graph` and mirrored `graph_nodes` / `graph_edges`.
|
||||
- Demo graph fallback so the stage never shows an empty canvas.
|
||||
|
||||
## What Is Intentionally Cut
|
||||
|
||||
- A separate graph database.
|
||||
- A broad analytics dashboard.
|
||||
- Showing every historical event by default.
|
||||
- Treating seeded demo data as live learning.
|
||||
|
||||
## Demo Proof Path
|
||||
|
||||
Observe screen/git state -> detect collision -> send intervention -> accept or
|
||||
dismiss outcome -> recall similar event -> show changed graph or changed
|
||||
behavior.
|
||||
@@ -1,68 +0,0 @@
|
||||
# Graph Discovery Plan
|
||||
|
||||
Status: demo-backed / active
|
||||
Goal: make MongoDB graph discovery visible as a dynamic learning observatory
|
||||
|
||||
## Must-Have
|
||||
|
||||
1. Keep live materializer as source of graph truth.
|
||||
2. Add optional loop and activity fields.
|
||||
3. Build a dynamic graph layout.
|
||||
4. Default to risk path.
|
||||
5. Make selected-node detail explain the story.
|
||||
|
||||
## Build Order
|
||||
|
||||
### R1: Stabilize discovered graph
|
||||
|
||||
- Keep file and engineer noise filters.
|
||||
- Keep collision collapse.
|
||||
- Keep priority for accepted-outcome paths.
|
||||
- Keep graph size capped.
|
||||
|
||||
### R2: Add observatory data
|
||||
|
||||
- Compute learning-loop snapshot.
|
||||
- Compute activity stream.
|
||||
- Preserve current graph contract.
|
||||
|
||||
### R3: Improve path selection
|
||||
|
||||
- Pick one primary risk path.
|
||||
- Include learned path when present.
|
||||
- Dim unrelated collisions and repeated interventions.
|
||||
|
||||
### R4: Render dynamically
|
||||
|
||||
- Use `d3-force` or animated layered layout.
|
||||
- Make nodes draggable.
|
||||
- Curve or bundle edges.
|
||||
- Animate `learned_from`.
|
||||
|
||||
### R5: Verify with real data
|
||||
|
||||
- Fetch live `demo-pod` graph.
|
||||
- Confirm labels do not collide badly.
|
||||
- Confirm red edges do not dominate.
|
||||
- Confirm activity and loop explain the graph.
|
||||
|
||||
## Nice-to-Have
|
||||
|
||||
- Reachability panel using `$graphLookup`.
|
||||
- Hover path previews.
|
||||
- Edge bundling by file or collision.
|
||||
- Time scrubber for graph snapshots.
|
||||
|
||||
## Cut
|
||||
|
||||
- Generic analytics dashboard.
|
||||
- Large graph database migration.
|
||||
- Rendering every historical event.
|
||||
- Static fixed-column final layout.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Risk path is obvious in 10 seconds.
|
||||
- Learned path is visible when data exists.
|
||||
- Whole graph mode exists but is not the default.
|
||||
- The graph remains backed by MongoDB, not hardcoded mock data.
|
||||
@@ -1,82 +0,0 @@
|
||||
# Graph Discovery Policy
|
||||
|
||||
Status: demo-backed / active
|
||||
Scope: graph hygiene, evidence thresholds, and UI truthfulness
|
||||
|
||||
## Prime Rule
|
||||
|
||||
The graph must be sparse enough to explain the learning loop and truthful enough
|
||||
to audit from MongoDB.
|
||||
|
||||
## Node Policy
|
||||
|
||||
Create nodes only when they add explanation value.
|
||||
|
||||
Allowed:
|
||||
|
||||
- Current engineers.
|
||||
- Real files.
|
||||
- Current or recent collisions.
|
||||
- Interventions tied to surviving collisions.
|
||||
- Learned ownership paths.
|
||||
|
||||
Avoid:
|
||||
|
||||
- Test engineers.
|
||||
- Scratch files.
|
||||
- URLs or environment values misread as files.
|
||||
- Repeated identical intervention diamonds.
|
||||
- Orphan nodes with no story value.
|
||||
|
||||
## Edge Policy
|
||||
|
||||
Edges need evidence.
|
||||
|
||||
| Edge | Required evidence |
|
||||
| --- | --- |
|
||||
| `editing` | observation or git state |
|
||||
| `touches` | file involved in collision |
|
||||
| `collides` | collision prediction |
|
||||
| `warns` | intervention record |
|
||||
| `learned_from` | accepted real outcome |
|
||||
| `owns` | learned or configured ownership |
|
||||
|
||||
## De-Hairball Policy
|
||||
|
||||
Default mode must not show every relationship equally.
|
||||
|
||||
Rules:
|
||||
|
||||
- Default to risk path.
|
||||
- Collapse repeated collision signatures.
|
||||
- Cap files and collisions.
|
||||
- Dim non-risk edges.
|
||||
- Bundle or curve dense edges.
|
||||
- Hide low-priority labels until hover or select.
|
||||
- Prefer selected-node explanation over labels everywhere.
|
||||
|
||||
## Truthfulness Policy
|
||||
|
||||
- Do not show `learned_from` for orphaned or dismissed outcomes.
|
||||
- Do not label vector similarity as learned memory.
|
||||
- Do not show demo seed as live learning unless labeled.
|
||||
- Do not hide false positives from activity or memory.
|
||||
|
||||
## Privacy Policy
|
||||
|
||||
Graph labels should not expose secrets, raw terminal output, or sensitive file
|
||||
contents. File paths are acceptable when they are repo paths and not secret
|
||||
values.
|
||||
|
||||
## Visual Policy
|
||||
|
||||
Semantic colors stay stable:
|
||||
|
||||
- Engineer: blue.
|
||||
- File: slate.
|
||||
- Feature: amber.
|
||||
- Collision: red.
|
||||
- Intervention: violet.
|
||||
- Learned: violet dashed edge.
|
||||
|
||||
Chrome should use the app's light shadcn tokens.
|
||||
@@ -1,81 +0,0 @@
|
||||
# Graph Discovery Prompt
|
||||
|
||||
Use this prompt for an agent that materializes or reviews PodMan's Team memory
|
||||
graph.
|
||||
|
||||
## Prompt
|
||||
|
||||
You are PodMan's graph discovery agent.
|
||||
|
||||
Your job is to turn MongoDB records into a sparse, truthful graph that explains
|
||||
the continual-learning loop. Do not maximize node count. Maximize legibility and
|
||||
evidence.
|
||||
|
||||
The default output should show the risk path and learned path, not every
|
||||
possible edge.
|
||||
|
||||
## Inputs
|
||||
|
||||
- Pod id.
|
||||
- Pod roster.
|
||||
- Recent engineer states.
|
||||
- Recent observations.
|
||||
- Collisions.
|
||||
- Interventions.
|
||||
- Outcomes.
|
||||
- Team model.
|
||||
- Existing graph nodes and edges.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. Normalize file paths.
|
||||
2. Remove noise.
|
||||
3. Create engineer and file nodes.
|
||||
4. Collapse repeated collisions by signature.
|
||||
5. Preserve accepted-outcome paths.
|
||||
6. Create intervention nodes for surviving collisions.
|
||||
7. Create learned edges only from accepted real outcomes.
|
||||
8. Select the primary risk path.
|
||||
9. Build activity and loop summaries.
|
||||
10. Explain selected-node stories.
|
||||
|
||||
## Output Format
|
||||
|
||||
```text
|
||||
Graph Summary
|
||||
- Pod:
|
||||
- Nodes:
|
||||
- Edges:
|
||||
- Primary risk path:
|
||||
- Learned path:
|
||||
|
||||
Discovery Decisions
|
||||
- Collapsed:
|
||||
- Dropped as noise:
|
||||
- Preserved because learned:
|
||||
|
||||
Loop
|
||||
- Observe:
|
||||
- Store:
|
||||
- Predict:
|
||||
- Outcome:
|
||||
- Adapt:
|
||||
|
||||
Activity
|
||||
- Recent events:
|
||||
|
||||
Risks
|
||||
- Missing evidence:
|
||||
- Potential hairball:
|
||||
- Demo caveat:
|
||||
```
|
||||
|
||||
## Hard Rules
|
||||
|
||||
- No `learned_from` without accepted real outcome.
|
||||
- No raw screenshots or secrets in labels.
|
||||
- Do not rewrite the backend materializer unless explicitly asked.
|
||||
- Prefer additive graph fields.
|
||||
- Default to risk path.
|
||||
- Keep whole graph optional.
|
||||
|
||||
@@ -1,159 +0,0 @@
|
||||
# Graph Discovery Spec
|
||||
|
||||
Status: demo-backed / active
|
||||
Scope: how PodMan discovers graph nodes, edges, risk paths, and learning paths from MongoDB
|
||||
Owner: graph discovery / Team memory observatory
|
||||
|
||||
## Purpose
|
||||
|
||||
Graph discovery turns MongoDB memory into a legible Team memory graph. It is not
|
||||
only layout. It decides which relationships matter, which path is highlighted,
|
||||
and which evidence explains the graph.
|
||||
|
||||
The graph must answer:
|
||||
|
||||
1. Who is working?
|
||||
2. Which files or symbols overlap?
|
||||
3. Where is the risk?
|
||||
4. What did PodMan do?
|
||||
5. What outcome changed memory?
|
||||
|
||||
## What Is Implemented Now
|
||||
|
||||
- Live materializer first: build from current MongoDB records.
|
||||
- Seeded graph second: read `team_model.graph` and mirrored graph collections.
|
||||
- Demo fallback third: return a grounded demo graph when live data is empty or
|
||||
unavailable.
|
||||
- Reachability uses MongoDB `$graphLookup` over `graph_edges`.
|
||||
|
||||
## What Is Intentionally Cut
|
||||
|
||||
- A graph database migration.
|
||||
- Whole-history rendering as the default view.
|
||||
- Claims that seeded graph data is live learning.
|
||||
|
||||
## Source Data
|
||||
|
||||
Graph discovery reads:
|
||||
|
||||
- `pods`
|
||||
- `engineer_states`
|
||||
- `observations`
|
||||
- `collisions`
|
||||
- `interventions`
|
||||
- `outcomes`
|
||||
- `team_model`
|
||||
- `graph_nodes`
|
||||
- `graph_edges`
|
||||
- optional `memory_vectors`
|
||||
- optional `agent_runs`
|
||||
- optional `strategy_versions`
|
||||
|
||||
## UI Graph Contract
|
||||
|
||||
```text
|
||||
PodGraph
|
||||
podId
|
||||
generatedAt
|
||||
nodes
|
||||
edges
|
||||
metrics
|
||||
loop?
|
||||
activity?
|
||||
```
|
||||
|
||||
Node kinds:
|
||||
|
||||
```text
|
||||
engineer, feature, file, collision, intervention
|
||||
```
|
||||
|
||||
Edge kinds:
|
||||
|
||||
```text
|
||||
owns, editing, touches, collides, warns, learned_from
|
||||
```
|
||||
|
||||
## Discovery Rules
|
||||
|
||||
### Engineer nodes
|
||||
|
||||
Create from pod roster, recent observations, git state, or collision membership.
|
||||
|
||||
### File nodes
|
||||
|
||||
Create only from normalized real file paths. Reject noise such as URLs, env
|
||||
values, scratch names, and non-file strings.
|
||||
|
||||
### Collision nodes
|
||||
|
||||
Create from distinct collision signatures. Collapse repeats. Prioritize
|
||||
collisions referenced by accepted outcomes.
|
||||
|
||||
### Intervention nodes
|
||||
|
||||
Create one visible intervention per surviving collision unless whole-graph mode
|
||||
explicitly expands history.
|
||||
|
||||
### Learned paths
|
||||
|
||||
Create `learned_from` only when an accepted real outcome links an intervention
|
||||
to a durable memory update.
|
||||
|
||||
## Path Modes
|
||||
|
||||
### Risk path
|
||||
|
||||
Default mode. Highlight the clearest current chain:
|
||||
|
||||
```text
|
||||
engineer -> file -> collision -> intervention -> learned owner
|
||||
```
|
||||
|
||||
Dim unrelated graph material.
|
||||
|
||||
### Learning edges
|
||||
|
||||
Highlight `learned_from`, `owns`, and the outcomes that produced them.
|
||||
|
||||
### Whole graph
|
||||
|
||||
Show all materialized nodes and edges with de-emphasized non-critical edges.
|
||||
|
||||
## MongoDB Traversal
|
||||
|
||||
Use `graph_edges` for reachability:
|
||||
|
||||
```text
|
||||
source -> target -> next target
|
||||
```
|
||||
|
||||
Primary traversal questions:
|
||||
|
||||
- What risks does this engineer reach?
|
||||
- Which files feed this collision?
|
||||
- Which intervention came from this collision?
|
||||
- Which learned owner came from this intervention?
|
||||
|
||||
## Metrics
|
||||
|
||||
Minimum metrics:
|
||||
|
||||
- Learned owners.
|
||||
- Open risk paths.
|
||||
- Accept rate.
|
||||
|
||||
Optional metrics:
|
||||
|
||||
- Observations.
|
||||
- Interventions.
|
||||
- Memory vectors.
|
||||
- Strategy versions.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Default graph is not a hairball.
|
||||
- Every visible learned edge has outcome evidence.
|
||||
- Every selected node can explain why it matters.
|
||||
- Activity stream matches graph events.
|
||||
- Graph can be rebuilt from MongoDB source records.
|
||||
@@ -1,167 +0,0 @@
|
||||
# Claude Code Handoff — Team-Memory **Dynamic Graph** (redesign + deploy reconciliation)
|
||||
|
||||
> **Fresh-session handoff.** Read this top-to-bottom before touching the Team-memory graph.
|
||||
> It captures the dynamic-graph redesign, the honest-metrics fix, the click-to-explain "Flow"
|
||||
> pane, and — most importantly — how it was **reconciled against `main`'s parallel
|
||||
> implementation** so it can actually deploy. Author: Claude Code (Opus 4.8) session, 2026-06-28.
|
||||
|
||||
---
|
||||
|
||||
## 0. TL;DR — what to do next
|
||||
|
||||
1. **Merge [PR #38](https://github.com/karti-ai/podman/pull/38) → `main`.** It is `MERGEABLE` (no conflicts) and is the deploy path. Deploy = push to `main` (`deploy_on_push`).
|
||||
2. **After deploy, hard-refresh** (Cmd-Shift-R) or use a private window — the PWA service worker caches aggressively, so you'll think nothing changed.
|
||||
3. **Close [PR #22](https://github.com/karti-ai/podman/pull/22)** (the older one into `feat/live-graph-glue`) with a note pointing at #38; it is superseded for the deploy path.
|
||||
4. Optionally delete the stale branches `feat/live-graph-glue` and `feat/team-memory-dynamic-graph` once #38 lands.
|
||||
|
||||
**State right now:** the live site (`165-22-129-249.sslip.io` / `podman.live`) runs `main`, which has a **static** graph with **inflated metrics** and **does not render the learning-loop / activity rails** (even though its backend computes them). PR #38 fixes all three.
|
||||
|
||||
---
|
||||
|
||||
## 1. Branches, PRs, deploy
|
||||
|
||||
| Branch | What's on it | Status |
|
||||
| --- | --- | --- |
|
||||
| `main` | Trunk. Has a **parallel** Team-memory impl: computes `loop`/`activity` in the API (rich types) but **static** graph, **inflated** metrics, rails **never rendered**. Deployed. | live |
|
||||
| `feat/team-memory-deploy` | **The reconciliation.** Dynamic graph + honest metrics + Flow pane, on top of `main`, rails adapted to `main`'s types. | **PR #38 → `main`, MERGEABLE** |
|
||||
| `feat/live-graph-glue` | Where the original redesign (v1) was merged (PR #22). Branched ~100 commits before `main`'s later work; a direct merge to `main` was unsafe/tangled. | superseded by #38 |
|
||||
| `feat/team-memory-dynamic-graph` | v1 PR branch (merged into glue via #22). | superseded |
|
||||
|
||||
- **Deploy = merge/push to `main`.** Do **not** push to `main` directly; merge the PR. `main` is shared by ~4 engineers and moves fast.
|
||||
- Live API to sanity-check: `curl https://165-22-129-249.sslip.io/api/pods/demo-pod/graph`.
|
||||
|
||||
---
|
||||
|
||||
## 2. The divergence (read this — it's the crux)
|
||||
|
||||
While the dynamic-graph redesign was being built on `feat/live-graph-glue`, **another engineer shipped a *parallel* version of the same feature on `main`.** They are not the same:
|
||||
|
||||
| Concern | `main` (deployed) | This redesign (PR #38) |
|
||||
| --- | --- | --- |
|
||||
| Graph layout | **Static** — renders server `x`/`y` columns, no motion | **Dynamic** force-directed (`forceSim.ts`), draggable, animated |
|
||||
| Learning-loop rail | **Computed in API, never rendered** | Rendered (`LearningLoop.tsx`) |
|
||||
| Activity stream | **Computed in API, never rendered** | Rendered (`ActivityStream.tsx`) |
|
||||
| Metric cards | **Inflated** (raw collision-signature / accepted-outcome counts → e.g. 50 risk paths for 4 files) | **De-noised** (distinct collision files / owner engineers) |
|
||||
| Selected-node pane | kind/status/relationships | + **Flow** narrative ("Karti and Yahya are both editing auth.ts…") |
|
||||
| `loop` type | `PodLearningLoop` = `{ activeStep, steps: PodLearningLoopStep[] }` (richer; step `status`) | **kept `main`'s** |
|
||||
| `activity` type | `PodGraphActivity` = `{ id, at, kind, title, detail, nodeId?, edgeId? }` (richer) | **kept `main`'s** |
|
||||
|
||||
**Reconciliation strategy (what PR #38 does):** keep `main` as the trunk; keep `main`'s **backend** materializer/`buildLoop`/activity and its **richer shared types**; swap in **this redesign's frontend graph layer**; adapt the rail components to consume `main`'s types; port only the **honest-metrics** fix into `main`'s `live.ts`.
|
||||
|
||||
> A literal `feat/live-graph-glue → main` merge was attempted first and produced **invalid auto-merge states** (duplicate `loop`/`activity` keys in `demo.ts`, duplicate imports in `live.ts`) because glue was ~100 commits stale. It was aborted; the same best-of-both was re-applied cleanly on a branch off `main`. The PR diff is just the graph layer (11 files).
|
||||
|
||||
---
|
||||
|
||||
## 3. What PR #38 changes (11 files)
|
||||
|
||||
**Frontend — new `frontend/src/components/graph/*`, composed into `GraphView.tsx`:**
|
||||
- `forceSim.ts` — **dependency-free** force layout (charge repulsion, link springs, centroid recenter + gentle pull, 2-pass collision, bounds clamp, alpha annealing). Driven by a `requestAnimationFrame` loop. **No new dependency / no `pnpm-lock.yaml` change.**
|
||||
- `GraphCanvas.tsx` — SVG render from the sim: **draggable + pinnable** nodes (double-click to release), curved edges that fan parallel pairs, weight-sized geometric shapes (square / outlined-square / circle / triangle / diamond), fade-in on new nodes/edges, animated `learned_from` dash, **risk-path lit / rest dimmed**, label collision-avoidance.
|
||||
- `encoding.ts` — node/edge kind colors, `highlightFor` (risk/learn/all modes), `flowNarrative(graph, nodeId)` (the plain-English path walk), `modeBlurb`, legends, `ACTIVITY_TAG` (keyed by `main`'s `PodGraphActivityKind`).
|
||||
- `MetricsRail.tsx`, `LearningLoop.tsx`, `ActivityStream.tsx`, `SelectedNodePanel.tsx` — the rails + stream + detail pane in light shadcn (`@/components/ui/*`). `LearningLoop` consumes `PodLearningLoop` (steps + `activeStep`); `ActivityStream` consumes `PodGraphActivity` (title + detail); `SelectedNodePanel` renders the **Flow** section + mode-aware default copy.
|
||||
- `GraphView.tsx` — composes everything; polls `/api/pods/:id/graph` every 5s and **diffs** (positions/pins preserved across refreshes — no hard replace), + best-effort `ws /api/events` nudge; drops a **stale selection** (selected node gone across a poll) so the canvas can't dim entirely.
|
||||
- `lib/graph.ts` — adds `backendEventsUrl()` (http→ws) for the nudge.
|
||||
|
||||
**Backend — surgical (keeps `main`'s materializer + `buildLoop` + activity builder):**
|
||||
- `backend/src/graph/live.ts` — the **headline metric cards** are now derived from the **final de-noised graph**: *Open risk paths* = distinct collision **files** (`touches` edges to file nodes); *Learned owners* = distinct **owner engineers** (`owns`/`learned_from` edges). On live data: **50 → 4** risk files, **16 → 1** owner. `riskPaths` (raw signatures) is still computed and fed to `buildLoop` (the loop is a throughput view, intentionally separate). **`buildLoop` and the activity builder are untouched.**
|
||||
- `backend/src/graph/demo.ts` — fallback metrics realigned to the demo graph (3 owners / 1 risk path / 100%) so the numbers never contradict the picture.
|
||||
|
||||
---
|
||||
|
||||
## 4. How it works (architecture)
|
||||
|
||||
```
|
||||
materializePodGraph(podId) // backend/src/graph/live.ts (main's, + metric fix)
|
||||
→ GET /api/pods/:id/graph // backend/src/server.ts (PodGraph incl. loop/activity)
|
||||
→ fetchPodGraph() poll every 5s // frontend/src/lib/graph.ts (+ ws /api/events nudge)
|
||||
→ GraphView // diffs snapshots, computes highlight + flow
|
||||
→ GraphCanvas (forceSim tick → SVG, draggable)
|
||||
→ MetricsRail / LearningLoop / ActivityStream / SelectedNodePanel
|
||||
```
|
||||
|
||||
- **`PodGraph` data contract** (`shared/src/graph.ts`, `main`'s): `nodes`, `edges`, `metrics`, optional `loop?: PodLearningLoop`, `activity?: PodGraphActivity[]`. Node kinds: engineer/feature/file/collision/intervention. Edge kinds: owns/editing/touches/collides/warns/learned_from.
|
||||
- **Force sim** ignores the server's `x`/`y` except as **seed** positions (mapped into the canvas). Key tuning constants in `forceSim.ts`: `REPEL=4400`, `CENTER_STRENGTH=0.014`, `RECENTER=0.5`, `COLLIDE_PAD=12`, `COLLIDE_ITERS=2`, `BOUND_PAD=30`. Repulsion must dominate centering or the graph collapses to a point.
|
||||
- **Default mode is "Risk path"** — lights the collision→intervention→`learned_from` chain, dims the rest to opacity `0.14`.
|
||||
- **Flow narrative** (`flowNarrative`) walks a node's incident edges to produce sentences, e.g. collision → "Karti and Yahya are both editing auth.ts before pushing — the overlap git can't see. PodMan stepped in and suggested a sync PR."
|
||||
|
||||
---
|
||||
|
||||
## 5. Is the data real / dynamic? (FAQ — was asked)
|
||||
|
||||
- **Real:** yes. The graph is materialized **live from the real Atlas collections** (`observations`, `collisions`, `interventions`, `outcomes`, `engineer_states`, `pods`, `team_model`) on every request — not the hardcoded demo. The demo only shows as a **fallback** when there's zero activity. `generatedAt` advances on each request (re-materialized, not cached).
|
||||
- **Dynamic:** the backend re-reads Mongo per request and the frontend polls ~5s + WS nudge, so the UI reflects current DB state within seconds. **But it only *changes* when the PodMan agent writes new data** (vision → observations → collisions → interventions → outcomes). When the agent is idle, the graph is static at last-known state. (At handoff time the newest activity was ~52 min old — no live ingestion.)
|
||||
- **Numbers look inflated** because a lot of the real data is **test churn** (repeated `infra/README.md` collisions). The honest-metrics fix (§3) counts distinct entities so the cards match the graph; `MAX_COLLISIONS=8` in the materializer caps the visible collisions.
|
||||
|
||||
---
|
||||
|
||||
## 6. Build / verify (toolchain quirks — important)
|
||||
|
||||
- **`pnpm` is not on PATH** in the dev sandbox; use **`CI=true npx pnpm@10.32.1 …`** (pin 10.32.1 to match CI and keep the lockfile v10-compatible; `CI=true` avoids the no-TTY abort).
|
||||
- `CI=true npx pnpm@10.32.1 lint`
|
||||
- `CI=true npx pnpm@10.32.1 -r typecheck`
|
||||
- `CI=true npx pnpm@10.32.1 -r build`
|
||||
- The `npm error config prefix cannot be changed from project config: .npmrc` line is a **non-fatal warning** (tsc/eslint still run; exit code 0).
|
||||
- **Adding a dependency is high-friction**: `pnpm install` wants to wipe + recreate `node_modules` (modules-dir version mismatch) and churns the shared `pnpm-lock.yaml` that CI's `--frozen-lockfile` depends on. That's why the force sim is **in-house**, not `d3-force`. Prefer zero-dep solutions.
|
||||
- **CI** (`.github/workflows/hermes-verify.yml`) runs `pnpm install --frozen-lockfile && pnpm lint && pnpm -r typecheck && pnpm -r build` (pnpm 10.32.1, node 22).
|
||||
|
||||
**Running it locally to eyeball:** the **backend can't run locally** (needs LiveKit/Gemini/GitHub secrets and hard-exits without Mongo). Verify the **frontend** by pointing a vite dev server at either the live backend (CORS is `*`) or a tiny mock that serves `createDemoPodGraph()` from `backend/dist/graph/demo.js`:
|
||||
|
||||
```
|
||||
# mock backend (node http) serving GET /api/pods/:id/graph from backend/dist/graph/demo.js
|
||||
# then: VITE_BACKEND_URL=http://localhost:8799 in frontend/.env.local
|
||||
CI=true npx pnpm@10.32.1 --filter @podman/frontend dev
|
||||
```
|
||||
|
||||
A throwaway entry (`frontend/graph-preview.html` + `frontend/src/graph-preview.tsx` mounting `<GraphView podId="demo-pod" .../>`) renders the view directly without the pods list. Drive it with Playwright (already a devDependency; chromium is cached) — sample node positions over time to confirm the sim ticks, `getComputedStyle` opacity to confirm dimming, click nodes to read the Flow text. **Delete all of these temp files before committing.**
|
||||
|
||||
---
|
||||
|
||||
## 7. Gotchas (these already bit; don't re-learn them)
|
||||
|
||||
- **PWA cache** — hard-refresh after every deploy or you'll think nothing changed.
|
||||
- **StrictMode RAF freeze (fixed):** the dev double-mount cancels the animation frame between effect passes; the loop must re-arm **unconditionally** after `setData` (`ensureRaf()` is idempotent via the `rafRef==null` guard), not gated on a topology change — else the sim is frozen at seed positions in dev until first interaction. Seed positions are a plausible layout, so this can hide.
|
||||
- **Dimming vs animation (fixed):** `.pm-dim` opacity is defeated if the fade-in uses `animation-fill-mode: both/forwards` (held final keyframe overrides the class). The enter animation must use **no fill-mode**.
|
||||
- **Stale selection (fixed):** after a poll, a selected node can vanish from the payload; `highlightFor(selected)` would then light only a dead id and dim the whole graph. `GraphView` derives `liveSelected = selected ∈ nodeById ? selected : null` and clears it.
|
||||
- **Force tuning:** repulsion must dominate centering (`REPEL ≫ CENTER_STRENGTH·r`) or the graph collapses; `COLLIDE_ITERS≥2` keeps linked nodes from stacking.
|
||||
- **Pathological data:** with the *uncapped* live materializer (pre-`MAX_COLLISIONS`), engineer labels can crowd the center because each engineer fans many `collides` edges. The materializer cap is the real fix; the frontend still spreads + draggable.
|
||||
- **`learned_from` "money" edge on `demo-pod`** won't draw on real data unless there's one intact accept flow (its one accepted outcome is orphaned). The demo fallback shows it.
|
||||
- **Parallel impl on `main`:** keep `main`'s `PodLearningLoop`/`PodGraphActivity` types and `buildLoop`/activity builder. Do **not** reintroduce the v1 `LearningStage`/`ActivityEvent` types — they were dropped in the reconciliation.
|
||||
|
||||
---
|
||||
|
||||
## 8. Open items / nice-to-haves
|
||||
|
||||
- **Merge PR #38, redeploy, hard-refresh** (the headline).
|
||||
- Close PR #22; delete `feat/live-graph-glue` + `feat/team-memory-dynamic-graph`.
|
||||
- Optional polish: more canvas spread on small graphs; label de-clutter for pathological/uncapped data; seed a clean collision→intervention→accept chain on `demo-pod` so the violet money edge draws on real data.
|
||||
- Optional: reconcile the loop "Predict" value (distinct signatures) vs the "Open risk paths" card (distinct files) — they intentionally differ today; could unify wording if it confuses.
|
||||
|
||||
---
|
||||
|
||||
## 9. Key files
|
||||
|
||||
| File | Role |
|
||||
| --- | --- |
|
||||
| `backend/src/graph/live.ts` | `materializePodGraph` — real-data graph + `buildLoop`/activity (main's) + **honest metric cards** |
|
||||
| `backend/src/graph/demo.ts` | demo fallback (graph + loop + activity + consistent metrics) |
|
||||
| `backend/src/graph/store.ts` | `loadPodGraph` (live → seeded `team_model.graph` → demo), `reachFrom` (`$graphLookup`) |
|
||||
| `shared/src/graph.ts` | `PodGraph` contract incl. `PodLearningLoop`, `PodGraphActivity` (main's types) |
|
||||
| `frontend/src/components/GraphView.tsx` | page: header, toggles, 3-panel grid, poll/WS, compose |
|
||||
| `frontend/src/components/graph/forceSim.ts` | the in-house force simulation |
|
||||
| `frontend/src/components/graph/GraphCanvas.tsx` | dynamic SVG graph (drag/animate) |
|
||||
| `frontend/src/components/graph/encoding.ts` | colors, `highlightFor`, `flowNarrative`, legends, activity tags |
|
||||
| `frontend/src/components/graph/{MetricsRail,LearningLoop,ActivityStream,SelectedNodePanel}.tsx` | rails/stream/detail |
|
||||
| `frontend/src/lib/graph.ts` | `fetchPodGraph`, `backendEventsUrl` |
|
||||
|
||||
---
|
||||
|
||||
## 10. Verification done (PR #38)
|
||||
|
||||
`pnpm lint` + `-r typecheck` + `-r build` pass. Playwright (dev/StrictMode) confirmed, against the demo payload served from a mock:
|
||||
- dynamic graph **ticks on load** (positions move with no interaction) and is **draggable**;
|
||||
- **learning-loop rail** renders main's 5 steps (ADAPT active) and the **activity stream** renders main's title+detail;
|
||||
- metric cards read **3 / 1 / 100%** (consistent with the graph);
|
||||
- **Flow** narrative is correct per node kind (collision / intervention / engineer / file);
|
||||
- **no overlapping nodes** (≥48px min separation), **zero page errors** (only an expected WS 404 against the mock, handled).
|
||||
|
||||
The honest-metric formula was also re-checked against the **real live graph**: **4** distinct risk files / **1** learned owner (vs the deployed 50 / 16).
|
||||
@@ -1,666 +0,0 @@
|
||||
# Codex Handoff: MongoDB Cleanup, Team Memory Graph, and RSI Learning Docs
|
||||
|
||||
Date: 2026-06-28
|
||||
Repo state checked: `main` at `1d097b1`
|
||||
Database checked: MongoDB Atlas database named `podman`
|
||||
Scope: docs/spec handoff, live Team memory graph verification, and safe DB cleanup path
|
||||
|
||||
## Current Repo State
|
||||
|
||||
The local checkout was moved to `main` and fast-forwarded to `origin/main`.
|
||||
Working tree was clean after inspection.
|
||||
|
||||
The learning and graph specification docs are present on `main`:
|
||||
|
||||
- `docs/agent-learning/README.md`
|
||||
- `docs/agent-learning/spec.md`
|
||||
- `docs/agent-learning/policy.md`
|
||||
- `docs/agent-learning/plan.md`
|
||||
- `docs/agent-learning/prompt.md`
|
||||
- `docs/continual-learning/README.md`
|
||||
- `docs/continual-learning/spec.md`
|
||||
- `docs/continual-learning/policy.md`
|
||||
- `docs/continual-learning/plan.md`
|
||||
- `docs/continual-learning/prompt.md`
|
||||
- `docs/graph-discovery/README.md`
|
||||
- `docs/graph-discovery/spec.md`
|
||||
- `docs/graph-discovery/policy.md`
|
||||
- `docs/graph-discovery/plan.md`
|
||||
- `docs/graph-discovery/prompt.md`
|
||||
|
||||
These docs describe the intended architecture, but only some pieces are backed
|
||||
by live Atlas collections today.
|
||||
|
||||
## Verification Summary
|
||||
|
||||
Atlas connection is valid through local `.env` `MONGODB_URI`. No secrets were
|
||||
printed during verification.
|
||||
|
||||
Public API checks:
|
||||
|
||||
- `https://165-22-129-249.sslip.io/health` returned `200` with `{ "ok": true }`.
|
||||
- `GET /api/pods` returned one real pod: `demo-pod`.
|
||||
- `GET /api/pods/demo-pod/graph` returned a live materialized Team memory graph.
|
||||
|
||||
Live graph response for `demo-pod`:
|
||||
|
||||
- Nodes: `31`
|
||||
- Edges: `59`
|
||||
- Learned owners metric: `2`
|
||||
- Open risk paths metric: `2`
|
||||
- Accept rate metric: `39%`
|
||||
- Learning loop active step: `adapt`
|
||||
- Learned edges: `2` `learned_from` edges
|
||||
- Activity stream populated from real records
|
||||
|
||||
Important conclusion:
|
||||
|
||||
The main Team memory graph endpoint is real and backed by Atlas live
|
||||
materialization. It is not merely returning the demo fallback.
|
||||
|
||||
## Atlas Collection Snapshot
|
||||
|
||||
Approximate counts observed:
|
||||
|
||||
| Collection | Count / status |
|
||||
| --- | ---: |
|
||||
| `pods` | 1 |
|
||||
| `engineer_states` | 9 |
|
||||
| `observations` | 2318 |
|
||||
| `collisions` | 451 |
|
||||
| `interventions` | 362 |
|
||||
| `outcomes` | 107 |
|
||||
| `team_model` | 65 |
|
||||
| `graph_nodes` | 715 |
|
||||
| `graph_edges` | 845 |
|
||||
| `hermes_jobs` | 29 |
|
||||
| `hermes_job_events` | 337 |
|
||||
| `memory_vectors` | missing |
|
||||
| `agent_runs` | missing |
|
||||
| `agent_trace_events` | missing |
|
||||
| `strategy_versions` | missing |
|
||||
| `learning_proposals` | missing |
|
||||
|
||||
Real pod:
|
||||
|
||||
```text
|
||||
demo-pod
|
||||
name: demo pod
|
||||
members: ram, Karti, yahya, shakthi
|
||||
```
|
||||
|
||||
Noise observed:
|
||||
|
||||
- Many `verify-pod-*` records.
|
||||
- Many `verify-graph-*` records.
|
||||
- `Verify ...` observations inside `demo-pod`.
|
||||
- Orphaned verify outcomes in `demo-pod`.
|
||||
- Some stale or duplicate engineer state casing, e.g. `Shakthi` and `shakthi`.
|
||||
- `frontend-pod` outcomes with no corresponding active pod.
|
||||
|
||||
## What Is Real Today
|
||||
|
||||
The following are real and active:
|
||||
|
||||
- Atlas connectivity.
|
||||
- `demo-pod` pod record.
|
||||
- Real `engineer_states` for demo members.
|
||||
- Real observation/collision/intervention/outcome collections.
|
||||
- Live graph materialization from source collections.
|
||||
- Graph response `loop` and `activity` fields.
|
||||
- Real `learned_from` edges produced by accepted real outcomes that still join
|
||||
back to surviving intervention/collision records.
|
||||
|
||||
The following are not yet real:
|
||||
|
||||
- `memory_vectors` collection.
|
||||
- `agent_runs` collection.
|
||||
- `agent_trace_events` collection.
|
||||
- `strategy_versions` collection.
|
||||
- `learning_proposals` collection.
|
||||
|
||||
That means the continual-learning story is currently supported by exact MongoDB
|
||||
records and graph edges. The richer agent-learning spec is documented but not
|
||||
implemented in Atlas yet.
|
||||
|
||||
## Claude Code Review Addendum
|
||||
|
||||
Source: pasted Claude Code review text approved for Codex to read. The review
|
||||
was treated as input, then reconciled against current `main` and the Atlas check
|
||||
above. Do not copy the review blindly; a few findings were from an older repo
|
||||
state or have since been superseded.
|
||||
|
||||
### Still Material Findings
|
||||
|
||||
The review correctly identifies the main mismatch:
|
||||
|
||||
```text
|
||||
The docs describe a broader self-improving platform, while the shipped product
|
||||
currently has a narrower but real recall-and-policy loop.
|
||||
```
|
||||
|
||||
The shipped loop lives in code, not in the aspirational agent-learning docs:
|
||||
|
||||
- `backend/src/agent/podman.ts`
|
||||
- Calls `recallSimilar(collision)`.
|
||||
- If prior memory exists, bumps severity to `critical`.
|
||||
- Calls `shouldIntervene(collision, prior)`.
|
||||
- Calls `preferredAction(collision, prior)`.
|
||||
- Adds the visible message suffix `Seen before.` when prior memory exists.
|
||||
- `backend/src/memory/policy.ts`
|
||||
- Suppresses known false-positive prior outcomes.
|
||||
- Enforces pod cooldown.
|
||||
- Reuses a prior accepted intervention action when available.
|
||||
- `backend/src/memory/vectors.ts`
|
||||
- Stores `memorySignature`, `memoryText`, and optional `embedding` on
|
||||
`collisions`.
|
||||
- `recallSimilar` tries vector recall first, then signature/file fallback.
|
||||
|
||||
This is the real recursive/self-improving asset today:
|
||||
|
||||
```text
|
||||
new collision -> recall prior collision -> adjust severity/action/message ->
|
||||
record outcome -> future collision changes behavior
|
||||
```
|
||||
|
||||
The current docs should eventually be reconciled around this loop instead of
|
||||
implying the full agent-learning platform already exists.
|
||||
|
||||
### Confirmed Aspirational Areas
|
||||
|
||||
The following are documented but not live in Atlas/code yet:
|
||||
|
||||
- `agent_runs`
|
||||
- `agent_trace_events`
|
||||
- `strategy_versions`
|
||||
- `learning_proposals`
|
||||
- `memory_vectors`
|
||||
|
||||
The `agent-learning` docs should be treated as future architecture unless a
|
||||
small, explicit slice is implemented. For demo purposes, do not build a broad
|
||||
strategy-versioning platform. If time allows, the smallest credible slice is one
|
||||
stored policy/strategy row that explains a concrete behavior change.
|
||||
|
||||
### Outcome Write Caveat
|
||||
|
||||
`backend/src/memory/store.ts` `recordOutcome` currently:
|
||||
|
||||
- inserts the outcome into `outcomes`;
|
||||
- updates the intervention status to `accepted` or `dismissed`.
|
||||
|
||||
It does not currently persist `team_model.ownership`. The live graph can still
|
||||
derive `owns` and `learned_from` from accepted real outcomes at read time, but a
|
||||
literal "before/after MongoDB ownership write" does not exist yet.
|
||||
|
||||
If the demo needs a concrete durable ownership diff, add a small explicit write
|
||||
on accepted real outcomes:
|
||||
|
||||
```text
|
||||
accepted && wasRealCollision -> team_model.ownership[normalizedFile] = learnedOwner
|
||||
```
|
||||
|
||||
That should be a separate code task, not part of the DB cleanup unless the user
|
||||
explicitly asks.
|
||||
|
||||
### Vector Recall Caveat
|
||||
|
||||
The docs often say "exact recall first." Current code does the reverse:
|
||||
|
||||
```text
|
||||
recallSimilar = vector recall first, then signature/file fallback
|
||||
```
|
||||
|
||||
Also:
|
||||
|
||||
- Embeddings live on `collisions`, not `memory_vectors`.
|
||||
- Atlas vector index name in code is `collision_embedding`.
|
||||
- Gemini embedding calls request `outputDimensionality: 768`.
|
||||
- Voyage embeddings may have a different dimensionality depending on model.
|
||||
|
||||
For the hackathon demo, exact/signature/file fallback is the reliable story.
|
||||
Vector recall should remain nice-to-have unless Atlas index configuration is
|
||||
verified.
|
||||
|
||||
### Demo Script Caveat
|
||||
|
||||
`docs/demo-setup.md` is stale relative to the Team memory observatory demo. It
|
||||
still describes an older Hermes/voice/blocker flow and does not script:
|
||||
|
||||
- graph observatory;
|
||||
- collision -> intervention -> outcome;
|
||||
- `learned_from`;
|
||||
- run 1 vs run 2 changed behavior.
|
||||
|
||||
Before stage rehearsal, rewrite `docs/demo-setup.md` around the actual
|
||||
observatory path.
|
||||
|
||||
### Superseded Review Findings
|
||||
|
||||
The pasted review included two findings that must be treated carefully:
|
||||
|
||||
- It claimed the current `shared/src/graph.ts` had an older `LearningStage` /
|
||||
`ActivityEvent.text` contract. Current `main` uses `PodLearningLoop` with
|
||||
`activeStep`, step `status`, and `PodGraphActivity` with `title` / `detail`.
|
||||
Always check `shared/src/graph.ts` before editing specs.
|
||||
- It claimed the hero `learned_from` edge did not render on `demo-pod`. The
|
||||
current Atlas/public API check returned two live `learned_from` edges. The
|
||||
risk is still real if cleanup deletes accepted real outcomes or their joined
|
||||
collision/intervention records. Preserve the intact accepted chains.
|
||||
|
||||
### Priority Reconciliation Tasks
|
||||
|
||||
After the DB cleanup script, the next documentation/code priorities should be:
|
||||
|
||||
1. Rewrite `docs/demo-setup.md` as the canonical graph observatory demo script.
|
||||
2. Add a short `docs/recursive-loop.md` or equivalent section that names the
|
||||
real shipped loop in `podman.ts`, `policy.ts`, and `vectors.ts`.
|
||||
3. Mark agent-learning collections and strategy versioning as not-yet-built
|
||||
unless implemented.
|
||||
4. Reconcile vector-recall language in docs with current `vectors.ts`.
|
||||
5. Optionally add the `recordOutcome` ownership write if a durable ownership
|
||||
diff is needed for judging.
|
||||
|
||||
## Main Data Issue
|
||||
|
||||
The live graph and the normalized graph mirror are out of sync.
|
||||
|
||||
Live materializer for `demo-pod`:
|
||||
|
||||
```text
|
||||
31 nodes
|
||||
59 edges
|
||||
2 learned_from edges
|
||||
```
|
||||
|
||||
Normalized mirror in `graph_nodes` / `graph_edges` for `demo-pod`:
|
||||
|
||||
```text
|
||||
11 seeded/demo-style nodes
|
||||
13 seeded/demo-style edges
|
||||
```
|
||||
|
||||
Impact:
|
||||
|
||||
- `GET /api/pods/demo-pod/graph` is good and real.
|
||||
- `GET /api/pods/demo-pod/graph/reach/:nodeId` uses `graph_edges`, so it can
|
||||
return stale seeded paths.
|
||||
- Example observed:
|
||||
- `/graph/reach/engineer:karti` returned a seeded path.
|
||||
- `/graph/reach/engineer:ram` returned `0`, even though Ram is present in the
|
||||
live graph.
|
||||
|
||||
The cleanup should therefore include a mirror rebuild after deleting test data.
|
||||
|
||||
## Relevant Code Paths
|
||||
|
||||
Graph and MongoDB:
|
||||
|
||||
- `backend/src/graph/live.ts`
|
||||
- Live materializer.
|
||||
- Reads `pods`, `engineer_states`, `observations`, `collisions`,
|
||||
`interventions`, `outcomes`, and `team_model`.
|
||||
- Produces `nodes`, `edges`, `metrics`, `loop`, and `activity`.
|
||||
- `backend/src/graph/store.ts`
|
||||
- `loadPodGraph`: live materializer first, then seeded `team_model.graph`,
|
||||
then demo fallback.
|
||||
- `seedGraph`: writes seeded graph into `team_model`, `graph_nodes`,
|
||||
`graph_edges`.
|
||||
- `reachFrom`: uses `$graphLookup` over `graph_edges`.
|
||||
- `backend/src/memory/db.ts`
|
||||
- MongoDB connection and core collection helpers.
|
||||
- `shared/src/graph.ts`
|
||||
- Public graph contract including optional `loop` and `activity`.
|
||||
|
||||
Docs:
|
||||
|
||||
- `docs/mongodb.md`
|
||||
- `docs/graph.md`
|
||||
- `docs/graph-discovery/`
|
||||
- `docs/continual-learning/`
|
||||
- `docs/agent-learning/`
|
||||
|
||||
## DB Cleanup Goal
|
||||
|
||||
Get Atlas into a demo-stable state:
|
||||
|
||||
1. Preserve real `demo-pod` learning history.
|
||||
2. Remove verification/orphan/test records.
|
||||
3. Rebuild `graph_nodes` and `graph_edges` from the live materialized graph.
|
||||
4. Keep cleanup repeatable and reversible.
|
||||
5. Avoid ad hoc shell deletes.
|
||||
|
||||
## Required Safety Rule
|
||||
|
||||
Take a backup before deleting anything.
|
||||
|
||||
```bash
|
||||
mongodump "$MONGODB_URI" --archive=podman-before-cleanup.archive --gzip
|
||||
```
|
||||
|
||||
Do not commit the archive.
|
||||
|
||||
## Cleanup Keep Set
|
||||
|
||||
Start with this conservative keep set:
|
||||
|
||||
```js
|
||||
const keepPods = ["demo-pod"];
|
||||
```
|
||||
|
||||
Records with `podId` outside this set are cleanup candidates unless there is a
|
||||
specific reason to preserve them.
|
||||
|
||||
## Phase 1: Dry-Run Counts
|
||||
|
||||
Write a script that defaults to dry-run. It should print counts only.
|
||||
|
||||
Candidate file:
|
||||
|
||||
```text
|
||||
scripts/db-cleanup.mjs
|
||||
```
|
||||
|
||||
Default behavior:
|
||||
|
||||
```bash
|
||||
node scripts/db-cleanup.mjs --dry-run
|
||||
```
|
||||
|
||||
Apply behavior:
|
||||
|
||||
```bash
|
||||
node scripts/db-cleanup.mjs --apply
|
||||
```
|
||||
|
||||
The script must not delete anything unless `--apply` is present.
|
||||
|
||||
## Phase 2: Remove Orphan/Test Pod Data
|
||||
|
||||
Delete records whose `podId` is not in `keepPods`.
|
||||
|
||||
Collections:
|
||||
|
||||
- `engineer_states`
|
||||
- `observations`
|
||||
- `collisions`
|
||||
- `interventions`
|
||||
- `outcomes`
|
||||
- `team_model`
|
||||
- `graph_nodes`
|
||||
- `graph_edges`
|
||||
- `hermes_jobs`
|
||||
- `hermes_job_events`
|
||||
|
||||
Filter:
|
||||
|
||||
```js
|
||||
{ podId: { $nin: ["demo-pod"] } }
|
||||
```
|
||||
|
||||
Note:
|
||||
|
||||
Some `pods` documents may use `id` instead of `podId`. For `pods`, do not use
|
||||
the filter above. Keep the document with `id: "demo-pod"` and delete obvious
|
||||
test pods only if they exist.
|
||||
|
||||
## Phase 3: Clean Demo-Pod Verification Artifacts
|
||||
|
||||
Within `demo-pod`, delete only obvious verification records.
|
||||
|
||||
### Observations
|
||||
|
||||
```js
|
||||
{
|
||||
podId: "demo-pod",
|
||||
$or: [
|
||||
{ engineerId: /^Verify\b/ },
|
||||
{ currentFile: /^PodMan verification screen$/ },
|
||||
{ currentFile: /^frame \d+$/ }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Outcomes
|
||||
|
||||
```js
|
||||
{
|
||||
podId: "demo-pod",
|
||||
$or: [
|
||||
{ interventionId: /^int-verify-/ },
|
||||
{ collisionId: /^col-verify-/ }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Collisions and Interventions
|
||||
|
||||
Be more conservative. Delete only records that clearly have verify IDs or no
|
||||
matching counterpart.
|
||||
|
||||
Safe candidate filters:
|
||||
|
||||
```js
|
||||
// collisions
|
||||
{
|
||||
podId: "demo-pod",
|
||||
id: /^col-verify-/
|
||||
}
|
||||
|
||||
// interventions
|
||||
{
|
||||
podId: "demo-pod",
|
||||
id: /^int-verify-/
|
||||
}
|
||||
```
|
||||
|
||||
Optional orphan cleanup:
|
||||
|
||||
- Delete interventions whose `collisionId` does not exist in `collisions`.
|
||||
- Delete outcomes whose `interventionId` does not exist in `interventions` and
|
||||
whose `collisionId` does not exist in `collisions`.
|
||||
|
||||
Run orphan cleanup only after dry-run prints exact IDs and counts.
|
||||
|
||||
## Phase 4: Normalize Demo-Pod Engineer State
|
||||
|
||||
Keep canonical active engineers:
|
||||
|
||||
```text
|
||||
ram
|
||||
Karti
|
||||
yahya
|
||||
shakthi
|
||||
```
|
||||
|
||||
Cleanup candidates:
|
||||
|
||||
```js
|
||||
{
|
||||
podId: "demo-pod",
|
||||
$or: [
|
||||
{ name: /^Verify\b/ },
|
||||
{ name: /^codex-check$/i },
|
||||
{ name: /^testrepo/i },
|
||||
{ name: "Shakthi" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Only delete `Shakthi` if `shakthi` is confirmed as the canonical current record.
|
||||
|
||||
## Phase 5: Rebuild Graph Mirror
|
||||
|
||||
This is the most important post-cleanup step.
|
||||
|
||||
The live graph endpoint is real, but reachability uses stale mirrored records.
|
||||
After cleanup:
|
||||
|
||||
1. Materialize the live graph for `demo-pod`.
|
||||
2. Delete mirrored rows for `demo-pod`.
|
||||
3. Insert live graph nodes into `graph_nodes`.
|
||||
4. Insert live graph edges into `graph_edges`.
|
||||
5. Update `team_model.graph` and `team_model.updatedAt`.
|
||||
|
||||
Pseudocode:
|
||||
|
||||
```js
|
||||
const graph = await materializePodGraph("demo-pod");
|
||||
|
||||
await db.collection("graph_nodes").deleteMany({ podId: "demo-pod" });
|
||||
await db.collection("graph_edges").deleteMany({ podId: "demo-pod" });
|
||||
|
||||
await db.collection("graph_nodes").insertMany(
|
||||
graph.nodes.map((node) => ({ ...node, podId: "demo-pod" }))
|
||||
);
|
||||
|
||||
await db.collection("graph_edges").insertMany(
|
||||
graph.edges.map((edge) => ({ ...edge, podId: "demo-pod" }))
|
||||
);
|
||||
|
||||
await db.collection("team_model").updateOne(
|
||||
{ podId: "demo-pod" },
|
||||
{ $set: { podId: "demo-pod", graph, updatedAt: new Date().toISOString() } },
|
||||
{ upsert: true }
|
||||
);
|
||||
```
|
||||
|
||||
Important:
|
||||
|
||||
Use `materializePodGraph`, not `createDemoPodGraph`, for this rebuild.
|
||||
`seedGraph` currently writes a demo graph and would recreate the stale mismatch.
|
||||
|
||||
## Phase 6: Add Helpful Indexes
|
||||
|
||||
Current `initMemory` creates the core indexes, but cleanup/reachability benefits
|
||||
from these as well:
|
||||
|
||||
```js
|
||||
db.graph_nodes.createIndex({ podId: 1, id: 1 }, { unique: true });
|
||||
db.graph_edges.createIndex({ podId: 1, id: 1 }, { unique: true });
|
||||
db.graph_edges.createIndex({ podId: 1, source: 1 });
|
||||
db.graph_edges.createIndex({ podId: 1, target: 1 });
|
||||
db.graph_edges.createIndex({ podId: 1, kind: 1 });
|
||||
db.team_model.createIndex({ podId: 1 }, { unique: true });
|
||||
db.collisions.createIndex({ podId: 1, memorySignature: 1 });
|
||||
db.outcomes.createIndex({ podId: 1, interventionId: 1 });
|
||||
db.interventions.createIndex({ podId: 1, collisionId: 1 });
|
||||
```
|
||||
|
||||
Make index creation idempotent.
|
||||
|
||||
## Validation After Cleanup
|
||||
|
||||
Run these checks after `--apply`.
|
||||
|
||||
### Atlas counts
|
||||
|
||||
Confirm:
|
||||
|
||||
- No `verify-pod-*` pod data remains.
|
||||
- No `verify-graph-*` team models or graph rows remain.
|
||||
- `demo-pod` still has meaningful observations, collisions, interventions, and
|
||||
outcomes.
|
||||
|
||||
### Public API
|
||||
|
||||
```bash
|
||||
curl https://165-22-129-249.sslip.io/health
|
||||
curl https://165-22-129-249.sslip.io/api/pods
|
||||
curl https://165-22-129-249.sslip.io/api/pods/demo-pod/graph
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- Health is `ok`.
|
||||
- `demo-pod` still exists.
|
||||
- Graph has nodes, edges, metrics, loop, activity.
|
||||
- Graph includes `learned_from` if accepted real outcomes remain.
|
||||
|
||||
### Reachability
|
||||
|
||||
After mirror rebuild, these should reflect the live graph, not old seed data:
|
||||
|
||||
```bash
|
||||
curl "https://165-22-129-249.sslip.io/api/pods/demo-pod/graph/reach/engineer%3Aram"
|
||||
curl "https://165-22-129-249.sslip.io/api/pods/demo-pod/graph/reach/engineer%3Ayahya"
|
||||
curl "https://165-22-129-249.sslip.io/api/pods/demo-pod/graph/reach/engineer%3Akarti"
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- At least engineers present in the live graph should have reachable edges when
|
||||
they have outbound graph edges.
|
||||
|
||||
## Known Tooling Caveat
|
||||
|
||||
One `pnpm exec` verification attempt was blocked by supply-chain policy:
|
||||
|
||||
```text
|
||||
prettier@3.9.0 was within the minimumReleaseAge cutoff
|
||||
```
|
||||
|
||||
This did not indicate a MongoDB or graph failure. Direct MongoDB reads and the
|
||||
existing local `backend/node_modules/.bin/tsx` binary were used instead.
|
||||
|
||||
Do not run broad dependency cleanup during DB cleanup unless explicitly asked.
|
||||
|
||||
## Recommended Cleanup Script Shape
|
||||
|
||||
The script should:
|
||||
|
||||
- Load `.env` with `dotenv`.
|
||||
- Require `MONGODB_URI`.
|
||||
- Print the database name.
|
||||
- Refuse to run against a DB whose name is not `podman` unless `--force-db` is
|
||||
passed.
|
||||
- Default to `--dry-run`.
|
||||
- Require `--apply` for deletes.
|
||||
- Print every collection and matched count before deleting.
|
||||
- Never print `MONGODB_URI`.
|
||||
- Rebuild graph mirror only after delete phase succeeds.
|
||||
- Print before/after counts.
|
||||
|
||||
Suggested flags:
|
||||
|
||||
```text
|
||||
--dry-run
|
||||
--apply
|
||||
--skip-mirror-rebuild
|
||||
--keep-pod demo-pod
|
||||
--force-db
|
||||
```
|
||||
|
||||
## Do Not Do
|
||||
|
||||
- Do not run `seedGraph("demo-pod")` as the fix; that writes the demo graph.
|
||||
- Do not delete all `outcomes`; accepted and dismissed outcomes are learning
|
||||
signals.
|
||||
- Do not delete all `team_model`; preserve or rebuild the `demo-pod` document.
|
||||
- Do not print secrets or raw terminal/screenshot content.
|
||||
- Do not rely on vector collections for the current demo; they are absent.
|
||||
- Do not treat seeded graph mirror data as proof of current live learning.
|
||||
|
||||
## Suggested Next Codex Prompt
|
||||
|
||||
Use this prompt for the implementation pass:
|
||||
|
||||
```text
|
||||
Create a safe MongoDB cleanup script for PodMan.
|
||||
|
||||
Read docs/handoff/README.md, docs/mongodb.md, docs/graph.md,
|
||||
docs/continual-learning/spec.md, backend/src/graph/live.ts,
|
||||
backend/src/graph/store.ts, backend/src/memory/store.ts,
|
||||
backend/src/memory/policy.ts, backend/src/memory/vectors.ts, and
|
||||
backend/src/agent/podman.ts before coding.
|
||||
|
||||
Implement scripts/db-cleanup.mjs with --dry-run default and --apply required for
|
||||
deletes. Keep demo-pod, remove verify/orphan pod data, remove obvious demo-pod
|
||||
verification artifacts, and rebuild graph_nodes/graph_edges/team_model.graph for
|
||||
demo-pod from materializePodGraph, not createDemoPodGraph. Do not print secrets.
|
||||
Run dry-run first and show counts before applying.
|
||||
|
||||
Do not implement broad agent-learning infrastructure in this task. The real
|
||||
shipped RSI loop today is recallSimilar -> shouldIntervene/preferredAction ->
|
||||
outcome -> future recall. Preserve accepted real outcome chains so learned_from
|
||||
continues to render.
|
||||
```
|
||||
+101
@@ -0,0 +1,101 @@
|
||||
# Hermes Spec
|
||||
|
||||
Status: active / matches code.
|
||||
|
||||
"Hermes" is PodMan's **action layer** — the part that turns a detected problem
|
||||
into something a teammate sees, hears, or gets done. It spans three things:
|
||||
|
||||
1. **Interventions** — cards, messages, and urgent voice in the pod room.
|
||||
2. **Async jobs** — longer tasks delegated from the live conversation agent.
|
||||
3. **Ops watchdog** — keeps the production services healthy.
|
||||
|
||||
The LiveKit identity for the main agent is `podman-hermes`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Interventions
|
||||
|
||||
**Code:** `backend/src/agent/podman.ts`, `backend/src/action/hermes.ts`,
|
||||
`backend/src/voice/live.ts`.
|
||||
|
||||
When the agent detects a collision, it runs the learning loop (recall → policy
|
||||
gate; see `docs/cont_learning.md`) and then publishes the **least intrusive**
|
||||
intervention that fits:
|
||||
|
||||
- **Card / message** — a data-channel packet on the `podman.intervention` topic
|
||||
(`publishHermesIntervention` / `publishHermesMessage`). Default path.
|
||||
- **Urgent voice** — only for `critical` collisions. `speak()` generates Gemini
|
||||
TTS audio and publishes it as a LiveKit audio track.
|
||||
|
||||
Intervention text is short and deterministic (template, not an LLM call):
|
||||
`Conflict: alice + bob both on detector.ts (unpushed). Seen before.` The spoken
|
||||
line is phrased for natural TTS prosody. Each intervention is persisted to the
|
||||
`interventions` collection; the teammate's accept/dismiss returns via
|
||||
`POST /api/outcome`.
|
||||
|
||||
A per-pod cooldown (`NUDGE_COOLDOWN_MS`, default 3 min) and a single-shot
|
||||
"active conflict" guard prevent repeat nagging; a conflict re-arms once it
|
||||
resolves.
|
||||
|
||||
---
|
||||
|
||||
## 2. Async Hermes jobs
|
||||
|
||||
**Code:** `backend/src/hermes/jobs.ts`. **Storage:** `hermes_jobs` +
|
||||
`hermes_job_events` (see `docs/mongodb.md`).
|
||||
|
||||
The live conversation agent can hand a longer task to Hermes via its
|
||||
`delegate_to_hermes` tool. Lifecycle:
|
||||
|
||||
```
|
||||
queued → running → (waiting_for_confirmation) → completed | aborted | failed
|
||||
```
|
||||
|
||||
`createHermesJob()` records the job, emits an `accepted` event, and kicks off
|
||||
`runHermesJob()` in the background. The runner gathers context and runs scoped,
|
||||
read-mostly steps based on the prompt and success criteria:
|
||||
|
||||
- always: `git status --short --branch`, `git diff --stat`
|
||||
- if the ask mentions GitHub: a repo reachability check via the GitHub API
|
||||
- if it mentions Mongo/memory/telemetry: collection counts
|
||||
- if it mentions build/test/typecheck/broken: `pnpm typecheck`
|
||||
|
||||
**Confirmation gate:** if `riskLevel === 'deploy_allowed'` and
|
||||
`requiresConfirmation`, the job parks at `waiting_for_confirmation` instead of
|
||||
acting. **Abort:** `abortHermesJob()` signals the runner's `AbortController`.
|
||||
|
||||
Every step appends a `hermes_job_event` (redacted + truncated), which is both
|
||||
stored and published live to the room as a `HERMES_JOB_EVENT` data message from a
|
||||
short-lived `podman-hermes-job-*` identity. The conversation UI streams these via
|
||||
`GET /api/.../hermes-job/events/stream`.
|
||||
|
||||
**Endpoints:** `POST /api/internal/hermes/jobs`,
|
||||
`GET /api/internal/hermes/jobs/:jobId`, `.../abort`, `.../events`,
|
||||
`.../events/stream`, plus the pod-scoped `.../live-conversation/:sessionId/hermes-job`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Ops watchdog
|
||||
|
||||
**Code:** `scripts/hermes-watchdog.mjs`, `scripts/hermes-sync-deploy.mjs`,
|
||||
`scripts/hermes-notify.mjs`. **Detail:** `docs/digitalocean.md`.
|
||||
|
||||
systemd supervises the app processes; Hermes owns the loop around them:
|
||||
|
||||
- `pnpm hermes:watchdog` checks systemd services, public routes, `/health`,
|
||||
`/api/pods`, and `pnpm deploy:doctor`. Failures trigger targeted restarts.
|
||||
- `podman-hermes-watchdog.timer` runs it every 5 minutes.
|
||||
- `podman-hermes-sync-deploy.timer` polls `origin/main` every 2 minutes and, on a
|
||||
clean tree, fast-forwards, builds, publishes `frontend/dist`, restarts
|
||||
API/agent/Caddy, and runs the strict watchdog.
|
||||
- Reports go to `/var/log/podman/hermes-watchdog-latest.json`; set
|
||||
`PODMAN_ALERT_WEBHOOK_URL` to forward failures to Discord/Slack/webhook.
|
||||
|
||||
---
|
||||
|
||||
## What Hermes is NOT
|
||||
|
||||
- Not an autonomous code-writing agent. Job steps are scoped, read-mostly checks;
|
||||
deploy-level actions require explicit confirmation.
|
||||
- Not a second collision detector. Detection is deterministic
|
||||
(`collision/detector.ts`); Hermes only acts on the result.
|
||||
@@ -1,93 +0,0 @@
|
||||
# PodMan — Idea
|
||||
|
||||
## One-line value prop
|
||||
|
||||
PodMan is a real-time AI team coordination agent that watches consented work signals, maintains live project memory, and proactively coordinates collaborators when collisions, blockers, or handoffs emerge before anyone has to ask.
|
||||
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
Teams working on the same project lose time because progress is fragmented across people, editors, terminals, and half-finished messages. Coordination gaps — a completed endpoint, a resolved blocker, two engineers duplicating work — are discovered too late, causing idle time, broken handoffs, and missed dependencies.
|
||||
|
||||
Slack doesn't help. Stand-ups are too slow. GitHub only knows pushed state.
|
||||
|
||||
---
|
||||
|
||||
## Solution
|
||||
|
||||
PodMan is an ambient AI agent that:
|
||||
|
||||
1. Watches each engineer's consented LiveKit screen-share signal
|
||||
2. Extracts structured context using Gemini Vision — current file, inferred task, terminal state
|
||||
3. Maintains a shared live model in MongoDB Atlas — observations, collisions, interventions, outcomes, and graph memory
|
||||
4. Detects coordination risks: same-file collision, blocker detected, duplicate work
|
||||
5. Sends the least intrusive intervention first: card, Hermes message, and urgent voice only when needed
|
||||
|
||||
**The AI's job is not to chat. It is to notice what teammates miss and say so, exactly when it matters.**
|
||||
|
||||
---
|
||||
|
||||
## Target user
|
||||
|
||||
Small software teams: hackathon squads, startup engineering teams, student dev teams collaborating in real time on a shared codebase.
|
||||
|
||||
---
|
||||
|
||||
## Core AI job
|
||||
|
||||
- Maintain per-person live context (file, task, terminal)
|
||||
- Infer shared project state (who owns what, what's blocked, what's ready)
|
||||
- Detect 3 coordination risk types:
|
||||
- `DEPENDENCY_READY` — engineer A was waiting on work engineer B just completed
|
||||
- `BLOCKER_DETECTED` — engineer appears stuck; another teammate can unblock
|
||||
- `DUPLICATE_WORK` — 2+ engineers working on the same file simultaneously
|
||||
- Generate a short intervention message
|
||||
- Deliver it as a LiveKit data message, with Gemini TTS audio reserved for urgent escalation
|
||||
|
||||
---
|
||||
|
||||
## How it fits the Continual Learning track
|
||||
|
||||
PodMan builds outcome-backed team memory in MongoDB that persists across sessions:
|
||||
|
||||
- Session 1: PodMan observes work, predicts a collision, sends an intervention, and stores the outcome
|
||||
- Session 2+: PodMan recalls the exact signature and changes the graph or behavior
|
||||
|
||||
The system gets demonstrably more useful the more it is used, with no user configuration required. That is the track definition met exactly.
|
||||
|
||||
---
|
||||
|
||||
## Architecture (one paragraph)
|
||||
|
||||
Each engineer opens a browser PWA on their laptop. The PWA shares live IDE context through LiveKit screen sharing, and the local git watcher writes dirty/unpushed state to MongoDB. The backend agent calls Gemini Vision to extract structured context, writes observations and collisions to MongoDB Atlas, recalls accepted or dismissed outcomes, and routes the smallest useful intervention. Cards and Hermes messages are default; Gemini TTS through LiveKit is reserved for urgent escalation. No Slack. No tab switching. No interruption to the editor flow.
|
||||
|
||||
---
|
||||
|
||||
## Demo wow moment
|
||||
|
||||
> Alice is building the auth endpoint. Carol is visibly blocked — her terminal shows `connection refused`. PodMan detects the blocker and says aloud: "Carol, looks like you're waiting on auth. Alice is actively building it — hang tight."
|
||||
>
|
||||
> Two minutes later, Alice's server starts. PodMan says: "Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate."
|
||||
>
|
||||
> Nobody asked. Nobody pinged anyone on Slack. PodMan just knew.
|
||||
|
||||
---
|
||||
|
||||
## What PodMan is NOT
|
||||
|
||||
- Not a chat interface
|
||||
- Not a dashboard product
|
||||
- Not raw surveillance — engineers consent by joining the room and sharing their screen
|
||||
- Not a task manager
|
||||
- Not a GitHub integration (v1)
|
||||
|
||||
---
|
||||
|
||||
## Prize alignment
|
||||
|
||||
| Prize | How PodMan earns it |
|
||||
| --------------------- | ----------------------------------------------------------------------------------------------------- |
|
||||
| Best Gemini 3.5 / 2.5 | Gemini Vision for screen understanding + Gemini TTS for urgent voice output |
|
||||
| Best LiveKit | LiveKit is the real-time backbone for room presence and voice delivery — load-bearing, not decorative |
|
||||
| Best DigitalOcean | Hermes deployed on DigitalOcean App Platform; MongoDB Atlas on DO-adjacent infrastructure |
|
||||
+57
-69
@@ -1,15 +1,24 @@
|
||||
# LiveKit Integration Spec
|
||||
|
||||
LiveKit is the real-time backbone for PodMan. It handles room presence and voice delivery. It is load-bearing — not decorative.
|
||||
Status: active / matches code.
|
||||
|
||||
LiveKit is the real-time backbone for PodMan. It carries the **screen-share
|
||||
perception input**, the **intervention data channel**, and **all room audio**
|
||||
(Gemini TTS escalations, the Lyria score, and the live conversation agent). It is
|
||||
load-bearing, not decorative.
|
||||
|
||||
---
|
||||
|
||||
## Room structure
|
||||
|
||||
- One LiveKit room per project pod: `room = podId`
|
||||
- Engineers join as named participants (e.g. `alice`, `bob`)
|
||||
- Hermes joins as `podman-hermes`
|
||||
- All participants stay connected for the duration of the session
|
||||
- One LiveKit room per pod: `room = podId`.
|
||||
- Engineers join as named participants (e.g. `alice`, `bob`).
|
||||
- PodMan runs **multiple agent identities** in/around a room:
|
||||
- `podman-hermes` — the main vision + intervention agent (`@livekit/rtc-node`).
|
||||
- `podman-live-conversation` — the Gemini Live voice agent (Python).
|
||||
- short-lived `podman-hermes-job-*` publishers for async job events.
|
||||
- A fixed identity matters: a second `podman-hermes` evicts the first and they
|
||||
flap, dropping interventions. systemd keeps exactly one alive in production.
|
||||
|
||||
---
|
||||
|
||||
@@ -17,100 +26,79 @@ LiveKit is the real-time backbone for PodMan. It handles room presence and voice
|
||||
|
||||
**Joining:**
|
||||
|
||||
1. PWA calls `POST /pods/:podId/token` → receives `{ token, url }`
|
||||
2. LiveKit client connects to the room with the token
|
||||
3. PWA publishes screen track via `getDisplayMedia`
|
||||
4. PWA sets mic enabled for ambient presence
|
||||
1. PWA calls `POST /api/token` with `{ podId, identity }` → `{ token, url }`.
|
||||
2. LiveKit client connects with the token.
|
||||
3. PWA publishes the screen track via `getDisplayMedia`.
|
||||
4. PWA enables mic for ambient presence (used by the conversation agent).
|
||||
|
||||
**Receiving:**
|
||||
|
||||
- LiveKit client subscribes to remote Hermes audio tracks and attaches them to
|
||||
a hidden audio sink in the DOM.
|
||||
- Browser autoplay restrictions still apply. The PWA calls `room.startAudio()`
|
||||
from user gestures such as first room click, `Enable audio`, `Test PodMan
|
||||
voice`, and `Share screen`.
|
||||
- PWA also listens for data channel messages from Hermes for UI card updates and
|
||||
`VOICE_CUE` fallback text.
|
||||
- Subscribes to remote agent audio tracks (TTS, Lyria, conversation) and attaches
|
||||
them to a hidden audio sink.
|
||||
- Browser autoplay restrictions apply: the PWA calls `room.startAudio()` from a
|
||||
user gesture (`Enable audio`, `Test PodMan voice`, `Share screen`, first room
|
||||
click).
|
||||
- Listens on the data channel for cards and `VOICE_CUE` fallback text.
|
||||
|
||||
**Data channel listener (PWA):**
|
||||
|
||||
```ts
|
||||
room.on(RoomEvent.DataReceived, (payload, participant) => {
|
||||
if (participant?.identity !== 'podman-hermes') return;
|
||||
const intervention = JSON.parse(new TextDecoder().decode(payload));
|
||||
// intervention: COLLISION, HERMES_MESSAGE, VOICE_CUE, ACK, or GIT_REPORT
|
||||
appendInterventionToFeed(intervention);
|
||||
if (!participant?.identity.startsWith('podman-')) return;
|
||||
const msg = JSON.parse(new TextDecoder().decode(payload));
|
||||
// msg.type: COLLISION | ACK | GIT_REPORT | VOICE_CUE | HERMES_JOB_EVENT
|
||||
appendInterventionToFeed(msg);
|
||||
});
|
||||
```
|
||||
|
||||
All data messages share the `podman.intervention` topic (`DATA_TOPIC`).
|
||||
|
||||
---
|
||||
|
||||
## Hermes side (PodMan LiveKit participant)
|
||||
## Agent side (`podman-hermes`)
|
||||
|
||||
**Framework:** `@livekit/rtc-node`
|
||||
**Framework:** `@livekit/rtc-node`. **Code:** `backend/src/agent/podman.ts`,
|
||||
`backend/src/action/hermes.ts`, `backend/src/voice/live.ts`.
|
||||
|
||||
**Startup:**
|
||||
1. Subscribes to engineers' screen-share tracks and samples frames for Gemini
|
||||
Vision.
|
||||
2. Detects collisions, gates them through the learning policy, then publishes a
|
||||
card/message on the data channel.
|
||||
3. For critical collisions, generates Gemini TTS audio and publishes it as a
|
||||
microphone-source audio track, held for the audio duration plus a tail/hold
|
||||
window so subscribers finish playout. Voice publishing logs frame count,
|
||||
estimated duration, queued playout, and hold time for diagnostics.
|
||||
|
||||
1. Hermes mints its own token via the same `createPodToken` function with `identity: 'podman-hermes'`
|
||||
2. Connects to the configured room as `podman-hermes`
|
||||
3. Publishes data-channel cards/messages and Gemini TTS audio tracks
|
||||
---
|
||||
|
||||
**Voice delivery:**
|
||||
## Live conversation agent (`podman-live-conversation`)
|
||||
|
||||
1. Urgent intervention text is ready (from Gemini text generation)
|
||||
2. Hermes sends a natural-speaking prompt to Gemini TTS
|
||||
3. Gemini returns PCM audio using the configured voice
|
||||
4. Hermes publishes the audio as a LiveKit microphone-source track
|
||||
5. Hermes keeps the track published for the generated audio duration plus tail
|
||||
silence and a hold window. This avoids browser-side cutoff when LiveKit's
|
||||
queued playout signal returns before subscribers finish playing buffered
|
||||
audio.
|
||||
6. All participants hear it after browser audio has been unlocked
|
||||
**Framework:** LiveKit Agents for Python (`AgentSession`, `function_tool`,
|
||||
`google.realtime.RealtimeModel`). **Code:**
|
||||
`agents/podman-live-conversation/agent.py`.
|
||||
|
||||
**Data channel message (sent alongside audio):**
|
||||
|
||||
```ts
|
||||
const intervention = {
|
||||
type: 'DEPENDENCY_READY' | 'BLOCKER_DETECTED' | 'DUPLICATE_WORK',
|
||||
message: string, // the spoken text
|
||||
involvedEngineers: string[],
|
||||
file: string | null,
|
||||
sentAt: string, // ISO timestamp
|
||||
};
|
||||
room.localParticipant.publishData(
|
||||
new TextEncoder().encode(JSON.stringify(intervention)),
|
||||
{ reliable: true }
|
||||
);
|
||||
```
|
||||
Joins the pod room on demand (`POST /api/pods/:id/live-conversation/start`),
|
||||
streams speech-to-speech with Gemini Live, and answers using repo/git/memory
|
||||
function tools. It can delegate long tasks to the async Hermes job runner and
|
||||
narrate progress. See `docs/hermes.md`.
|
||||
|
||||
---
|
||||
|
||||
## Token endpoint
|
||||
|
||||
Already implemented at `POST /api/token`.
|
||||
|
||||
Hermes uses the same endpoint. Grants:
|
||||
`POST /api/token` mints room tokens for engineers and agents alike. Grants:
|
||||
|
||||
- `roomJoin: true`
|
||||
- `canPublish: true` (for audio track)
|
||||
- `canPublishData: true` (for data channel)
|
||||
- `canPublish: true` (audio + screen)
|
||||
- `canPublishData: true` (data channel)
|
||||
- `canSubscribe: true`
|
||||
|
||||
---
|
||||
|
||||
## Gemini voice model
|
||||
|
||||
- Model ID: `gemini-3.1-flash-tts-preview`
|
||||
- Default voice: `Charon` (`GEMINI_TTS_VOICE`)
|
||||
- Hermes generates Gemini TTS audio and publishes it as a LiveKit audio track.
|
||||
- Voice publishing logs generated frame count, estimated duration, queued
|
||||
playout, and the final subscriber hold time for diagnostics.
|
||||
- The backend keeps a Gemini Live path for future model availability, but the verified deployment path uses TTS.
|
||||
Short-lived job publishers use `canSubscribe: false`.
|
||||
|
||||
---
|
||||
|
||||
## What LiveKit does NOT do in PodMan
|
||||
|
||||
- No video tracks from Hermes
|
||||
- No mic transcription (not needed for v1)
|
||||
- No SFU mixing — standard room behavior is sufficient
|
||||
- No video tracks published by agents.
|
||||
- No mic transcription outside the live conversation agent.
|
||||
- No custom SFU mixing — standard room behavior is sufficient.
|
||||
|
||||
+28
-18
@@ -3,17 +3,14 @@
|
||||
Status: demo-backed / active
|
||||
|
||||
MongoDB Atlas is PodMan's shared memory. It stores live work observations,
|
||||
collision predictions, interventions, outcomes, latest engineer state, the
|
||||
materialized Team memory graph, and optional future recall records.
|
||||
collision predictions (with vector embeddings for recall), interventions,
|
||||
outcomes, latest engineer state, the materialized Team memory graph, and async
|
||||
Hermes job runs.
|
||||
|
||||
See also:
|
||||
|
||||
- [`docs/continual-learning/`](continual-learning/) for outcome-backed team
|
||||
memory.
|
||||
- [`docs/graph-discovery/`](graph-discovery/) for graph materialization and
|
||||
`$graphLookup` traversal.
|
||||
- [`docs/agent-learning/`](agent-learning/) for planned strategy-version
|
||||
records.
|
||||
- [`docs/cont_learning.md`](cont_learning.md) for outcome-backed team memory,
|
||||
graph materialization, and `$graphLookup` traversal.
|
||||
|
||||
---
|
||||
|
||||
@@ -62,7 +59,7 @@ graph.
|
||||
|
||||
### `collisions`
|
||||
|
||||
Predicted coordination risks.
|
||||
Predicted coordination risks, with memory enrichment for recall.
|
||||
|
||||
Key fields:
|
||||
|
||||
@@ -75,8 +72,16 @@ Key fields:
|
||||
- `memorySignature`
|
||||
- `githubState`
|
||||
- `detectedAt`
|
||||
- `memoryText` — short text embedded for recall
|
||||
- `embedding` — vector (Voyage `voyage-4-lite` or Gemini `gemini-embedding-001`)
|
||||
- `embeddingProvider` — `voyage` | `gemini`
|
||||
|
||||
Primary use: collision cards, exact signature recall, and graph risk paths.
|
||||
Vector index `collision_embedding` (Atlas Vector Search) powers `$vectorSearch`
|
||||
recall in `backend/src/memory/vectors.ts`. When Atlas vector search is
|
||||
unavailable, recall falls back to app-side cosine, then exact signature/file
|
||||
matching.
|
||||
|
||||
Primary use: collision cards, vector + signature recall, and graph risk paths.
|
||||
|
||||
### `interventions`
|
||||
|
||||
@@ -138,16 +143,21 @@ Indexes:
|
||||
|
||||
Primary use: `GET /api/pods/:podId/graph/reach/:id` with `$graphLookup`.
|
||||
|
||||
### Optional Future Collections
|
||||
### `hermes_jobs` and `hermes_job_events`
|
||||
|
||||
These are documented for planned work and should not be treated as active write
|
||||
paths unless implementation is added:
|
||||
Async Hermes task runs delegated from the live conversation agent (see
|
||||
`docs/hermes.md`).
|
||||
|
||||
- `memory_vectors`
|
||||
- `agent_runs`
|
||||
- `agent_trace_events`
|
||||
- `strategy_versions`
|
||||
- `learning_proposals`
|
||||
- `hermes_jobs` — one doc per job (`id` unique; `{ sessionId, status, updatedAt }`
|
||||
index). Fields: `id`, `podId`, `sessionId`, `prompt`, `contextScope`,
|
||||
`riskLevel`, `successCriteria`, `status`, `finalSummary`, timestamps.
|
||||
- `hermes_job_events` — append-only step log (`{ jobId, createdAt }` index):
|
||||
`accepted`, `heartbeat`, `step_started`, `step_output`, `needs_confirmation`,
|
||||
`step_completed`, `completed`, `aborted`, `failed`. Output is redacted +
|
||||
truncated before storage and mirrored to the room over LiveKit.
|
||||
|
||||
Primary use: durable, replayable record of what Hermes did, streamed live to the
|
||||
conversation UI.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1,108 +0,0 @@
|
||||
# Stream Categorization — My Stream / Team Stream
|
||||
|
||||
## Why
|
||||
|
||||
Judges must read the pod stream in 10 seconds. Right now both lanes (`My stream`,
|
||||
`Team stream`) dump every event into one flat chronological list. The two things
|
||||
that prove "self-improving agent" are mushed together:
|
||||
|
||||
- **Sources of decisions** — raw signals the agent observed (screen vision, git).
|
||||
- **Reasoning decisions** — what Hermes concluded and did (conflict detected,
|
||||
intervention spoken, outcome/verifier result).
|
||||
|
||||
`source` (vision/git/memory/hermes/policy) is currently buried as a plain outline
|
||||
badge next to the filename. `kind` is only an icon. No sectioning. Result: bloated,
|
||||
undifferentiated, doesn't tell the loop story (observe → reason → act → learn).
|
||||
|
||||
## Goal
|
||||
|
||||
Split each stream lane into clear sections and promote provenance, so a judge sees:
|
||||
"agent ingests **signals**, then makes **reasoning decisions** from them."
|
||||
|
||||
No backend / shared-type changes. The data already carries `kind` + `source`.
|
||||
Pure presentation change in `frontend/src/components/PodView.tsx` —
|
||||
`ActivitySidebar` + `ActivityItem` + new helpers only. Additive, localized.
|
||||
|
||||
## Categorization (the contract)
|
||||
|
||||
Two sections, derived from existing `PodActivityKind`:
|
||||
|
||||
| Section | Heading | kinds | Meaning |
|
||||
|---|---|---|---|
|
||||
| `signal` | **Signals** | `observation`, `git` | Raw inputs the agent saw. The *sources*. |
|
||||
| `decision` | **Reasoning & decisions** | `collision`, `intervention`, `outcome` | What Hermes reasoned and did. |
|
||||
|
||||
```ts
|
||||
const CATEGORY_OF: Record<PodActivityKind, 'signal' | 'decision'> = {
|
||||
observation: 'signal',
|
||||
git: 'signal',
|
||||
collision: 'decision',
|
||||
intervention: 'decision',
|
||||
outcome: 'decision',
|
||||
};
|
||||
```
|
||||
|
||||
Section order: **Signals** first, **Reasoning & decisions** second (top-to-bottom =
|
||||
the loop direction). A section with zero events renders nothing.
|
||||
|
||||
## Provenance chip (sources)
|
||||
|
||||
Promote `source` to a leading color-coded chip with an icon. This is the "source of
|
||||
decision" tag judges look for.
|
||||
|
||||
| source | label | icon | tint |
|
||||
|---|---|---|---|
|
||||
| `vision` | Vision | `EyeIcon` | chart-1 |
|
||||
| `git` | Git | `GitBranchIcon` | chart-2 |
|
||||
| `memory` | Memory | `BrainIcon` | chart-4 |
|
||||
| `hermes` | Hermes | `SparklesIcon` | primary |
|
||||
| `policy` | Policy | `ShieldIcon` | chart-3 |
|
||||
|
||||
Chip class pattern (use `variant="outline"` so the default primary fill is overridden):
|
||||
`border-{tint}/40 bg-{tint}/10 text-{tint}`.
|
||||
|
||||
## Kind label
|
||||
|
||||
Render `kind` as readable text next to the provenance chip, not just an icon:
|
||||
|
||||
| kind | label |
|
||||
|---|---|
|
||||
| observation | Observed |
|
||||
| git | Git |
|
||||
| collision | Conflict |
|
||||
| intervention | Intervention |
|
||||
| outcome | Outcome |
|
||||
|
||||
Tag row order in each card: **source chip → kind label → actors → file**.
|
||||
|
||||
## Implementation steps (after teammate lands frontend work)
|
||||
|
||||
1. **Rebase / pull teammate's PodView.tsx first.** Do not start before it lands —
|
||||
this file is being actively rewritten right now.
|
||||
2. Add icon imports: `BrainIcon`, `EyeIcon`, `ShieldIcon`, `WorkflowIcon` (and
|
||||
reuse `GitBranchIcon`, `SparklesIcon`, `RadioTowerIcon`). Import
|
||||
`PodActivitySource` type from `@podman/shared`.
|
||||
3. Add module-level consts: `CATEGORY_OF`, `CATEGORIES` (id/label/hint/icon),
|
||||
`SOURCE_META` (label/icon/className), `KIND_LABEL`.
|
||||
4. In `ActivitySidebar`'s expanded `SidebarContent`, replace the flat
|
||||
`events.map(...)` with a `CATEGORIES.map(...)` that filters events per category,
|
||||
skips empty sections, and renders a section header (icon + label + count + hint)
|
||||
above each group.
|
||||
5. In `ActivityItem`, replace the buried source `<Badge variant="outline">{source}</Badge>`
|
||||
with the colored provenance chip + a kind-label badge; keep actors and file badges.
|
||||
6. Collapsed icon-rail (the `group-data-[collapsible=icon]` mini list) stays flat —
|
||||
no sectioning needed there.
|
||||
7. Verify: `pnpm --filter @podman/frontend build` typechecks; empty-state and a
|
||||
live pod with mixed events both render correctly.
|
||||
|
||||
## Out of scope (do not do)
|
||||
|
||||
- No changes to `shared/src/activity.ts`, the SSE hook, or backend event emission.
|
||||
- No new event kinds/sources.
|
||||
- No third section / per-kind lanes — two buckets is the whole point (signals vs
|
||||
reasoning). Keeps a sparse demo stream from fragmenting.
|
||||
|
||||
## Merge-safety note
|
||||
|
||||
Single file, two functions. Hold until the teammate improving the frontend pushes,
|
||||
then pull and apply on top to avoid clobbering their design pass.
|
||||
@@ -1,147 +0,0 @@
|
||||
---
|
||||
name: podman-design
|
||||
description: Full system design for PodMan — real-time AI team coordination agent using Gemini Vision, Gemini Live 2.5, LiveKit, and MongoDB Atlas
|
||||
metadata:
|
||||
type: project
|
||||
---
|
||||
|
||||
# PodMan — System Design
|
||||
|
||||
Status: historical reference. Current implementation truth lives in
|
||||
[`../../PLAN.md`](../../PLAN.md), [`../../mongodb.md`](../../mongodb.md),
|
||||
[`../../continual-learning/`](../../continual-learning/), and
|
||||
[`../../graph-discovery/`](../../graph-discovery/).
|
||||
|
||||
## Concept
|
||||
|
||||
PodMan is a real-time AI team coordination agent for software teams. Engineers join a consented LiveKit room and publish screen share when they want PodMan to observe active work. The backend agent samples the LiveKit screen track, uses Gemini Vision to extract structured context, detects coordination risks, and sends intervention cards, Hermes messages, or urgent voice cues through LiveKit. MongoDB Atlas stores observations, collisions, interventions, outcomes, latest engineer state, and the Team memory graph.
|
||||
|
||||
**Track:** Continual Learning — accepted and dismissed outcomes make later exact-signature recall and graph memory more useful.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────── Engineer laptop (Browser PWA) ──────────────────┐
|
||||
│ getDisplayMedia → LiveKit screen-share track │
|
||||
│ Local git watcher → MongoDB engineer_states │
|
||||
│ LiveKit room joined → receives cards, messages, voice cues │
|
||||
│ Earbuds: hears PodMan urgent voice cues │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│ LiveKit media + data
|
||||
▼
|
||||
┌────────────────── HERMES (DigitalOcean) ─────────────────────────┐
|
||||
│ 1. Subscribe to screen-share track → Gemini Vision │
|
||||
│ 2. Write observations and per-user state to MongoDB │
|
||||
│ 3. Fuse local git truth from engineer_states │
|
||||
│ 4. Run collision detector over active contexts │
|
||||
│ 5. If risk detected → card/message first, voice only if urgent │
|
||||
│ 6. Push data and optional audio into LiveKit room │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│ read/write
|
||||
▼
|
||||
MongoDB Atlas
|
||||
(engineer_states, observations,
|
||||
collisions, interventions, outcomes,
|
||||
team_model, graph_nodes, graph_edges)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Components
|
||||
|
||||
### PWA (local agent)
|
||||
|
||||
- Joins LiveKit room via existing `joinPod` flow
|
||||
- Publishes screen share through LiveKit after explicit user action
|
||||
- Receives Hermes audio track through LiveKit when voice is urgent
|
||||
- Listens for data channel messages → renders intervention feed
|
||||
- Two screens: join screen (built), active session screen (to build)
|
||||
|
||||
### Hermes (orchestrator)
|
||||
|
||||
- Express server + LiveKit Agent on DigitalOcean
|
||||
- LiveKit agent worker receives sampled screen-share frames and queues them for vision
|
||||
- Vision pipeline: Gemini 2.0 Flash → `EngineerContext`
|
||||
- Confidence gate: discard frames with confidence < 0.6
|
||||
- State writer: write `observations`, `collisions`, `interventions`, `outcomes`, and `engineer_states`
|
||||
- Event detector: Gemini text prompt over all active states
|
||||
- Message generator: Gemini text → short intervention message
|
||||
- Voice publisher: Gemini TTS via LiveKit audio into room for urgent escalation
|
||||
- Data channel: sends structured intervention payload
|
||||
- Cooldown: 3 min between voice cues per pod
|
||||
|
||||
### Gemini usage
|
||||
|
||||
- **Vision:** `gemini-2.0-flash` — screen → `{ currentFile, inferredTask, terminalVisible, recentTerminalOutput, confidence }`
|
||||
- **Event detection:** `gemini-2.0-flash` — all engineer states → `{ event, involvedEngineers, file, reason }`
|
||||
- **Message generation:** `gemini-2.0-flash` — risk → intervention text
|
||||
- **Voice:** `gemini-3.1-flash-tts-preview` via LiveKit audio publication — text → audio
|
||||
|
||||
### MongoDB Atlas
|
||||
|
||||
- `engineer_states`: latest context per engineer
|
||||
- `observations`: structured perception records
|
||||
- `collisions`: detected coordination risks
|
||||
- `interventions`: cards, messages, and voice cues sent or suggested
|
||||
- `outcomes`: accepted and dismissed learning signals
|
||||
- `team_model`: durable per-pod summary and seeded graph
|
||||
- `graph_nodes` / `graph_edges`: normalized graph records for `$graphLookup`
|
||||
|
||||
### LiveKit
|
||||
|
||||
- One room per pod
|
||||
- Engineers publish screen-share tracks
|
||||
- PodMan joins as an agent participant, subscribes to screen share, and publishes audio + data channel messages
|
||||
- Engineers receive audio automatically
|
||||
|
||||
---
|
||||
|
||||
## Event types
|
||||
|
||||
| Event | Trigger | Example intervention |
|
||||
| ------------------ | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
|
||||
| `BLOCKER_DETECTED` | Engineer stuck (error in terminal, same file N frames) + teammate can help | "Carol, looks like you're waiting on auth. Alice is actively building it — hang tight." |
|
||||
| `DEPENDENCY_READY` | Engineer A completes work that Engineer B was waiting on | "Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate." |
|
||||
| `DUPLICATE_WORK` | 2+ engineers on same file simultaneously | "Alice and Bob — you're both in login.tsx. Coordinate before pushing." |
|
||||
|
||||
---
|
||||
|
||||
## Continual learning story
|
||||
|
||||
The `team_model` graph and accepted outcomes persist across sessions. On graph
|
||||
load:
|
||||
|
||||
1. Materialize from live MongoDB records when real activity exists.
|
||||
2. Fall back to seeded `team_model.graph`.
|
||||
3. Fall back to a labeled demo graph for stage stability.
|
||||
4. Exact signature recall uses accepted and dismissed outcomes before vector recall.
|
||||
|
||||
**Demo:** The first collision writes an outcome. The second similar collision
|
||||
recalls that memory and changes the graph or behavior. That is the learning
|
||||
visible on stage.
|
||||
|
||||
---
|
||||
|
||||
## Demo flow (3 min)
|
||||
|
||||
1. **(0:00)** Three engineers join pod. PodMan greets by voice.
|
||||
2. **(0:20)** Alice opens `auth/middleware.ts`. Hermes infers ownership.
|
||||
3. **(0:45)** Bob opens `frontend/login.tsx`. Carol's terminal shows connection refused.
|
||||
4. **(1:20) BLOCKER_DETECTED:** "Carol, looks like you're waiting on auth. Alice is actively building it — hang tight."
|
||||
5. **(2:00) DEPENDENCY_READY:** "Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate."
|
||||
6. **(2:20)** Optional: session 2 warm-start comparison.
|
||||
7. **(2:45)** Close: "PodMan — the teammate that sees what Slack can't."
|
||||
|
||||
---
|
||||
|
||||
## Key risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
| -------------------------------------------- | ------------------------------------------------- |
|
||||
| Gemini Vision accuracy | Large font, single editor window, confidence gate |
|
||||
| Gemini Live 2.5 + LiveKit Agents integration | Build together hour 5–7, have TTS fallback |
|
||||
| Frame POST latency | JPEG compression, target < 500ms |
|
||||
| Event false positives | 3-min cooldown, pre-staged demo |
|
||||
| DO deploy failure | Hermes runs local, PWA defaults to localhost:8787 |
|
||||
@@ -1,79 +0,0 @@
|
||||
# Shared Background Music — pod-wide audio + connectivity check
|
||||
|
||||
> Spec for the `frontend/src/livekit/useBeat.ts` + `PodView.tsx` background-music
|
||||
> behavior, the `lib/beat.ts` audio source, the `GET /api/pods/:id/music`
|
||||
> endpoint, and the additive `BEAT_STOP` data message. Satisfies the
|
||||
> documentation-first gate for those files.
|
||||
|
||||
## Why
|
||||
|
||||
The **Background Music** button is PodMan's pod-wide audio: a calm, looping
|
||||
background track unique to each pod, generated by Gemini **Lyria 3**. It opens
|
||||
with the pod's name sung once, then settles into a soft instrumental bed. It also
|
||||
doubles as the pre-flight check that the LiveKit audio path works for the whole
|
||||
pod — the same path the urgent Gemini-TTS voice escalation rides on. Its on/off
|
||||
state is shared pod-wide so a judge sees it flip on every screen at once.
|
||||
|
||||
## Music generation (backend)
|
||||
|
||||
- `GET /api/pods/:id/music` → streams the pod's background-music MP3
|
||||
(`audio/mpeg`). On first request it calls Lyria 3 (`lyria-3-clip-preview`) via
|
||||
the Gemini **interactions** endpoint with a prompt that sings the pod name in
|
||||
the first ~3s then stays instrumental, and **caches** the MP3 in the
|
||||
`pod_music` Mongo collection (keyed by pod id; regenerated if the pod name
|
||||
changes). Subsequent requests are instant. The Gemini key stays server-side.
|
||||
- `backend/src/voice/music.ts` owns generation + caching (`getPodMusic`).
|
||||
- Override the model with `GEMINI_MUSIC_MODEL` (default `lyria-3-clip-preview`).
|
||||
|
||||
## Behavior (frontend)
|
||||
|
||||
- Any participant clicks **Background Music** → the client fetches the pod's MP3
|
||||
and loops it (`lib/beat.ts` `startMusic`, Web Audio `AudioBufferSource.loop`),
|
||||
publishing it as the `podman-beat` track. Everyone auto-subscribes and hears
|
||||
it; the publisher hears it locally too.
|
||||
- The shared on/off state is **derived from the track's presence**, not a synced
|
||||
flag — so it self-syncs across joins/leaves and can't drift. The publisher is
|
||||
the **owner**.
|
||||
- Anyone can stop it:
|
||||
- Owner clicks **Stop music** → unpublishes its own track directly.
|
||||
- Non-owner clicks **Stop (`<owner>`)** → sends `BEAT_STOP`; the owner
|
||||
unpublishes. (LiveKit forbids unpublishing another participant's track.)
|
||||
- `PodView` warms the cache with a fire-and-forget fetch on mount so the first
|
||||
click plays instantly.
|
||||
|
||||
## State derivation (source of truth = the track)
|
||||
|
||||
`useBeat(room, musicUrl)` returns `{ beat, toggleBeat }` where `beat` is
|
||||
`{ on, by, mine }`, recomputed from the presence of a track named `podman-beat`
|
||||
across `localParticipant` + `remoteParticipants` on these events:
|
||||
`LocalTrackPublished/Unpublished`, `TrackPublished/Unpublished`,
|
||||
`TrackSubscribed/Unsubscribed`, `ParticipantConnected/Disconnected`. Owner
|
||||
disconnect and late-join sync therefore need no extra messaging.
|
||||
|
||||
## Contract (additive)
|
||||
|
||||
`shared/src/messages.ts` — `{ type: 'BEAT_STOP' }` on the existing
|
||||
`podman.intervention` data topic (any participant → owner: stop the shared
|
||||
track). Additive to the `DataMessage` union; existing consumers ignore unknown
|
||||
types.
|
||||
|
||||
## Known limitation (LiveKit constraint)
|
||||
|
||||
A client can only unpublish **its own** tracks, so a non-owner's **Stop** is a
|
||||
`BEAT_STOP` _request_ the owner must honor. If the owner disconnects **uncleanly**
|
||||
(crash / network drop), the SFU keeps the track published until it times the
|
||||
participant out — during that window the music keeps playing and non-owners
|
||||
can't stop it. A clean disconnect clears it immediately via
|
||||
`ParticipantDisconnected`. Demo mitigation: have the same person who starts it
|
||||
also stop it.
|
||||
|
||||
## Files
|
||||
|
||||
- `backend/src/voice/music.ts` — Lyria generation + `pod_music` cache.
|
||||
- `backend/src/server.ts` — `GET /api/pods/:id/music` (streams MP3).
|
||||
- `frontend/src/lib/api.ts` — `podMusicUrl(id)` helper.
|
||||
- `frontend/src/lib/beat.ts` — `startMusic(url)` (loops the MP3); legacy
|
||||
`startBeat()` (synthesized kick/hat) kept as a fallback.
|
||||
- `frontend/src/livekit/useBeat.ts` — `useBeat(room, musicUrl)` hook.
|
||||
- `frontend/src/components/PodView.tsx` — button label + cache warm-up.
|
||||
- `shared/src/messages.ts` — `BEAT_STOP` message (additive).
|
||||
Reference in New Issue
Block a user