Add member work history and learning docs
This commit is contained in:
Yahya Alhinai
+16
-3
@@ -4,7 +4,8 @@
|
||||
> strategy, public interfaces, risks, sponsor story, and next build order.
|
||||
>
|
||||
> If this file conflicts with `README.md`, `docs/idea.md`, `docs/livekit.md`,
|
||||
> `docs/gemini.md`, `docs/mongodb.md`, `docs/digitalocean.md`,
|
||||
> `docs/gemini.md`, `docs/mongodb.md`, `docs/continual-learning/`,
|
||||
> `docs/graph-discovery/`, `docs/agent-learning/`, `docs/digitalocean.md`,
|
||||
> `docs/demo-setup.md`, or `docs/superpowers/specs/*`, follow this file and
|
||||
> treat the older docs as reference material to reconcile later.
|
||||
|
||||
@@ -293,6 +294,8 @@ scripts together.
|
||||
- `POST /api/sync-pr`
|
||||
- `POST /api/outcome`
|
||||
- `GET /api/memory/stats`
|
||||
- `GET /api/pods/:id/graph`
|
||||
- `GET /api/pods/:id/graph/reach/:nodeId`
|
||||
- `GET /api/pods`
|
||||
- `POST /api/pods`
|
||||
- `GET /api/pods/:id`
|
||||
@@ -471,8 +474,8 @@ artifact.
|
||||
- Escalate to voice only when urgent.
|
||||
|
||||
10. **Action artifact**
|
||||
- If demo uses same-file collision, click the card to open a real draft sync
|
||||
PR or visible GitHub artifact.
|
||||
- If demo uses same-file collision, click the card to open a real sync PR
|
||||
artifact or visible GitHub artifact.
|
||||
- If demo uses research recommendation, show the accepted recommendation and
|
||||
memory outcome instead.
|
||||
|
||||
@@ -607,6 +610,16 @@ MongoDB is the learning proof:
|
||||
- Voyage + Atlas Vector Search is the stronger sponsor-grade version after exact
|
||||
recall works.
|
||||
|
||||
Canonical docs:
|
||||
|
||||
- [`docs/continual-learning/`](continual-learning/) owns team memory and
|
||||
outcome-backed recall.
|
||||
- [`docs/graph-discovery/`](graph-discovery/) owns graph materialization,
|
||||
hygiene, and `$graphLookup` reachability.
|
||||
- [`docs/agent-learning/`](agent-learning/) owns the planned narrow
|
||||
strategy-version layer. Full autonomous promotion is not implemented unless
|
||||
backed by records.
|
||||
|
||||
### DigitalOcean
|
||||
|
||||
DigitalOcean earns its place when:
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
# Agent Learning
|
||||
|
||||
Status: planned / narrow v1
|
||||
|
||||
Agent learning owns how PodMan can improve its own prompts, detector rules,
|
||||
policies, verifier choices, and routing strategies. This is deliberately
|
||||
narrower than team memory: it is a versioned strategy layer, not autonomous code
|
||||
rewriting.
|
||||
|
||||
The constraints in [`../../CLAUDE.md`](../../CLAUDE.md) still govern this track:
|
||||
one visible self-improving loop, demo stability, no broad platform rewrite, no
|
||||
dashboard-first product, and no overclaiming.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| [`spec.md`](spec.md) | Read-only data contract for runs, traces, strategies, and proposals |
|
||||
| [`policy.md`](policy.md) | Promotion, rejection, evidence, and safety rules |
|
||||
| [`prompt.md`](prompt.md) | Evaluator prompt for narrow strategy improvements |
|
||||
| [`plan.md`](plan.md) | v1 implementation order if this track is added |
|
||||
|
||||
## What Is Implemented Now
|
||||
|
||||
- Shared TypeScript contracts for `AgentRun`, `AgentTraceEvent`,
|
||||
`StrategyVersion`, and `LearningProposal`.
|
||||
- Exact signature recall and accepted/dismissed outcomes that can later feed
|
||||
strategy decisions.
|
||||
- Documentation of future collections and indexes.
|
||||
|
||||
## What Is Intentionally Cut
|
||||
|
||||
- Full autonomous strategy promotion.
|
||||
- Autonomous code rewriting.
|
||||
- Multi-agent strategy debates.
|
||||
- Claims that PodMan trains or rewrites itself from live usage today.
|
||||
|
||||
## Demo Proof Path
|
||||
|
||||
Observe screen/git state -> detect collision -> send intervention -> accept or
|
||||
dismiss outcome -> recall similar event -> show changed graph or changed
|
||||
behavior. In the current demo, this proof is team-memory learning; agent
|
||||
strategy promotion remains planned unless records are added.
|
||||
@@ -1,6 +1,6 @@
|
||||
# Agent Learning Plan
|
||||
|
||||
Status: draft
|
||||
Status: planned / narrow v1
|
||||
Goal: ship a visible recursive self-improvement loop without overbuilding
|
||||
|
||||
## Must-Have
|
||||
@@ -86,4 +86,3 @@ rejected or open.
|
||||
- The changed behavior is visible.
|
||||
- The strategy has a parent and evidence.
|
||||
- Rejected or failed changes are not deleted.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Agent Learning Policy
|
||||
|
||||
Status: draft
|
||||
Status: planned / narrow v1
|
||||
Scope: guardrails for recursive self-improvement
|
||||
|
||||
## Prime Rule
|
||||
@@ -81,4 +81,3 @@ Reject and retain the candidate when:
|
||||
Seeded strategy versions are acceptable when labeled as demo-backed. Do not claim
|
||||
a strategy was learned live unless a run and outcome actually created the
|
||||
promotion evidence.
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Agent Learning Spec
|
||||
|
||||
Status: draft
|
||||
Scope: how PodMan agents improve their own prompts, policies, detectors, and routing behavior
|
||||
Status: planned / narrow v1
|
||||
Scope: how PodMan agents improve their own prompts, policies, detectors, and routing behavior
|
||||
Owner: agent learning / recursive self-improvement
|
||||
|
||||
## Purpose
|
||||
@@ -19,6 +19,20 @@ The demo claim:
|
||||
5. The new strategy is versioned.
|
||||
6. A later run uses the improved strategy and shows a better result.
|
||||
|
||||
## What Is Implemented Now
|
||||
|
||||
- Shared TypeScript record shapes exist for the core objects below.
|
||||
- Exact signature recall and accepted/dismissed outcomes exist in the team
|
||||
memory loop.
|
||||
- No write path currently promotes autonomous strategy changes.
|
||||
|
||||
## What Is Intentionally Cut
|
||||
|
||||
- Autonomous code rewriting.
|
||||
- Full autonomous strategy promotion.
|
||||
- Multi-agent strategy debates.
|
||||
- Claims that model self-evaluation alone can promote a strategy.
|
||||
|
||||
## Core Objects
|
||||
|
||||
### Agent run
|
||||
@@ -182,4 +196,3 @@ Graph discovery may show:
|
||||
- Rejected strategies are retained with a reason.
|
||||
- Agent traces are append-only.
|
||||
- The system can answer: "What changed, why, and did it help?"
|
||||
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
# Continual Learning
|
||||
|
||||
Status: demo-backed / active
|
||||
|
||||
PodMan's continual-learning track owns team memory: what the system learns about
|
||||
files, collisions, interventions, outcomes, and future routing for a pod.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| [`spec.md`](spec.md) | Data model and observe/store/predict/outcome/adapt loop |
|
||||
| [`policy.md`](policy.md) | What PodMan may and may not remember |
|
||||
| [`prompt.md`](prompt.md) | Memory-agent prompt for outcome-backed learning |
|
||||
| [`plan.md`](plan.md) | Demo build order and acceptance criteria |
|
||||
|
||||
## What Is Implemented Now
|
||||
|
||||
- MongoDB-backed `observations`, `collisions`, `interventions`, `outcomes`,
|
||||
`engineer_states`, and `team_model` records.
|
||||
- Exact signature recall for prior accepted and dismissed outcomes.
|
||||
- Outcome writes through `POST /api/outcome`.
|
||||
- Team memory graph edges from accepted real outcomes.
|
||||
- No raw screenshots or recordings are stored.
|
||||
|
||||
## What Is Intentionally Cut
|
||||
|
||||
- Autonomous model training.
|
||||
- Broad cross-pod generalization.
|
||||
- Raw screen capture retention.
|
||||
- Vector recall as a dependency for the demo proof.
|
||||
|
||||
## Demo Proof Path
|
||||
|
||||
Observe screen/git state -> detect collision -> send intervention -> accept or
|
||||
dismiss outcome -> recall similar event -> show changed graph or changed
|
||||
behavior.
|
||||
@@ -1,6 +1,6 @@
|
||||
# Continual Learning Plan
|
||||
|
||||
Status: draft
|
||||
Status: demo-backed / active
|
||||
Goal: prove PodMan learns from outcomes in the hackathon demo
|
||||
|
||||
## Must-Have Demo Loop
|
||||
@@ -66,4 +66,3 @@ Goal: prove PodMan learns from outcomes in the hackathon demo
|
||||
- The second similar event behaves differently.
|
||||
- Exact MongoDB records prove the loop.
|
||||
- The graph remains legible with real data.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Continual Learning Policy
|
||||
|
||||
Status: draft
|
||||
Status: demo-backed / active
|
||||
Scope: what PodMan may learn about a team
|
||||
|
||||
## Prime Rule
|
||||
@@ -94,4 +94,3 @@ Delete immediately:
|
||||
Seeded data is acceptable only if the demo script is honest about it. Live
|
||||
learning requires a live or staged outcome write that visibly updates the graph
|
||||
or future decision.
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Continual Learning Spec
|
||||
|
||||
Status: draft
|
||||
Scope: how PodMan learns team memory from live work and outcomes
|
||||
Status: demo-backed / active
|
||||
Scope: how PodMan learns team memory from live work and outcomes
|
||||
Owner: continual learning / Team memory
|
||||
|
||||
## Purpose
|
||||
@@ -16,6 +16,22 @@ The visible loop:
|
||||
observe -> store -> predict -> outcome -> adapt
|
||||
```
|
||||
|
||||
## What Is Implemented Now
|
||||
|
||||
- `observations`, `collisions`, `interventions`, `outcomes`,
|
||||
`engineer_states`, `team_model`, `graph_nodes`, and `graph_edges` are the
|
||||
current memory truth.
|
||||
- Exact signature recall and accepted/dismissed outcomes exist.
|
||||
- Accepted real outcomes can produce `learned_from` graph edges and ownership
|
||||
memory.
|
||||
- Raw screenshots and recordings are not stored.
|
||||
|
||||
## What Is Intentionally Cut
|
||||
|
||||
- Full autonomous training.
|
||||
- Broad threshold changes from one example.
|
||||
- Making vector search required for the demo learning proof.
|
||||
|
||||
## Source Collections
|
||||
|
||||
### `engineer_states`
|
||||
@@ -214,4 +230,3 @@ editing, collision, intervention, outcome, learned, agent
|
||||
- The Team memory graph can explain the learning loop.
|
||||
- Dismissals and false positives are retained.
|
||||
- The demo does not rely on raw screenshots or hidden state.
|
||||
|
||||
|
||||
+4
-4
@@ -69,9 +69,9 @@ Pre-create these files in the demo repo before the demo:
|
||||
| 0:20 | Alice opens `auth/middleware.ts`, starts typing | Alice |
|
||||
| 0:45 | Bob opens `frontend/login.tsx` | Bob |
|
||||
| 0:50 | Carol runs `curl` command, sees error | Carol |
|
||||
| ~1:20 | BLOCKER_DETECTED nudge fires | Hermes auto |
|
||||
| ~1:20 | BLOCKER_DETECTED intervention fires | Hermes auto |
|
||||
| 1:50 | Alice starts her server (`node server.js`) | Alice |
|
||||
| ~2:00 | DEPENDENCY_READY nudge fires | Hermes auto |
|
||||
| ~2:00 | DEPENDENCY_READY intervention fires | Hermes auto |
|
||||
| 2:20 | Optional: show session 2 ownership warm-start | Presenter |
|
||||
| 2:45 | Close | Presenter |
|
||||
|
||||
@@ -89,7 +89,7 @@ Pre-create these files in the demo repo before the demo:
|
||||
|
||||
## Cooldown note
|
||||
|
||||
Hermes has a 3-minute cooldown between nudges per pod. For the demo, if you need to trigger a second event quickly:
|
||||
Hermes has a 3-minute cooldown between urgent voice cues per pod. For the demo, if you need to trigger a second urgent voice event quickly:
|
||||
|
||||
Option 1: restart Hermes between the two demo scenarios (resets cooldown state)
|
||||
Option 2: set `NUDGE_COOLDOWN_MS=0` via env var during demo (add this override to Hermes)
|
||||
@@ -102,7 +102,7 @@ If any system fails on stage:
|
||||
|
||||
1. **Hermes unreachable:** switch to local (`pnpm --filter backend dev`) — PWA auto-falls back to `localhost:8787`
|
||||
2. **Gemini Vision low confidence:** presenter narrates what PodMan "saw" while playing the backup video
|
||||
3. **LiveKit audio not working:** play backup video — show the nudge text cards on screen instead
|
||||
3. **LiveKit audio not working:** play backup video — show the intervention cards on screen instead
|
||||
4. **Full system failure:** play the backup recording, narrate the demo live
|
||||
|
||||
Always have the backup video on a separate device, not the same laptop running Hermes.
|
||||
|
||||
+4
-4
@@ -1,6 +1,6 @@
|
||||
# Gemini Integration Spec
|
||||
|
||||
PodMan uses Gemini for two distinct jobs: **vision** (understanding screens) and **voice** (speaking nudges).
|
||||
PodMan uses Gemini for two distinct jobs: **vision** (understanding screens) and **voice** (urgent voice cues).
|
||||
|
||||
---
|
||||
|
||||
@@ -75,7 +75,7 @@ Respond with valid JSON only.
|
||||
|
||||
---
|
||||
|
||||
## 3. Nudge Generation — Voice Message
|
||||
## 3. Intervention Text Generation
|
||||
|
||||
**Model:** `gemini-2.0-flash` (text only)
|
||||
|
||||
@@ -118,7 +118,7 @@ Respond with the message text only.
|
||||
|
||||
**Flow:**
|
||||
|
||||
1. Nudge message text generated (step 3)
|
||||
1. Intervention message text generated (step 3)
|
||||
2. Hermes wraps it in a natural-speaking prompt for Gemini TTS
|
||||
3. Gemini returns audio with the configured prebuilt voice
|
||||
4. Hermes publishes the audio into the LiveKit room
|
||||
@@ -135,4 +135,4 @@ Respond with the message text only.
|
||||
|
||||
## Cooldown
|
||||
|
||||
Per-pod cooldown of **3 minutes** between nudges. Prevents spam if multiple events fire simultaneously. Implemented in Hermes, not in Gemini.
|
||||
Per-pod cooldown of **3 minutes** between urgent voice cues. Prevents spam if multiple risks fire simultaneously. Implemented in Hermes, not in Gemini.
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
# Graph Discovery
|
||||
|
||||
Status: demo-backed / active
|
||||
|
||||
Graph discovery owns how MongoDB records become the Team memory graph. It
|
||||
materializes a sparse, auditable graph from real records first, seeded graph
|
||||
second, and demo fallback third.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| [`spec.md`](spec.md) | Source data, graph contract, and discovery rules |
|
||||
| [`policy.md`](policy.md) | Graph hygiene, evidence thresholds, and truthfulness |
|
||||
| [`prompt.md`](prompt.md) | Graph materialization and review prompt |
|
||||
| [`plan.md`](plan.md) | Risk-path and observatory build plan |
|
||||
|
||||
## What Is Implemented Now
|
||||
|
||||
- `GET /api/pods/:podId/graph`.
|
||||
- `GET /api/pods/:podId/graph/reach/:id` backed by MongoDB `$graphLookup`.
|
||||
- Live graph materialization from `pods`, `engineer_states`, `observations`,
|
||||
`collisions`, `interventions`, and `outcomes`.
|
||||
- Seeded graph in `team_model.graph` and mirrored `graph_nodes` / `graph_edges`.
|
||||
- Demo graph fallback so the stage never shows an empty canvas.
|
||||
|
||||
## What Is Intentionally Cut
|
||||
|
||||
- A separate graph database.
|
||||
- A broad analytics dashboard.
|
||||
- Showing every historical event by default.
|
||||
- Treating seeded demo data as live learning.
|
||||
|
||||
## Demo Proof Path
|
||||
|
||||
Observe screen/git state -> detect collision -> send intervention -> accept or
|
||||
dismiss outcome -> recall similar event -> show changed graph or changed
|
||||
behavior.
|
||||
@@ -1,6 +1,6 @@
|
||||
# Graph Discovery Plan
|
||||
|
||||
Status: draft
|
||||
Status: demo-backed / active
|
||||
Goal: make MongoDB graph discovery visible as a dynamic learning observatory
|
||||
|
||||
## Must-Have
|
||||
@@ -66,4 +66,3 @@ Goal: make MongoDB graph discovery visible as a dynamic learning observatory
|
||||
- Learned path is visible when data exists.
|
||||
- Whole graph mode exists but is not the default.
|
||||
- The graph remains backed by MongoDB, not hardcoded mock data.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Graph Discovery Policy
|
||||
|
||||
Status: draft
|
||||
Status: demo-backed / active
|
||||
Scope: graph hygiene, evidence thresholds, and UI truthfulness
|
||||
|
||||
## Prime Rule
|
||||
@@ -80,4 +80,3 @@ Semantic colors stay stable:
|
||||
- Learned: violet dashed edge.
|
||||
|
||||
Chrome should use the app's light shadcn tokens.
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Graph Discovery Spec
|
||||
|
||||
Status: draft
|
||||
Scope: how PodMan discovers graph nodes, edges, risk paths, and learning paths from MongoDB
|
||||
Status: demo-backed / active
|
||||
Scope: how PodMan discovers graph nodes, edges, risk paths, and learning paths from MongoDB
|
||||
Owner: graph discovery / Team memory observatory
|
||||
|
||||
## Purpose
|
||||
@@ -18,6 +18,20 @@ The graph must answer:
|
||||
4. What did PodMan do?
|
||||
5. What outcome changed memory?
|
||||
|
||||
## What Is Implemented Now
|
||||
|
||||
- Live materializer first: build from current MongoDB records.
|
||||
- Seeded graph second: read `team_model.graph` and mirrored graph collections.
|
||||
- Demo fallback third: return a grounded demo graph when live data is empty or
|
||||
unavailable.
|
||||
- Reachability uses MongoDB `$graphLookup` over `graph_edges`.
|
||||
|
||||
## What Is Intentionally Cut
|
||||
|
||||
- A graph database migration.
|
||||
- Whole-history rendering as the default view.
|
||||
- Claims that seeded graph data is live learning.
|
||||
|
||||
## Source Data
|
||||
|
||||
Graph discovery reads:
|
||||
@@ -143,4 +157,3 @@ Optional metrics:
|
||||
- Every selected node can explain why it matters.
|
||||
- Activity stream matches graph events.
|
||||
- Graph can be rebuilt from MongoDB source records.
|
||||
|
||||
|
||||
+6
-2
@@ -1,8 +1,10 @@
|
||||
# Continual-Learning Graph Spec
|
||||
|
||||
> Owner: graph data + visualization. Status: demo-backed (live `team_model` reads land later).
|
||||
> Owner: graph data + visualization. Status: demo-backed / active.
|
||||
> Satisfies the documentation-first gate for the `backend/src/graph/*` and
|
||||
> `frontend/src/components/GraphView.tsx` files.
|
||||
>
|
||||
> Canonical module docs live in [`docs/graph-discovery/`](graph-discovery/).
|
||||
|
||||
## What this is (and is NOT)
|
||||
|
||||
@@ -26,7 +28,9 @@ The graph lives in two places, both keyed by `podId`:
|
||||
{ podId, graph: PodGraph, updatedAt }
|
||||
```
|
||||
|
||||
`GET /api/pods/:podId/graph` returns `team_model.graph`, or a demo graph when none exists yet.
|
||||
`GET /api/pods/:podId/graph` returns the live materialized graph first, then
|
||||
`team_model.graph`, then a labeled demo graph when neither live nor seeded
|
||||
data exists.
|
||||
|
||||
2. **Normalized (for traversal):** the same nodes/edges are mirrored into two collections so
|
||||
the model can be walked with MongoDB `$graphLookup` (the graph-database pattern):
|
||||
|
||||
+12
-12
@@ -2,7 +2,7 @@
|
||||
|
||||
## One-line value prop
|
||||
|
||||
PodMan is a real-time AI team coordination agent that watches consented work signals, maintains live project memory, and proactively notifies collaborators when dependencies, blockers, or handoffs emerge — before anyone has to ask.
|
||||
PodMan is a real-time AI team coordination agent that watches consented work signals, maintains live project memory, and proactively coordinates collaborators when collisions, blockers, or handoffs emerge before anyone has to ask.
|
||||
|
||||
---
|
||||
|
||||
@@ -18,11 +18,11 @@ Slack doesn't help. Stand-ups are too slow. GitHub only knows pushed state.
|
||||
|
||||
PodMan is an ambient AI agent that:
|
||||
|
||||
1. Watches each engineer's screen via periodic snapshots (consented, browser-native)
|
||||
1. Watches each engineer's consented LiveKit screen-share signal
|
||||
2. Extracts structured context using Gemini Vision — current file, inferred task, terminal state
|
||||
3. Maintains a shared live model of the team in MongoDB Atlas — who is doing what, who owns which files
|
||||
4. Detects coordination events: dependency ready, blocker detected, duplicate work
|
||||
5. Speaks proactively into the team's LiveKit room — engineers hear PodMan through their earbuds without leaving their editor
|
||||
3. Maintains a shared live model in MongoDB Atlas — observations, collisions, interventions, outcomes, and graph memory
|
||||
4. Detects coordination risks: same-file collision, blocker detected, duplicate work
|
||||
5. Sends the least intrusive intervention first: card, Hermes message, and urgent voice only when needed
|
||||
|
||||
**The AI's job is not to chat. It is to notice what teammates miss and say so, exactly when it matters.**
|
||||
|
||||
@@ -38,21 +38,21 @@ Small software teams: hackathon squads, startup engineering teams, student dev t
|
||||
|
||||
- Maintain per-person live context (file, task, terminal)
|
||||
- Infer shared project state (who owns what, what's blocked, what's ready)
|
||||
- Detect 3 coordination event types:
|
||||
- Detect 3 coordination risk types:
|
||||
- `DEPENDENCY_READY` — engineer A was waiting on work engineer B just completed
|
||||
- `BLOCKER_DETECTED` — engineer appears stuck; another teammate can unblock
|
||||
- `DUPLICATE_WORK` — 2+ engineers working on the same file simultaneously
|
||||
- Generate a 1–2 sentence proactive voice nudge
|
||||
- Deliver it into the LiveKit room as Gemini TTS audio
|
||||
- Generate a short intervention message
|
||||
- Deliver it as a LiveKit data message, with Gemini TTS audio reserved for urgent escalation
|
||||
|
||||
---
|
||||
|
||||
## How it fits the Continual Learning track
|
||||
|
||||
PodMan builds an **ownership map** in MongoDB that persists across sessions:
|
||||
PodMan builds outcome-backed team memory in MongoDB that persists across sessions:
|
||||
|
||||
- Session 1: PodMan needs 3–5 minutes of screen observations to infer who owns what
|
||||
- Session 2+: PodMan already knows. First nudge fires in under 30 seconds.
|
||||
- Session 1: PodMan observes work, predicts a collision, sends an intervention, and stores the outcome
|
||||
- Session 2+: PodMan recalls the exact signature and changes the graph or behavior
|
||||
|
||||
The system gets demonstrably more useful the more it is used, with no user configuration required. That is the track definition met exactly.
|
||||
|
||||
@@ -60,7 +60,7 @@ The system gets demonstrably more useful the more it is used, with no user confi
|
||||
|
||||
## Architecture (one paragraph)
|
||||
|
||||
Each engineer opens a browser PWA on their laptop. The PWA captures live IDE context through LiveKit screen sharing and scheduled local git reports. Hermes, the server-side orchestrator running on DigitalOcean, calls Gemini Vision to extract structured context, writes it to MongoDB Atlas, updates the ownership map, and runs event detection across all active engineers. When a coordination event fires, Hermes generates a short spoken message, asks Gemini TTS for natural audio, and publishes that audio into the team's LiveKit room. Engineers hear PodMan through their earbuds. No Slack. No tab switching. No interruption to the editor flow.
|
||||
Each engineer opens a browser PWA on their laptop. The PWA shares live IDE context through LiveKit screen sharing, and the local git watcher writes dirty/unpushed state to MongoDB. The backend agent calls Gemini Vision to extract structured context, writes observations and collisions to MongoDB Atlas, recalls accepted or dismissed outcomes, and routes the smallest useful intervention. Cards and Hermes messages are default; Gemini TTS through LiveKit is reserved for urgent escalation. No Slack. No tab switching. No interruption to the editor flow.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+6
-6
@@ -37,9 +37,9 @@ LiveKit is the real-time backbone for PodMan. It handles room presence and voice
|
||||
```ts
|
||||
room.on(RoomEvent.DataReceived, (payload, participant) => {
|
||||
if (participant?.identity !== 'podman-hermes') return;
|
||||
const nudge = JSON.parse(new TextDecoder().decode(payload));
|
||||
// nudge: { type, message, involvedEngineers, file, sentAt }
|
||||
appendNudgeToFeed(nudge);
|
||||
const intervention = JSON.parse(new TextDecoder().decode(payload));
|
||||
// intervention: COLLISION, HERMES_MESSAGE, VOICE_CUE, ACK, or GIT_REPORT
|
||||
appendInterventionToFeed(intervention);
|
||||
});
|
||||
```
|
||||
|
||||
@@ -57,7 +57,7 @@ room.on(RoomEvent.DataReceived, (payload, participant) => {
|
||||
|
||||
**Voice delivery:**
|
||||
|
||||
1. Nudge message text is ready (from Gemini text generation)
|
||||
1. Urgent intervention text is ready (from Gemini text generation)
|
||||
2. Hermes sends a natural-speaking prompt to Gemini TTS
|
||||
3. Gemini returns PCM audio using the configured voice
|
||||
4. Hermes publishes the audio as a LiveKit microphone-source track
|
||||
@@ -70,7 +70,7 @@ room.on(RoomEvent.DataReceived, (payload, participant) => {
|
||||
**Data channel message (sent alongside audio):**
|
||||
|
||||
```ts
|
||||
const nudge = {
|
||||
const intervention = {
|
||||
type: 'DEPENDENCY_READY' | 'BLOCKER_DETECTED' | 'DUPLICATE_WORK',
|
||||
message: string, // the spoken text
|
||||
involvedEngineers: string[],
|
||||
@@ -78,7 +78,7 @@ const nudge = {
|
||||
sentAt: string, // ISO timestamp
|
||||
};
|
||||
room.localParticipant.publishData(
|
||||
new TextEncoder().encode(JSON.stringify(nudge)),
|
||||
new TextEncoder().encode(JSON.stringify(intervention)),
|
||||
{ reliable: true }
|
||||
);
|
||||
```
|
||||
|
||||
+151
-114
@@ -1,143 +1,180 @@
|
||||
# MongoDB Atlas Integration Spec
|
||||
|
||||
MongoDB Atlas is PodMan's shared memory. It stores live engineer state, the ownership map that enables continual learning, coordination events, and nudge history.
|
||||
Status: demo-backed / active
|
||||
|
||||
MongoDB Atlas is PodMan's shared memory. It stores live work observations,
|
||||
collision predictions, interventions, outcomes, latest engineer state, the
|
||||
materialized Team memory graph, and optional future recall records.
|
||||
|
||||
See also:
|
||||
|
||||
- [`docs/continual-learning/`](continual-learning/) for outcome-backed team
|
||||
memory.
|
||||
- [`docs/graph-discovery/`](graph-discovery/) for graph materialization and
|
||||
`$graphLookup` traversal.
|
||||
- [`docs/agent-learning/`](agent-learning/) for planned strategy-version
|
||||
records.
|
||||
|
||||
---
|
||||
|
||||
## Collections
|
||||
## Current Collections
|
||||
|
||||
### `engineer_states`
|
||||
|
||||
Latest context per engineer. Two writers, one collection — vision pipeline upserts vision fields, git watcher script upserts git fields independently. Hermes reads the merged document for event detection.
|
||||
Latest context per engineer. The local git watcher writes git fields; the vision
|
||||
pipeline may write screen-derived fields. Each writer updates only its own
|
||||
fields so MongoDB upserts merge cleanly.
|
||||
|
||||
```ts
|
||||
{
|
||||
_id: string, // engineerId (stable across sessions)
|
||||
podId: string,
|
||||
name: string, // display name
|
||||
Key fields:
|
||||
|
||||
// --- Vision fields (written by Hermes via POST /ingest) ---
|
||||
currentFile: string | null, // active file inferred from screen
|
||||
inferredTask: string | null, // what engineer appears to be doing
|
||||
terminalVisible: boolean,
|
||||
recentTerminalOutput: string | null,
|
||||
confidence: number, // Gemini Vision confidence (0–1)
|
||||
visionUpdatedAt: Date,
|
||||
- `podId`
|
||||
- `name`
|
||||
- `currentFile`
|
||||
- `inferredTask`
|
||||
- `confidence`
|
||||
- `changedFiles`
|
||||
- `diffStat`
|
||||
- `recentCommit`
|
||||
- `branch`
|
||||
- `visionUpdatedAt`
|
||||
- `gitUpdatedAt`
|
||||
- `updatedAt`
|
||||
|
||||
// --- Git fields (written directly by scripts/podman-agent.mjs) ---
|
||||
changedFiles: string[], // files with uncommitted changes (git status)
|
||||
diffStat: string | null, // e.g. "auth/middleware.ts | 24 +++++"
|
||||
recentCommit: string | null, // most recent commit message
|
||||
branch: string | null, // current branch name
|
||||
gitUpdatedAt: Date,
|
||||
Primary use: deterministic dirty/unpushed truth for collision detection and
|
||||
graph discovery.
|
||||
|
||||
// --- Shared ---
|
||||
updatedAt: Date // most recent write from either source
|
||||
}
|
||||
```
|
||||
### `observations`
|
||||
|
||||
**Index:** `{ podId: 1, updatedAt: -1 }`
|
||||
Structured perception events from consented screen context and agent inference.
|
||||
|
||||
**Two writers, no conflict:** vision upsert uses `$set` on vision fields only; git upsert uses `$set` on git fields only. MongoDB upsert semantics merge them cleanly.
|
||||
Key fields:
|
||||
|
||||
**Usage:** Hermes reads all documents for a given `podId` after each update to run event detection. Both vision and git context are available in the same document — `changedFiles` provides ground truth, `currentFile` provides screen context.
|
||||
- `podId`
|
||||
- `engineerId`
|
||||
- `currentFile`
|
||||
- `symbol`
|
||||
- `activity`
|
||||
- `confidence`
|
||||
- `observedAt`
|
||||
|
||||
Primary use: observe/store proof and active editing edges in the Team memory
|
||||
graph.
|
||||
|
||||
### `collisions`
|
||||
|
||||
Predicted coordination risks.
|
||||
|
||||
Key fields:
|
||||
|
||||
- `id`
|
||||
- `podId`
|
||||
- `file`
|
||||
- `symbol`
|
||||
- `engineers`
|
||||
- `severity`
|
||||
- `memorySignature`
|
||||
- `githubState`
|
||||
- `detectedAt`
|
||||
|
||||
Primary use: collision cards, exact signature recall, and graph risk paths.
|
||||
|
||||
### `interventions`
|
||||
|
||||
Actions PodMan sent or suggested.
|
||||
|
||||
Key fields:
|
||||
|
||||
- `id`
|
||||
- `podId`
|
||||
- `collisionId`
|
||||
- `kind`
|
||||
- `message`
|
||||
- `suggestedAction`
|
||||
- `status`
|
||||
- `createdAt`
|
||||
|
||||
Primary use: closing the loop from prediction to a visible card, Hermes message,
|
||||
or urgent voice cue.
|
||||
|
||||
### `outcomes`
|
||||
|
||||
Human or verifier supervision recorded through `POST /api/outcome`.
|
||||
|
||||
Key fields:
|
||||
|
||||
- `podId`
|
||||
- `interventionId`
|
||||
- `collisionId`
|
||||
- `accepted`
|
||||
- `wasRealCollision`
|
||||
- `recordedAt`
|
||||
|
||||
Primary use: accepted and dismissed outcomes drive exact recall, suppression,
|
||||
and learned graph paths.
|
||||
|
||||
### `team_model`
|
||||
|
||||
Durable per-pod summary memory.
|
||||
|
||||
Key fields:
|
||||
|
||||
- `podId`
|
||||
- `ownership`
|
||||
- `hotspots`
|
||||
- `graph`
|
||||
- `updatedAt`
|
||||
|
||||
Primary use: stable Team memory, including seeded `graph` snapshots used after
|
||||
live materialization and before demo fallback.
|
||||
|
||||
### `graph_nodes` and `graph_edges`
|
||||
|
||||
Normalized mirror of the Team memory graph for MongoDB traversal.
|
||||
|
||||
Indexes:
|
||||
|
||||
- `graph_nodes`: `{ podId: 1, id: 1 }` unique
|
||||
- `graph_edges`: `{ podId: 1, source: 1 }`
|
||||
|
||||
Primary use: `GET /api/pods/:podId/graph/reach/:id` with `$graphLookup`.
|
||||
|
||||
### Optional Future Collections
|
||||
|
||||
These are documented for planned work and should not be treated as active write
|
||||
paths unless implementation is added:
|
||||
|
||||
- `memory_vectors`
|
||||
- `agent_runs`
|
||||
- `agent_trace_events`
|
||||
- `strategy_versions`
|
||||
- `learning_proposals`
|
||||
|
||||
---
|
||||
|
||||
### `ownership_map`
|
||||
## Graph Truth Order
|
||||
|
||||
Tracks who works on which files. Built up over the session. **Persists across sessions** — this is the continual learning artifact.
|
||||
`GET /api/pods/:podId/graph` follows this order:
|
||||
|
||||
```ts
|
||||
{
|
||||
_id: string, // `${podId}:${file}`
|
||||
podId: string,
|
||||
file: string,
|
||||
primaryOwner: string, // engineerId with most recent activity on this file
|
||||
contributors: string[], // all engineerIds observed on this file
|
||||
observationCount: number, // total frames where this file was seen
|
||||
lastSeenAt: Date
|
||||
}
|
||||
```
|
||||
1. Live graph from real collections.
|
||||
2. Seeded graph from `team_model.graph` and mirrored graph records.
|
||||
3. Demo fallback graph for stage safety.
|
||||
|
||||
**Index:** `{ podId: 1, file: 1 }` (unique)
|
||||
|
||||
**Upsert logic:**
|
||||
|
||||
- On each context update where `currentFile` is non-null:
|
||||
- Increment `observationCount`
|
||||
- Update `primaryOwner` to the engineer with the most recent `lastSeenAt` on this file
|
||||
- Add engineerId to `contributors` if not present
|
||||
- Update `lastSeenAt`
|
||||
|
||||
**Continual learning:** Hermes loads this collection on startup for the pod. If history exists, it pre-populates the in-memory ownership cache before the first frame arrives.
|
||||
Seeded and fallback graphs are acceptable for demos only when labeled honestly.
|
||||
|
||||
---
|
||||
|
||||
### `events`
|
||||
## Demo Proof Path
|
||||
|
||||
Every coordination event detected by Hermes.
|
||||
|
||||
```ts
|
||||
{
|
||||
_id: ObjectId,
|
||||
podId: string,
|
||||
type: 'DEPENDENCY_READY' | 'BLOCKER_DETECTED' | 'DUPLICATE_WORK',
|
||||
involvedEngineers: string[],
|
||||
file: string | null,
|
||||
reason: string, // 1-sentence explanation from Gemini
|
||||
nudgeSent: boolean, // false if suppressed by cooldown
|
||||
detectedAt: Date
|
||||
}
|
||||
```
|
||||
|
||||
**Index:** `{ podId: 1, detectedAt: -1 }`
|
||||
Observe screen/git state -> detect collision -> send intervention -> accept or
|
||||
dismiss outcome -> recall similar event -> show changed graph or changed
|
||||
behavior.
|
||||
|
||||
---
|
||||
|
||||
### `nudges`
|
||||
## What MongoDB Does Not Store
|
||||
|
||||
Every voice nudge sent to the room.
|
||||
|
||||
```ts
|
||||
{
|
||||
_id: ObjectId,
|
||||
podId: string,
|
||||
eventId: ObjectId, // ref to events collection
|
||||
targetEngineers: string[],
|
||||
message: string, // the spoken text
|
||||
sentAt: Date
|
||||
}
|
||||
```
|
||||
|
||||
**Index:** `{ podId: 1, sentAt: -1 }`
|
||||
|
||||
**Cooldown check:** before sending a nudge, Hermes queries this collection for any nudge in the last 3 minutes for the same `podId`. If found, suppresses the new nudge and marks the event as `nudgeSent: false`.
|
||||
|
||||
---
|
||||
|
||||
## Hermes startup sequence
|
||||
|
||||
```
|
||||
1. Connect to Atlas using MONGODB_URI
|
||||
2. Load ownership_map for this podId
|
||||
3. Build in-memory cache: Map<file, { primaryOwner, contributors }>
|
||||
4. Begin accepting /ingest requests
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Atlas configuration
|
||||
|
||||
- **Cluster tier:** M0 (free) is sufficient for hackathon scale
|
||||
- **Region:** same as DigitalOcean deployment (e.g. NYC1)
|
||||
- **Auth:** connection string in `MONGODB_URI` env var
|
||||
- **Collections created automatically** on first write (no schema migration needed)
|
||||
|
||||
---
|
||||
|
||||
## What MongoDB does NOT store
|
||||
|
||||
- Raw screenshot frames (too large — frames are processed in-memory by Hermes and discarded)
|
||||
- Full Gemini response objects (only extracted fields are stored)
|
||||
- Session recordings
|
||||
- Raw screenshot frames.
|
||||
- Screen recordings.
|
||||
- Secrets or credentials.
|
||||
- Full terminal logs.
|
||||
- Full Gemini response objects beyond extracted fields needed for memory.
|
||||
|
||||
@@ -7,11 +7,16 @@ metadata:
|
||||
|
||||
# PodMan — System Design
|
||||
|
||||
Status: historical reference. Current implementation truth lives in
|
||||
[`../../PLAN.md`](../../PLAN.md), [`../../mongodb.md`](../../mongodb.md),
|
||||
[`../../continual-learning/`](../../continual-learning/), and
|
||||
[`../../graph-discovery/`](../../graph-discovery/).
|
||||
|
||||
## Concept
|
||||
|
||||
PodMan is a real-time AI team coordination agent for software teams. Engineers join a LiveKit room with earbuds. Each engineer's browser PWA captures their screen every 30s and sends it to Hermes (server-side orchestrator on DigitalOcean). Hermes uses Gemini Vision to extract structured context per engineer, detects coordination events, and speaks proactive nudges into the room via Gemini Live 2.5 through LiveKit. MongoDB Atlas stores team state and an ownership map that persists across sessions.
|
||||
PodMan is a real-time AI team coordination agent for software teams. Engineers join a consented LiveKit room and publish screen share when they want PodMan to observe active work. The backend agent samples the LiveKit screen track, uses Gemini Vision to extract structured context, detects coordination risks, and sends intervention cards, Hermes messages, or urgent voice cues through LiveKit. MongoDB Atlas stores observations, collisions, interventions, outcomes, latest engineer state, and the Team memory graph.
|
||||
|
||||
**Track:** Continual Learning — the ownership map makes PodMan faster and smarter each session with no user configuration.
|
||||
**Track:** Continual Learning — accepted and dismissed outcomes make later exact-signature recall and graph memory more useful.
|
||||
|
||||
---
|
||||
|
||||
@@ -19,26 +24,27 @@ PodMan is a real-time AI team coordination agent for software teams. Engineers j
|
||||
|
||||
```
|
||||
┌──────────────── Engineer laptop (Browser PWA) ──────────────────┐
|
||||
│ getDisplayMedia → frame every 30s │
|
||||
│ HTTP POST /ingest → { screenshot, engineerId, podId } │
|
||||
│ LiveKit room joined → receives voice audio from Hermes │
|
||||
│ Earbuds: hears PodMan proactive nudges │
|
||||
│ getDisplayMedia → LiveKit screen-share track │
|
||||
│ Local git watcher → MongoDB engineer_states │
|
||||
│ LiveKit room joined → receives cards, messages, voice cues │
|
||||
│ Earbuds: hears PodMan urgent voice cues │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│ POST /ingest
|
||||
│ LiveKit media + data
|
||||
▼
|
||||
┌────────────────── HERMES (DigitalOcean) ─────────────────────────┐
|
||||
│ 1. Receive frame → Gemini Vision → EngineerContext │
|
||||
│ 2. Write context to MongoDB (per-user state) │
|
||||
│ 3. Update ownership map (file → engineer) │
|
||||
│ 4. Run event detector over all active contexts │
|
||||
│ 5. If event detected → Gemini generates voice message │
|
||||
│ 6. Push audio into LiveKit room via Gemini Live 2.5 │
|
||||
│ 1. Subscribe to screen-share track → Gemini Vision │
|
||||
│ 2. Write observations and per-user state to MongoDB │
|
||||
│ 3. Fuse local git truth from engineer_states │
|
||||
│ 4. Run collision detector over active contexts │
|
||||
│ 5. If risk detected → card/message first, voice only if urgent │
|
||||
│ 6. Push data and optional audio into LiveKit room │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│ read/write
|
||||
▼
|
||||
MongoDB Atlas
|
||||
(engineer_states, ownership_map,
|
||||
events, nudges)
|
||||
(engineer_states, observations,
|
||||
collisions, interventions, outcomes,
|
||||
team_model, graph_nodes, graph_edges)
|
||||
```
|
||||
|
||||
---
|
||||
@@ -48,51 +54,53 @@ PodMan is a real-time AI team coordination agent for software teams. Engineers j
|
||||
### PWA (local agent)
|
||||
|
||||
- Joins LiveKit room via existing `joinPod` flow
|
||||
- Captures frame every 30s via `getDisplayMedia`, compresses to JPEG (1280×720, quality 0.7)
|
||||
- POSTs `{ engineerId, podId, screenshotBase64, capturedAt }` to `POST /ingest`
|
||||
- Receives Hermes audio track (automatic via LiveKit)
|
||||
- Listens for data channel messages → renders nudge feed
|
||||
- Publishes screen share through LiveKit after explicit user action
|
||||
- Receives Hermes audio track through LiveKit when voice is urgent
|
||||
- Listens for data channel messages → renders intervention feed
|
||||
- Two screens: join screen (built), active session screen (to build)
|
||||
|
||||
### Hermes (orchestrator)
|
||||
|
||||
- Express server + LiveKit Agent on DigitalOcean
|
||||
- `POST /ingest`: receives frame, queues for vision
|
||||
- LiveKit agent worker receives sampled screen-share frames and queues them for vision
|
||||
- Vision pipeline: Gemini 2.0 Flash → `EngineerContext`
|
||||
- Confidence gate: discard frames with confidence < 0.6
|
||||
- State writer: upsert `engineer_states` + `ownership_map` in MongoDB
|
||||
- State writer: write `observations`, `collisions`, `interventions`, `outcomes`, and `engineer_states`
|
||||
- Event detector: Gemini text prompt over all active states
|
||||
- Nudge generator: Gemini text → 1–2 sentence spoken message
|
||||
- Voice publisher: Gemini Live 2.5 via LiveKit Agents → audio into room
|
||||
- Data channel: sends structured nudge payload alongside audio
|
||||
- Cooldown: 3 min between nudges per pod
|
||||
- Message generator: Gemini text → short intervention message
|
||||
- Voice publisher: Gemini TTS via LiveKit audio into room for urgent escalation
|
||||
- Data channel: sends structured intervention payload
|
||||
- Cooldown: 3 min between voice cues per pod
|
||||
|
||||
### Gemini usage
|
||||
|
||||
- **Vision:** `gemini-2.0-flash` — screen → `{ currentFile, inferredTask, terminalVisible, recentTerminalOutput, confidence }`
|
||||
- **Event detection:** `gemini-2.0-flash` — all engineer states → `{ event, involvedEngineers, file, reason }`
|
||||
- **Nudge generation:** `gemini-2.0-flash` — event → spoken message text
|
||||
- **Message generation:** `gemini-2.0-flash` — risk → intervention text
|
||||
- **Voice:** `gemini-3.1-flash-tts-preview` via LiveKit audio publication — text → audio
|
||||
|
||||
### MongoDB Atlas (4 collections)
|
||||
### MongoDB Atlas
|
||||
|
||||
- `engineer_states`: latest context per engineer, upserted each ingest
|
||||
- `ownership_map`: file → primaryOwner + contributors, persists across sessions (continual learning)
|
||||
- `events`: all detected coordination events
|
||||
- `nudges`: all voice nudges sent + cooldown history
|
||||
- `engineer_states`: latest context per engineer
|
||||
- `observations`: structured perception records
|
||||
- `collisions`: detected coordination risks
|
||||
- `interventions`: cards, messages, and voice cues sent or suggested
|
||||
- `outcomes`: accepted and dismissed learning signals
|
||||
- `team_model`: durable per-pod summary and seeded graph
|
||||
- `graph_nodes` / `graph_edges`: normalized graph records for `$graphLookup`
|
||||
|
||||
### LiveKit
|
||||
|
||||
- One room per pod
|
||||
- Engineers publish screen track (used client-side for capture — Hermes does not subscribe)
|
||||
- Hermes joins as `podman-hermes`, publishes audio + data channel messages
|
||||
- Engineers publish screen-share tracks
|
||||
- PodMan joins as an agent participant, subscribes to screen share, and publishes audio + data channel messages
|
||||
- Engineers receive audio automatically
|
||||
|
||||
---
|
||||
|
||||
## Event types
|
||||
|
||||
| Event | Trigger | Example nudge |
|
||||
| Event | Trigger | Example intervention |
|
||||
| ------------------ | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
|
||||
| `BLOCKER_DETECTED` | Engineer stuck (error in terminal, same file N frames) + teammate can help | "Carol, looks like you're waiting on auth. Alice is actively building it — hang tight." |
|
||||
| `DEPENDENCY_READY` | Engineer A completes work that Engineer B was waiting on | "Carol, Bob — Alice just got the auth endpoint running. You're clear to integrate." |
|
||||
@@ -102,13 +110,17 @@ PodMan is a real-time AI team coordination agent for software teams. Engineers j
|
||||
|
||||
## Continual learning story
|
||||
|
||||
The `ownership_map` collection persists across sessions. On Hermes startup:
|
||||
The `team_model` graph and accepted outcomes persist across sessions. On graph
|
||||
load:
|
||||
|
||||
1. Load ownership map for this pod from Atlas
|
||||
2. Build in-memory cache: `Map<file, { primaryOwner, contributors }>`
|
||||
3. Event detection uses priors immediately — no ramp-up phase
|
||||
1. Materialize from live MongoDB records when real activity exists.
|
||||
2. Fall back to seeded `team_model.graph`.
|
||||
3. Fall back to a labeled demo graph for stage stability.
|
||||
4. Exact signature recall uses accepted and dismissed outcomes before vector recall.
|
||||
|
||||
**Demo:** Session 1 takes 3 min to first nudge. Session 2 fires in < 30 seconds. That is the learning, visible on stage.
|
||||
**Demo:** The first collision writes an outcome. The second similar collision
|
||||
recalls that memory and changes the graph or behavior. That is the learning
|
||||
visible on stage.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user