openmail/ARCHITECTURE.md

# OpenMail — Architecture

Self-hosted, AI-native mailbox for agents. One Go service, Postgres, and object storage on a VPS,
fronted by Caddy for TLS. This document is the design of record; it favors an MVP that actually
receives and sends real mail over a perfect feature set.

## 0. The honest framing

An agent-mailbox API is ~5% of the work. The other 95% is **mail infrastructure**:

| Hard part | Why it's hard |
|---|---|
| **Inbound** | You must run an SMTP server on port 25 with MX records pointing at it, accept from arbitrary senders, survive spam/abuse, and parse messy real-world MIME. |
| **Outbound deliverability** | Anyone can send SMTP; getting it to the *inbox* needs SPF + DKIM signing + DMARC alignment, PTR/rDNS, warmed IP reputation, feedback loops (FBLs), and bounce/complaint handling. This is the real moat. |
| **MIME parsing** | Multipart trees, encodings, inline vs attachment, calendar parts, broken senders. |
| **Threading** | Stitching messages into conversations from `Message-ID` / `In-Reply-To` / `References` plus subject heuristics. |
| **Quoted-history stripping** | The `extracted_text` feature (reply content without the quoted thread) is its own parsing problem. |

OpenMail is structured so each of these is an isolated, swappable component, and so v1 can lean on a
relay for deliverability while the self-hosted SMTP path matures.

## 1. Component overview

```
                         ┌──────────────────────────────────────────────┐
   Internet senders ───▶ │  smtpd  (inbound, port 25, go-smtp)           │
   (MX → this VPS)       │   → parse MIME → normalize → enqueue          │
                         └───────────────┬──────────────────────────────┘
                                         │ internal queue (Postgres LISTEN/NOTIFY or NATS)
                         ┌───────────────▼──────────────────────────────┐
                         │  core  (Go)                                   │
   Agents / SDKs ──REST──▶   • inbox/message/thread/draft services       │
   MCP clients ───MCP──▶ │   • threading, extraction, labels, search     │
                         │   • auth (API keys, pods), webhooks/WS        │
                         └──────┬───────────────────────┬───────────────┘
                                │                        │
                   ┌────────────▼─────┐      ┌───────────▼───────────────┐
                   │ Postgres         │      │ Object store (S3/MinIO)    │
                   │ metadata + FTS   │      │ raw MIME + attachments     │
                   └──────────────────┘      └────────────────────────────┘
                                         ▲
                         ┌───────────────┴──────────────────────────────┐
   Outbound to world ◀── │  sender  (DKIM-sign → self-host SMTP OR relay)│
                         └──────────────────────────────────────────────┘
```

All four roles (`smtpd`, `core` API, `sender`, MCP) ship in one Go binary with subcommands; deploy
together or split later. Caddy terminates TLS for the HTTP API and auto-manages certs.

## 2. Tech stack

- **Language:** Go (single static binary; matches the deployment-simplicity goal and OpenMail's
  systems nature).
- **HTTP:** `chi` router; `sqlc` + `pgx` for typed Postgres access.
- **Inbound SMTP:** `emersion/go-smtp` (server) — listens on :25, hands raw bytes to the parser.
- **MIME:** `emersion/go-message` (+ `enmime` for lenient real-world parsing); a quote-stripper
  (port of `talon`/`EmailReplyParser` heuristics) for `extracted_text`/`extracted_html`.
- **DKIM:** `emersion/go-msgauth/dkim` for signing (outbound) and verifying (inbound).
- **Outbound:** pluggable `Sender` interface — `smtp` (self-host, MX delivery) or `relay`
  (SES / Postmark / Resend API). Default to relay in v1 for deliverability; self-host is opt-in.
- **Storage:** Postgres (metadata + full-text search via `tsvector`); S3-compatible object store
  (MinIO self-hosted, or AWS S3) for raw `.eml` and attachment blobs.
- **Events:** Postgres `LISTEN/NOTIFY` for the internal inbound queue and for fanning out
  webhooks + a WebSocket stream. (Swap to NATS if volume warrants.)
- **Deploy:** `docker compose` — `caddy`, `openmail` (one image, multiple commands), `postgres`,
  `minio`. MX + SPF/DKIM/DMARC DNS documented in `deploy/`.

## 3. Data model (Postgres)

Mirrors the resource shapes agent tooling already expects, so SDKs/MCP map cleanly.

```sql
pods          (id, name, created_at)                       -- tenant isolation
inboxes       (id, pod_id, address UNIQUE, display_name, client_id, metadata jsonb, created_at, updated_at)
threads       (id, inbox_id, subject, last_message_id, message_count, labels text[], updated_at, created_at)
messages      (id, inbox_id, thread_id, message_id_hdr, in_reply_to, references text[],
               from_addr, to_addrs text[], cc text[], bcc text[], subject,
               preview, text, html, extracted_text, extracted_html,
               raw_object_key,            -- pointer to raw .eml in object store
               labels text[], size_bytes, headers jsonb, ts tsvector,  -- FTS
               created_at, updated_at)
attachments   (id, message_id, filename, content_type, size_bytes, object_key, inline bool, content_id)
drafts        (id, inbox_id, thread_id, to_addrs text[], cc, bcc, subject, text, html, send_at, client_id)
api_keys      (id, pod_id, hash, scopes text[], created_at)   -- bearer tokens, store only the hash
webhooks      (id, pod_id, url, event_types text[], secret)   -- HMAC-signed deliveries
outbox        (id, message_id, status, attempts, last_error, next_attempt_at)  -- send queue + retries
events        (id, inbox_id, type, payload jsonb, created_at) -- audit + WS replay
```

Indexes: `inboxes(address)`, `messages(thread_id)`, `messages(inbox_id, created_at desc)`,
GIN on `messages.ts` (search) and `*.labels`. Raw MIME and attachments live in object storage, not
the DB — rows hold pointers.

## 4. Request / data flows

### Inbound (receive)
1. `smtpd` accepts on :25, enforces basic anti-abuse (rate limit, SPF check, size cap), and writes
   the raw `.eml` to the object store.
2. Parser builds the MIME tree → extracts `text`/`html`, attachments, headers; computes
   `extracted_text` (strip quoted history) and a `preview`.
3. Threading resolves `thread_id` from `In-Reply-To`/`References` → fallback to normalized-subject +
   participants within a window.
4. Row inserted; `LISTEN/NOTIFY` fires → webhook + WebSocket `message.received` event.

### Outbound (send / reply)
1. API builds the RFC 5322 message (sets `In-Reply-To`/`References` for replies), persists a
   `messages` row + `outbox` entry.
2. `sender` DKIM-signs and dispatches via the configured backend (self-host SMTP or relay API).
3. Bounces/complaints (SMTP DSN, or relay webhook) update `outbox.status` and emit events.

### API surface (v1 — AgentMail-shaped)
```
POST   /v0/inboxes                         create inbox
GET    /v0/inboxes / /v0/inboxes/{id}      list / get
POST   /v0/inboxes/{id}/messages/send      send
GET    /v0/inboxes/{id}/messages           list (limit, page_token, labels)
GET    /v0/inboxes/{id}/messages/{id}      get
POST   /v0/inboxes/{id}/messages/{id}/reply
GET    /v0/inboxes/{id}/threads /{id}      list / get
…drafts, webhooks, search…
```
Bearer auth (`Authorization: Bearer`). Keeping paths/shapes close to AgentMail's public v0 means an
existing Go/Python/Node client (or MCP server) can target a self-hosted OpenMail by base-URL swap.

### MCP server
A thin MCP front-end over the same core: tools `create_inbox`, `list_messages`, `get_thread`,
`send_message`, `reply`, `search` — so an agent (Claude Code, etc.) drives its own mailbox directly.

## 5. Deliverability plan (the part that actually matters)

- **v1: relay-by-default.** Send through SES/Postmark/Resend for inbox placement on day one; OpenMail
  still owns inbound, storage, threading, API. Lowest-risk way to be useful immediately.
- **Self-host SMTP send (opt-in):** dedicated IP with PTR/rDNS, SPF (`v=spf1 ip4:… -all`), DKIM
  signing (publish the public key), DMARC (`p=quarantine` → `p=reject`), gradual IP warmup, FBL
  enrollment, and bounce/complaint suppression lists. Documented honestly as a multi-week reputation
  effort, not a flag flip.
- **Inbound auth:** verify SPF/DKIM/DMARC on receipt; expose pass/fail in message metadata; optional
  reject/quarantine policy per inbox.

## 6. MVP milestones

1. **Core API + storage** — inboxes/messages/threads/drafts CRUD against Postgres + MinIO; bearer
   auth; no real mail yet (seed via an ingest endpoint). *Proves the data model + API.*
2. **Inbound** — `go-smtp` on :25, MIME parse, extraction, threading, `message.received` webhook.
   MX a test domain at the VPS. *Proves receive.*
3. **Outbound via relay** — `send`/`reply` through SES/Postmark with DKIM. *Proves the loop:
   an agent receives and replies.*
4. **MCP server** — agent owns and operates an inbox end-to-end.
5. **Self-host SMTP send + deliverability hardening** — the long tail.

## 7. Repo layout (planned)

```
cmd/openmail/         # single binary: `serve`, `smtpd`, `sender`, `mcp` subcommands
internal/core/        # inbox/message/thread/draft services
internal/smtp/        # inbound server + MIME parse + extraction + threading
internal/sender/      # Sender interface + smtp & relay backends + DKIM
internal/store/       # sqlc queries, object-store client
internal/api/         # chi HTTP handlers (v0 surface)
internal/mcp/         # MCP server over core
deploy/               # docker-compose.yml, Caddyfile, DNS (MX/SPF/DKIM/DMARC) notes
db/migrations/        # SQL migrations
```

## 8. Open questions

- Relay vs self-host SMTP as the *default* — relay is pragmatic, self-host is the point. Likely ship
  relay default + self-host documented.
- How far to chase AgentMail API compatibility vs a cleaner native shape.
- Search: Postgres FTS is enough for v1; revisit (Meilisearch/Typesense) only if needed.
- Multi-tenancy depth — `pods` now, full RBAC later.