OSS/openmail
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Architecture v2: embed Mox (MIT) as a library; native Postgres agent layer

Browse Source 
- Choose Option B: single binary embedding Mox's stateless protocol/crypto/
  delivery packages (dkim, spf, dmarc, dane, mtasts, message, smtpclient, junk,
  dsn, anti-abuse) while owning server loop (go-smtp) + Postgres data model.
- Boundary rule: never import mox smtpserver/imapserver/queue/store/config
  (bbolt/global-config coupled) — that's the line between embed and fork.
- Add MIT LICENSE; reframe README as a real MIT OSS product.
- Deliverability moat leans on mox smtpclient+dane+mtasts; relay fallback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
karti-ai
2026-06-21 12:07:37 -07:00
parent 9812e25973
commit aab2a8558e
3 changed files with 195 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ARCHITECTURE.md
+139 -93
View File
@@ -1,171 +1,217 @@
# OpenMail — Architecture
Self-hosted, AI-native mailbox for agents. One Go service, Postgres, and object storage on a VPS,
fronted by Caddy for TLS. This document is the design of record; it favors an MVP that actually
receives and sends real mail over a perfect feature set.
Agent-native, self-hosted mail server. **One Go binary** that embeds [Mox](https://github.com/mjl-/mox)'s
(MIT) mail internals for the correctness-critical 95% — DKIM, SPF/DMARC, DANE + MTA-STS delivery,
real-world MIME, spam filtering — and owns a **native, agent-shaped data model** (Postgres + object
store) with a REST API and MCP server on top. This document is the design of record. It favors an
MVP that actually receives and sends real mail over a perfect feature set.
## 0. The honest framing
## 0. The decision: embed Mox, don't rebuild and don't just wrap it
An agent-mailbox API is ~5% of the work. The other 95% is **mail infrastructure**:
An agent-mailbox API is ~5% of the work. The other 95% is mail infrastructure (inbound :25 hardening,
outbound deliverability, MIME, threading, quote-stripping, anti-abuse). Three ways to get that 95%:
| Hard part | Why it's hard |
|---|---|
| **Inbound** | You must run an SMTP server on port 25 with MX records pointing at it, accept from arbitrary senders, survive spam/abuse, and parse messy real-world MIME. |
| **Outbound deliverability** | Anyone can send SMTP; getting it to the *inbox* needs SPF + DKIM signing + DMARC alignment, PTR/rDNS, warmed IP reputation, feedback loops (FBLs), and bounce/complaint handling. This is the real moat. |
| **MIME parsing** | Multipart trees, encodings, inline vs attachment, calendar parts, broken senders. |
| **Threading** | Stitching messages into conversations from `Message-ID` / `In-Reply-To` / `References` plus subject heuristics. |
| **Quoted-history stripping** | The `extracted_text` feature (reply content without the quoted thread) is its own parsing problem. |
1. **Build from scratch** — re-implement DKIM/SPF/DMARC/DANE/MTA-STS/MIME ourselves. Correct but slow,
and we'd do it worse than a production server at first.
2. **Sidecar an existing server** (run Mox as a separate process, read it over IMAP). Fast, but we
don't own the data model — threads/labels/`extracted_text` would be re-derived from IMAP, and we
ship two coupled processes.
3. **Embed Mox's packages** into our own binary and own the storage + API. ← **chosen.**
OpenMail is structured so each of these is an isolated, swappable component, and so v1 can lean on a
relay for deliverability while the self-hosted SMTP path matures.
Why (3) is viable: **Mox has no `internal/` directory** — every package is importable — and the
README states *"most non-server Go packages are written to be reusable."* The real boundary is not
visibility, it's **storage coupling**: Mox's *server + store* packages are welded to its embedded
`bstore`/`bbolt` store and global config; its *protocol/crypto/parsing* packages are stateless and
clean. We embed the latter and replace the former with our own Postgres-backed server.
### What we embed from Mox vs. what we own
| Concern | Source | Notes |
|---|---|---|
| MIME parsing | **mox `message`** | Battle-tested on real, broken senders. Better than enmime for our needs. |
| DKIM sign + verify | **mox `dkim`** | |
| SPF / DMARC | **mox `spf`, `dmarc`** | Inbound auth verification + policy. |
| Secure outbound delivery | **mox `smtpclient`, `dane`, `mtasts`, `dns`** | The deliverability moat — DANE/MTA-STS MX delivery, mostly solved by reusing this. |
| Bounces (DSN) | **mox `dsn`** | Parse/generate delivery status notifications. |
| Anti-abuse | **mox `iprev`, `dnsbl`, `ratelimit`** | rDNS check, blocklists, rate limiting on :25. |
| Spam filtering | **mox `junk`** | Built-in Bayesian filter; per-inbox train/score. |
| SMTP auth | **mox `sasl`, `scram`** | For the submission (send) path. |
| — | — | — |
| Inbound SMTP **server loop** | **`emersion/go-smtp`** | Clean, embeddable listener/session. We do NOT import mox `smtpserver` (store-coupled). |
| Storage | **OpenMail (Postgres + S3)** | Native agent data model. NOT mox `store`/bbolt. |
| API, MCP, threading, extraction, webhooks, auth, multi-tenancy | **OpenMail** | The agent-native layer — our differentiation. |
> **Boundary rule:** never import `smtpserver`, `imapserver`, `queue`, `store`, or `mox-` (config) —
> they assume Mox's storage/config. Importing them is the line between "embed as a library" and
> "fork Mox." We stay on the library side.
## 1. Component overview
```
┌──────────────────────────────────────────────┐
Internet senders ───▶ │ smtpd (inbound, port 25, go-smtp) │
(MX → this VPS) │ → parse MIME → normalize → enqueue │
└───────────────┬──────────────────────────────┘
│ internal queue (Postgres LISTEN/NOTIFY or NATS)
┌───────────────▼──────────────────────────────┐
│ core (Go) │
Agents / SDKs ──REST──▶ • inbox/message/thread/draft services │
MCP clients ───MCP──▶ │ • threading, extraction, labels, search │
│ • auth (API keys, pods), webhooks/WS │
└──────┬───────────────────────┬───────────────┘
│ │
┌────────────▼─────┐ ┌───────────▼───────────────┐
│ Postgres │ │ Object store (S3/MinIO) │
│ metadata + FTS │ │ raw MIME + attachments │
└──────────────────┘ └────────────────────────────┘
┌──────────────────────────────────────────────────────┐
Internet senders ───▶ │ smtpd (go-smtp on :25) │
(MX → this VPS) │ iprev/dnsbl/ratelimit → SPF/DKIM/DMARC verify │
│ → mox/message parse → mox/junk score → enqueue │ ← mox pkgs
└───────────────┬──────────────────────────────────────┘
│ internal queue (Postgres LISTEN/NOTIFY)
┌───────────────▼──────────────────────────────────────┐
│ core (Go) │
Agents / SDKs ──REST──▶ • inbox/message/thread/draft services │
MCP clients ───MCP──▶ │ • threading, extracted_text, labels, FTS search │
│ • auth (API keys, pods), webhooks/WS │
└──────┬───────────────────────────────┬────────────────┘
│ │
┌────────────▼─────┐ ┌─────────────▼──────────────┐
│ Postgres │ │ Object store (S3/MinIO) │
│ metadata + FTS │ │ raw .eml + attachments │
└──────────────────┘ └────────────────────────────┘
▲
┌───────────────┴──────────────────────────────┐
Outbound to world ◀── │ sender (DKIM-sign → self-host SMTP OR relay)│
└──────────────────────────────────────────────┘
┌───────────────┴──────────────────────────────────────┐
Outbound to world ◀── │ sender (mox/dkim sign → backend) │
│ self-host: mox/smtpclient + dane + mtasts (MX) │ ← mox pkgs
│ relay: SES / Postmark / Resend API │
└──────────────────────────────────────────────────────┘
```
All four roles (`smtpd`, `core` API, `sender`, MCP) ship in one Go binary with subcommands; deploy
together or split later. Caddy terminates TLS for the HTTP API and auto-manages certs.
All roles (`smtpd`, `core` API, `sender`, `mcp`) ship in one binary with subcommands; deploy together
or split later. Caddy terminates TLS for the HTTP API and auto-manages certs.
## 2. Tech stack
- **Language:** Go (single static binary; matches the deployment-simplicity goal and OpenMail's
systems nature).
- **Language:** Go — single static binary, and the whole mature self-hosted-mail ecosystem (Mox,
go-smtp, go-message) is Go, keeping us MIT-compatible. Rust (Stalwart) is AGPL → ruled out.
- **HTTP:** `chi` router; `sqlc` + `pgx` for typed Postgres access.
- **Inbound SMTP:** `emersion/go-smtp` (server) — listens on :25, hands raw bytes to the parser.
- **MIME:** `emersion/go-message` (+ `enmime` for lenient real-world parsing); a quote-stripper
(port of `talon`/`EmailReplyParser` heuristics) for `extracted_text`/`extracted_html`.
- **DKIM:** `emersion/go-msgauth/dkim` for signing (outbound) and verifying (inbound).
- **Outbound:** pluggable `Sender` interface — `smtp` (self-host, MX delivery) or `relay`
(SES / Postmark / Resend API). Default to relay in v1 for deliverability; self-host is opt-in.
- **Storage:** Postgres (metadata + full-text search via `tsvector`); S3-compatible object store
(MinIO self-hosted, or AWS S3) for raw `.eml` and attachment blobs.
- **Events:** Postgres `LISTEN/NOTIFY` for the internal inbound queue and for fanning out
webhooks + a WebSocket stream. (Swap to NATS if volume warrants.)
- **Inbound server:** `emersion/go-smtp` listener → handed to mox packages for verify/parse/score.
- **Mail correctness/delivery:** mox `message`, `dkim`, `spf`, `dmarc`, `dane`, `mtasts`, `dns`,
`smtpclient`, `dsn`, `junk`, `iprev`, `dnsbl`, `ratelimit`, `sasl`, `scram` (see §0 table).
- **Quote stripping:** a `talon`/`EmailReplyParser`-style heuristic pass for `extracted_text`.
- **Storage:** Postgres (metadata + `tsvector` FTS); S3-compatible object store (MinIO self-host, or
S3) for raw `.eml` and attachment blobs.
- **Events:** Postgres `LISTEN/NOTIFY` for the internal inbound queue and to fan out webhooks + a
WebSocket stream. (Swap to NATS only if volume warrants.)
- **Deploy:** `docker compose` — `caddy`, `openmail` (one image, multiple commands), `postgres`,
`minio`. MX + SPF/DKIM/DMARC DNS documented in `deploy/`.
`minio`. MX + SPF/DKIM/DMARC/DANE/MTA-STS DNS documented in `deploy/`.
## 3. Data model (Postgres)
Mirrors the resource shapes agent tooling already expects, so SDKs/MCP map cleanly.
Native and agent-shaped (not Mox's per-account bbolt index). Resource shapes mirror what agent tooling
expects so SDKs/MCP map cleanly.
```sql
pods (id, name, created_at) -- tenant isolation
inboxes (id, pod_id, address UNIQUE, display_name, client_id, metadata jsonb, created_at, updated_at)
inboxes (id, pod_id, address UNIQUE, display_name, dkim_selector, metadata jsonb, created_at, updated_at)
threads (id, inbox_id, subject, last_message_id, message_count, labels text[], updated_at, created_at)
messages (id, inbox_id, thread_id, message_id_hdr, in_reply_to, references text[],
from_addr, to_addrs text[], cc text[], bcc text[], subject,
preview, text, html, extracted_text, extracted_html,
raw_object_key, -- pointer to raw .eml in object store
labels text[], size_bytes, headers jsonb, ts tsvector, -- FTS
spf, dkim, dmarc, -- inbound auth verdicts (from mox pkgs)
junk_score, labels text[], size_bytes, headers jsonb, ts tsvector, -- FTS
created_at, updated_at)
attachments (id, message_id, filename, content_type, size_bytes, object_key, inline bool, content_id)
drafts (id, inbox_id, thread_id, to_addrs text[], cc, bcc, subject, text, html, send_at, client_id)
api_keys (id, pod_id, hash, scopes text[], created_at) -- bearer tokens, store only the hash
api_keys (id, pod_id, hash, scopes text[], created_at) -- bearer tokens; store only the hash
webhooks (id, pod_id, url, event_types text[], secret) -- HMAC-signed deliveries
outbox (id, message_id, status, attempts, last_error, next_attempt_at) -- send queue + retries
events (id, inbox_id, type, payload jsonb, created_at) -- audit + WS replay
domains (id, pod_id, name, dkim_privkey_ref, verified bool) -- per-domain DKIM keys + DNS state
```
Indexes: `inboxes(address)`, `messages(thread_id)`, `messages(inbox_id, created_at desc)`,
GIN on `messages.ts` (search) and `*.labels`. Raw MIME and attachments live in object storage, not
the DB — rows hold pointers.
GIN on `messages.ts` (search) and `*.labels`. Raw MIME + attachments live in object storage; rows
hold pointers.
## 4. Request / data flows
### Inbound (receive)
1. `smtpd` accepts on :25, enforces basic anti-abuse (rate limit, SPF check, size cap), and writes
the raw `.eml` to the object store.
2. Parser builds the MIME tree → extracts `text`/`html`, attachments, headers; computes
`extracted_text` (strip quoted history) and a `preview`.
3. Threading resolves `thread_id` from `In-Reply-To`/`References` → fallback to normalized-subject +
participants within a window.
4. Row inserted; `LISTEN/NOTIFY` fires → webhook + WebSocket `message.received` event.
1. `go-smtp` accepts on :25; anti-abuse gate via mox `iprev`/`dnsbl`/`ratelimit` + size cap; raw `.eml`
written to object store.
2. mox `spf`/`dkim`/`dmarc` verify the message; verdicts recorded. mox `message` builds the MIME tree
→ `text`/`html`, attachments, headers. mox `junk` scores spam. Quote-stripper computes
`extracted_text` + `preview`.
3. Threading resolves `thread_id` from `In-Reply-To`/`References`, falling back to normalized-subject
+ participants within a window.
4. Row inserted; `LISTEN/NOTIFY` fires → `message.received` webhook + WebSocket event.
### Outbound (send / reply)
1. API builds the RFC 5322 message (sets `In-Reply-To`/`References` for replies), persists a
`messages` row + `outbox` entry.
2. `sender` DKIM-signs and dispatches via the configured backend (self-host SMTP or relay API).
3. Bounces/complaints (SMTP DSN, or relay webhook) update `outbox.status` and emit events.
1. API builds the RFC 5322 message (sets `In-Reply-To`/`References` on replies); persists a `messages`
row + `outbox` entry.
2. `sender` mox-`dkim`-signs, then dispatches via the configured backend:
- **self-host:** mox `smtpclient` + `dane` + `mtasts` for authenticated, secure MX delivery.
- **relay:** SES/Postmark/Resend API.
3. Bounces/complaints (mox `dsn` parse, or relay webhook) update `outbox.status` and emit events.
### API surface (v1 — AgentMail-shaped)
```
POST /v0/inboxes create inbox
GET /v0/inboxes / /v0/inboxes/{id} list / get
GET /v0/inboxes / /v0/inboxes/{id} list / get
POST /v0/inboxes/{id}/messages/send send
GET /v0/inboxes/{id}/messages list (limit, page_token, labels)
GET /v0/inboxes/{id}/messages/{id} get
POST /v0/inboxes/{id}/messages/{id}/reply
GET /v0/inboxes/{id}/threads /{id} list / get
GET /v0/inboxes/{id}/threads / /{id} list / get
…drafts, webhooks, search…
```
Bearer auth (`Authorization: Bearer`). Keeping paths/shapes close to AgentMail's public v0 means an
existing Go/Python/Node client (or MCP server) can target a self-hosted OpenMail by base-URL swap.
Bearer auth (`Authorization: Bearer`). Keeping paths/shapes close to AgentMail's public v0 lets an
existing client (or MCP server) target a self-hosted OpenMail by base-URL swap. Compatibility yields
to a cleaner native shape where they conflict.
### MCP server
A thin MCP front-end over the same core: tools `create_inbox`, `list_messages`, `get_thread`,
`send_message`, `reply`, `search` — so an agent (Claude Code, etc.) drives its own mailbox directly.
A thin MCP front-end over `core`: tools `create_inbox`, `list_messages`, `get_thread`, `send_message`,
`reply`, `search` — so an agent drives its own mailbox directly.
## 5. Deliverability plan (the part that actually matters)
## 5. Deliverability plan (the moat — and why embedding Mox helps most here)
- **v1: relay-by-default.** Send through SES/Postmark/Resend for inbox placement on day one; OpenMail
still owns inbound, storage, threading, API. Lowest-risk way to be useful immediately.
- **Self-host SMTP send (opt-in):** dedicated IP with PTR/rDNS, SPF (`v=spf1 ip4:… -all`), DKIM
signing (publish the public key), DMARC (`p=quarantine` → `p=reject`), gradual IP warmup, FBL
enrollment, and bounce/complaint suppression lists. Documented honestly as a multi-week reputation
effort, not a flag flip.
- **Inbound auth:** verify SPF/DKIM/DMARC on receipt; expose pass/fail in message metadata; optional
reject/quarantine policy per inbox.
- **Self-host send is achievable from day one for the protocol parts** because mox `smtpclient` +
`dane` + `mtasts` + `dkim` already implement authenticated, DANE/MTA-STS-secured MX delivery. What
remains is *operational*, not code: dedicated IP with PTR/rDNS, SPF (`v=spf1 ip4:… -all`), published
DKIM key, DMARC (`p=quarantine`→`p=reject`), gradual IP warmup, FBL enrollment, suppression lists.
- **Relay backend** (SES/Postmark/Resend) ships alongside for instant inbox placement while a
self-host IP warms up — chosen per-domain.
- **Inbound auth:** SPF/DKIM/DMARC verified on receipt (mox pkgs); verdicts exposed in message
metadata; optional reject/quarantine policy per inbox.
## 6. MVP milestones
1. **Core API + storage** — inboxes/messages/threads/drafts CRUD against Postgres + MinIO; bearer
auth; no real mail yet (seed via an ingest endpoint). *Proves the data model + API.*
2. **Inbound** — `go-smtp` on :25, MIME parse, extraction, threading, `message.received` webhook.
1. **Core API + storage** — inboxes/messages/threads/drafts CRUD on Postgres + MinIO; bearer auth;
ingest endpoint to seed without real mail. *Proves the data model + API.*
2. **Inbound** — `go-smtp` on :25 + mox verify/parse/junk + threading + `message.received` webhook.
MX a test domain at the VPS. *Proves receive.*
3. **Outbound via relay** — `send`/`reply` through SES/Postmark with DKIM. *Proves the loop:
an agent receives and replies.*
3. **Outbound via relay** — `send`/`reply`, mox-DKIM-signed, through a relay. *Proves the loop: an
agent receives and replies.*
4. **MCP server** — agent owns and operates an inbox end-to-end.
5. **Self-host SMTP send + deliverability hardening** — the long tail.
5. **Self-host SMTP send** — wire mox `smtpclient`/`dane`/`mtasts`; the deliverability/warmup long tail.
## 7. Repo layout (planned)
```
cmd/openmail/ # single binary: `serve`, `smtpd`, `sender`, `mcp` subcommands
internal/core/ # inbox/message/thread/draft services
internal/smtp/ # inbound server + MIME parse + extraction + threading
internal/sender/ # Sender interface + smtp & relay backends + DKIM
internal/smtp/ # go-smtp inbound server; calls mox verify/parse/junk + threading
internal/sender/ # Sender interface: smtp (mox smtpclient+dane+mtasts) & relay backends; mox dkim
internal/store/ # sqlc queries, object-store client
internal/api/ # chi HTTP handlers (v0 surface)
internal/mcp/ # MCP server over core
deploy/ # docker-compose.yml, Caddyfile, DNS (MX/SPF/DKIM/DMARC) notes
deploy/ # docker-compose.yml, Caddyfile, DNS (MX/SPF/DKIM/DMARC/DANE/MTA-STS) notes
db/migrations/ # SQL migrations
```
## 8. Open questions
## 8. Licensing & attribution
- Relay vs self-host SMTP as the *default* — relay is pragmatic, self-host is the point. Likely ship
relay default + self-host documented.
- How far to chase AgentMail API compatibility vs a cleaner native shape.
OpenMail is **MIT** (see `LICENSE`). Every dependency is permissive: Mox (MIT), `emersion/go-smtp`
and `go-message` (MIT), `pgx`/`chi` (MIT/BSD). We import Mox as a Go module (no vendored source in
this repo), but credit it prominently in the README; if we ever vendor Mox source we must retain its
MIT notice. **No GPL/AGPL** code anywhere — Maddy (GPL-3.0) and Stalwart (AGPL) are deliberately
avoided so the project stays MIT.
## 9. Open questions
- **mox `message` ⇄ go-smtp seam:** confirm mox's parser consumes a raw reader cleanly outside mox's
store (spike in milestone 2). If any Tier-1 package drags in `mox-`/`store` transitively, isolate
behind a thin adapter or vendor just that file.
- How far to chase AgentMail API compatibility vs. a cleaner native shape.
- Search: Postgres FTS is enough for v1; revisit (Meilisearch/Typesense) only if needed.
- Multi-tenancy depth — `pods` now, full RBAC later.
- Whether to expose an optional IMAP read path (via mox `imapserver` over our store would mean forking
it — more likely a thin standalone IMAP front-end later, kept out of v1).
```