Engineering9 min read

How webhook fan-out works under the hood

A technical walkthrough of our delivery queue, retry logic, and how we guarantee at-least-once delivery for every inbound message.

SC

Sam Chen

Engineer · June 24, 2025

When an email arrives at one of your Programmable Inbox inboxes, it passes through a delivery pipeline before hitting your webhook endpoint. This post walks through exactly what happens and why we built it the way we did.

The delivery pipeline

Every inbound message goes through five stages:

  1. 1Receive — SMTP server accepts the connection, validates the recipient domain
  2. 2Parse — Raw MIME is parsed into structured JSON (headers, body, attachments)
  3. 3Enqueue — Message is written to a durable queue before any delivery attempt
  4. 4Fan-out — Delivery workers pick up the message and POST to each registered endpoint
  5. 5Confirm — Endpoints that return 2xx are marked delivered; others are queued for retry

The queue is the critical part. We write to it synchronously during the SMTP transaction, so even if all your webhook endpoints are down, no messages are lost.

Retry logic

We use exponential backoff with jitter:

attempt 1: immediate
attempt 2: 30s ± 5s
attempt 3: 2m ± 15s
attempt 4: 10m ± 60s
attempt 5: 1h ± 10m

After five failures, the message moves to a dead-letter queue. You can inspect and replay dead-letter messages via the API:

bash
curl https://api.programmableinbox.dev/inboxes/{id}/dead-letter \
  -H "Authorization: Bearer $API_KEY"

Ordering guarantees

We guarantee at-least-once delivery, not exactly-once. Your endpoint may receive the same message more than once in edge cases (network partitions, retries after timeouts). We include an X-Inbox-Message-Id header on every POST so you can deduplicate on your side.

We considered exactly-once delivery but ultimately decided the complexity wasn't worth it for most use cases. If you need strict deduplication, idempotency keys are the right tool.

What's next

We're working on streaming delivery via WebSockets for latency-sensitive use cases, and a replay-from-timestamp feature that lets you reprocess all messages from a given point in time. Both are tracked in our public roadmap on GitHub.