Webhook Reliability Patterns Every Backend Developer Should Know

Why webhooks fail in production

Webhooks are the backbone of modern event-driven APIs. When a meeting ends on Gregnote, a payment succeeds on Stripe, or a repository gets a push on GitHub, a POST request fires to your configured endpoint. Simple in theory. Fragile in practice.

In production, your webhook endpoint will be unavailable — for deploys, for database failures, for memory pressure, for network partitions. The meeting transcript your user is waiting for will arrive at exactly the moment your server is restarting. The question isn't whether this will happen, it's whether your system handles it correctly.

Here are the reliability patterns that matter.

Pattern 1: Acknowledge first, process second

The most important rule in webhook handling: return a 2xx status code as fast as possible, before doing any real work.

Why? Webhook providers set timeouts on delivery attempts — typically 5–30 seconds. If your handler makes an LLM API call, writes to a database, sends a Slack message, and posts to Notion before responding, you will time out on any long meeting. The provider will retry, you'll process the same event again, and if your handler isn't idempotent, you'll create duplicate records.

Wrong: `` app.post("/hooks/gregnote", async (req, res) => { const { transcript } = req.body; const summary = await callOpenAI(transcript); // 5-20 seconds await db.save(summary); // might fail await slack.post(summary); // might fail res.sendStatus(200); // too late });

Right: `` app.post("/hooks/gregnote", async (req, res) => { verifySignature(req); // fast, synchronous res.sendStatus(200); // acknowledge immediately queue.add("meeting", req.body); // process async });

The queue can be Redis + BullMQ, SQS, or even a simple in-process queue if you're starting out. The critical thing is that the acknowledgement is decoupled from the processing.

Pattern 2: Always verify signatures

Every serious webhook API signs its payloads. Gregnote uses the same HMAC-SHA256 scheme as Stripe — the signature format you're probably already familiar with.

If you skip signature verification, you're accepting requests from anyone who knows your webhook URL. In security terms, this is a critical vulnerability — an attacker can craft a fake "meeting.completed" event and inject arbitrary meeting data into your application.

import crypto from "crypto";

function verifyGregnoteSig(req: Request, secret: string): void { const header = req.headers["x-gregnote-signature"] as string; const [tPart, vPart] = header.split(","); const timestamp = tPart.slice(2); // "t=..." → timestamp const received = vPart.slice(3); // "v1=..." → hex digest

const payload = timestamp + "." + JSON.stringify(req.body); const expected = crypto .createHmac("sha256", secret) .update(payload) .digest("hex");

if (!crypto.timingSafeEqual( Buffer.from(received, "hex"), Buffer.from(expected, "hex") )) { throw new Error("Invalid signature"); }

// Replay attack protection const age = Date.now() / 1000 - parseInt(timestamp, 10); if (age > 300) throw new Error("Stale timestamp"); } ```

Note the crypto.timingSafeEqual — a regular string comparison is vulnerable to timing attacks. Use the constant-time comparison.

Pattern 3: Idempotent processing

A webhook that was delivered successfully might still be retried. Network conditions mean the provider may not receive your 2xx acknowledgement even though you did respond. Gregnote's retry logic (1m, 5m, 30m, 2h) means you might receive the same event up to five times.

Your processing logic must be idempotent: running it twice produces the same result as running it once.

The simplest implementation: store a set of processed event IDs and skip duplicates.

async function processMeetingEvent(event: GregnoteEvent): Promise<void> {
  const alreadyProcessed = await db.processedEvents.findOne({
    id: event.id
  });
  if (alreadyProcessed) return;

// Process the event await doActualWork(event);

// Mark as processed await db.processedEvents.insert({ id: event.id, processedAt: new Date() }); } ```

For database-backed idempotency, a unique constraint on event_id gives you atomicity for free — if two deliveries of the same event race, one insert will fail with a constraint violation, which you catch and treat as "already processed."

Pattern 4: Exponential backoff for downstream calls

Your webhook handler calls downstream services — an LLM, a Slack API, a database. These calls will fail. Not often, but they will fail.

Don't retry in a tight loop. Use exponential backoff with jitter:

async function withRetry<T>(
  fn: () => Promise<T>,
  maxAttempts = 5
): Promise<T> {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      if (attempt === maxAttempts - 1) throw err;
      const base = Math.pow(2, attempt) * 1000;   // 1s, 2s, 4s, 8s...
      const jitter = Math.random() * 1000;         // ± 1s
      await sleep(base + jitter);
    }
  }
  throw new Error("unreachable");
}

The jitter prevents the thundering herd problem — if 100 webhook jobs fail simultaneously and all retry at the exact same interval, you'll overwhelm the downstream service on the retry. Jitter spreads them out.

Pattern 5: Dead letter queues

Some events will fail no matter how many times you retry. The OpenAI API is down for 3 hours. Your database ran a migration that changed the schema your handler expected. The transcript for a particular meeting is malformed.

These events should not be lost. Put them in a dead letter queue (DLQ) — a separate queue for events that failed all retries. Then:

1. Alert on DLQ depth (pagerduty, Slack alert, email — whatever your team uses).
2. Investigate and fix the root cause.
3. Replay the events from the DLQ.

With BullMQ: ``` const queue = new Queue("meeting-events", { defaultJobOptions: { attempts: 5, backoff: { type: "exponential", delay: 1000 }, }, });

const dlq = new Queue("meeting-events-dlq");

worker.on("failed", async (job, err) => { if (job.attemptsMade >= job.opts.attempts!) { await dlq.add("failed", { job: job.data, error: err.message }); } }); ```

Pattern 6: Observability

You cannot debug what you cannot observe. Log every webhook event, its processing status, any errors, and the time taken.

At minimum: ``` logger.info("webhook.received", { eventId: event.id, eventType: event.type, meetingId: event.meeting_id, });

logger.info("webhook.processed", { eventId: event.id, durationMs: Date.now() - startTime, }); ```

Add structured logging with a consistent schema so you can query by event type, meeting ID, or date range without regex.

Pulling it together

The full pattern for a production webhook handler:

1. Verify signature (synchronous, before anything else)
2. Acknowledge with 200 immediately
3. Enqueue the job
4. Job worker: check idempotency, process with retries, DLQ on final failure
5. Log everything, alert on DLQ

This is the same pattern that Stripe, GitHub, and every other serious webhook provider recommends. Follow it and your integration will survive your server restarting at the worst possible moment.

Related guides

For a full example of handling Gregnote webhooks in a Next.js app, see how to add a meeting bot to Google Meet. For the Slack integration use case, see auto-sending meeting summaries to Slack.