At-least-once webhooks with a one-row claim table

We forward certain loan lifecycle events to a partner's clickstream webhook. Some of those events have to fire exactly once per lead — a phone verification, a completed credit check — no matter how many times our own systems happen to report the same transition. The problem is that our event stream is anything but single-fire, and the reasons it duplicates are precisely the reasons that make duplication hard to prevent.

The same event reaches us from a controller retry, from a user double-tapping, and — most subtly — from our own notification engine superseding one intent with another. When an initiated event gets quickly overtaken by a credit check inside the drop-off window, the initiated event still needs to forward its webhook before it's superseded and forgotten. So we hook the forwarder at the very top of the orchestration flow, before any supersession logic runs, which guarantees we see every event. The price of that guarantee is that we also see some of them more than once.

Why this mattered

Two failure modes sit on opposite sides of this, and they are not symmetric. A duplicate webhook corrupts a partner's funnel analytics — we double-count a verification, and now their conversion numbers are wrong and we're the cause. That's an awkward conversation. But the opposite failure is worse: dropping an event because we were too aggressive about deduplication silently under-counts, and silent undercounting is the kind of bug you don't discover until someone reconciles two systems months later and asks why they disagree.

So the requirement had to be stated precisely, because the imprecise version ("don't send duplicates") leads you straight into the under-counting trap. What we actually wanted was: at-least-once delivery, deduplicated to exactly-once only on confirmed success. A transient network failure must remain retryable. A delivery we know succeeded must never repeat. The word "confirmed" is doing a lot of work in that sentence, and most of the design is about honoring it.

The approach

The deduplication primitive is almost embarrassingly small: a single table with a unique constraint on the pair of values that identifies a delivery — the lead and the event. Before any HTTP call goes out, a caller tries to insert its claim row using an insert-or-ignore. The database makes that atomic. Exactly one caller's insert actually takes effect; everyone else's quietly does nothing. The caller who won the insert is the one — and the only one — that proceeds to make the webhook call. Everyone who lost simply stops.

Repo.insert_all(Delivery, [row],
  on_conflict: :nothing,
  conflict_target: [:lead_id, :event])

This is the part worth pausing on, because it's why we reached for the database instead of a queue or a distributed lock. Two callers can race into that insert at the same instant; the unique constraint is what arbitrates between them, and it arbitrates atomically with no coordination on our side. We didn't need Redis, we didn't need an idempotency-key middleware, and we didn't need to think about lock timeouts. The database we already trust for everything else hands us atomic claiming as a side effect of a constraint we'd want anyway. The claim table doubles as an audit log of what was delivered and when — the same rows that prevent duplicates also answer "did the partner ever get told about this?"

The detail that turns this from clever into correct is what happens when the webhook call fails. We claim the slot before sending, which means a failed send would, if we did nothing, permanently burn that slot — the event could never be delivered again, and we'd have manufactured exactly the silent-undercount failure we set out to avoid. So on failure we delete the claim row, returning the slot for a future retry. The claim isn't a tombstone; it's a lease. It's held when delivery succeeds and returned when delivery fails.

That ordering — claim, then send, then release on failure — is a deliberate choice over the alternative of sending first and recording afterward. Send-first would never burn a slot, but it would allow a duplicate whenever a send succeeded and the recording crashed. We judged a duplicate webhook to be the worse outcome for a partner's analytics than a delayed one, so we ordered the steps to make duplicates impossible and delays merely possible. It's the kind of decision that's only safe because the failure path — the release — is explicit, deliberate, and tested. Forget the release, and you've built a system that silently stops delivering after its first hiccup.

Not every event touches this machinery, and that restraint is itself a design choice. Only the two events that genuinely need single-fire semantics ever claim a slot. The lifecycle events — sanctioned, disbursed — are meant to fire on every reported transition, so they skip the claim table entirely. We resisted the temptation to make everything idempotent "to be safe," because deduplicating an event that's supposed to repeat is just as much a bug as duplicating one that isn't.

The transport itself is fire-and-forget, run inside a supervised background task, so forwarding never blocks the request that triggered the event. And the HTTP layer is careful to distinguish three outcomes — a clean success, a non-success response from the partner, and a transport-level failure — because only the failures should trip the release valve. Collapsing "the partner said no" and "we couldn't reach the partner" into a single error bucket would have made the release logic lie about what actually happened.

What we learned

A unique constraint plus insert-or-ignore is a surprisingly complete distributed lock, and reaching for it instead of new infrastructure was the right instinct. The thing we'd most want a reader to take away, though, isn't the mechanism — it's that the ordering of claim and send is load-bearing, and the release valve is what makes a claim-before-send safe rather than dangerous.

The honest cost lives entirely in that release path. It's easy to write, easy to forget, and invisible when it's missing until deliveries mysteriously stop. We treat it as a first-class part of the design, not an error-handling afterthought.

What surprised us most was how much the placement of the hook mattered, independent of all the dedup machinery. Putting the forwarder above the supersession logic — rather than buried inside the message-send path — is what decoupled "did we tell the partner this happened" from "did we send the applicant a message." Those turned out to be two different questions, with two different correctness bars, and the original bug was simply trying to answer both in one place.