Webhooks are a distributed systems problem
Delivery is the easy part. Ordering, duplicates, and the customer endpoint that is down for six hours are the actual work.
Webhooks look like the simple half of an API. You make an HTTP request when something happens. Everything difficult about them is a consequence of that request being made to a machine you do not control.
Delivery is not the hard part
Sending the POST is trivial. What matters is what happens when it fails, and it will fail: the endpoint is down, or behind a firewall that changed, or returning 200 while silently discarding the body, or slow enough that you cannot tell the difference between processing and hanging.
We retry with exponential backoff for twenty-four hours — roughly a dozen attempts, spreading out. After that the event goes to a dead letter queue you can replay from the dashboard. We do not retry forever, because an endpoint that has been down for a day is not coming back in the next thirty seconds and the retries become an attack on a system already having a bad time.
Ordering does not survive
Two events generated a millisecond apart will arrive in whatever order the network and the retries produce. If the first delivery fails and the second succeeds, the second arrives first. There is no fix for this that does not involve a queue per destination and head-of-line blocking, which trades one problem for a worse one.
So we do not promise ordering. Every event carries a sequence number and a timestamp, and the correct way to consume them is to treat each as a statement about state at a time, discarding anything older than what you have already applied.
Handlers that assume ordering work fine for months and then corrupt something during the first incident.
Duplicates are guaranteed
At-least-once is the honest guarantee. Exactly-once delivery over a network is not available to anyone, whatever their marketing says — the acknowledgement can be lost after the work is done, which is the same problem idempotency keys solve on the request side.
Every event has a stable ID. Record the IDs you have processed and ignore repeats. This is five lines of code and it is the difference between a handler that survives an incident and one that double-charges during it.
Signing
Every delivery carries an HMAC-SHA256 signature over the raw body with a secret only you and we hold, plus a timestamp inside the signed payload so an old delivery cannot be replayed at you.
Verify it against the raw bytes, before parsing. Parsing and re-serialising changes the bytes and the signature will not match, which is a support ticket we have answered many times.