A webhook is a request TRUSTIVAN makes to a destination you chose. Most problems with one fall into four groups: the destination was refused when you registered it, deliveries are being attempted and failing, your receiver rejects the signature, or the same event arrived twice. Start with the delivery history, which is the troubleshooting view:
Each delivery carries status (pending, delivering, delivered or failed), attempts and max_attempts, next_attempt_at, response_status, duration_ms, and error when something went wrong. It is bounded rather than paginated, because the answer is always near the top. event_id identifies the domain event; id identifies this delivery of it. One event fans out to one delivery per subscribed endpoint, and they share event_id.

​
Signature verification fails

TRUSTIVAN signs every delivery. Your receiver should verify before doing anything with the body. Four headers arrive with each request: The signature is HMAC-SHA256 over exactly this byte string:
Every field that changes the meaning of a delivery is inside the MAC: the version, so a future v2 signature cannot be presented as a v1 one; the delivery ID, so a captured signature cannot be replayed as a different delivery even inside the timestamp window; the timestamp, so it cannot be edited to refresh a stale capture; and the body. When verification fails, it is nearly always one of five things. You are hashing the wrong bytes. The MAC covers the raw body as received, not a re-serialised object. A framework that parses JSON into a map and hands you the map has already destroyed the byte sequence: key order, whitespace and number formatting will not survive a round trip. Capture the raw body before any parsing — most frameworks offer this explicitly, and it is usually the whole bug. You are hashing only the body. The prefix is part of the signing base. A MAC over just the payload will never match. The timestamp is outside the tolerance. A delivery is accepted only if its timestamp is within 5 minutes of your clock in either direction. The check exists because a MAC over a fixed body does not expire on its own, so replay protection has to come from bounding the age — and the timestamp is only trustworthy because it is itself inside the MAC. If verification fails intermittently, especially under load, check clock skew first. Run NTP on the receiver. A container host whose clock has drifted by ten minutes produces exactly this symptom, and nothing else about it looks wrong. Do not widen the window as a fix; you would be trading replay protection for a clock nobody is watching. You are comparing in variable time. Use a constant-time comparison — hmac.Equal in Go, hmac.compare_digest in Python, crypto.timingSafeEqual in Node. A byte-by-byte == on a MAC lets an attacker recover a valid signature from response timing. It will also appear to work perfectly in testing, which is what makes it worth stating. You are checking only the first signature. The header may carry more than one signature, space-separated, during a secret rotation. Accept the delivery if any signature in the header verifies. A receiver that checks only the first will drop deliveries during every rotation.

​
A reference verification

Note raw_body is bytes, taken before parsing.

​
The destination was refused

Symptom. POST /webhooks returns 400 naming the address class, rather than “invalid URL”. A webhook is a request to a destination the customer chooses, made from inside TRUSTIVAN’s network. That is the definition of server-side request forgery, so the destination is checked against a deny list of everything that is not globally routable — loopback, private ranges, carrier-grade NAT, link-local (which is where cloud instance metadata lives), multicast, reserved and documentation ranges, and their IPv6 equivalents. The error names the class — “a private network address”, “a link-local address — this range hosts cloud instance metadata services” — because “10.0.0.5 is a private network address” is actionable and “invalid URL” is not. Two things are worth knowing: The check at registration is the shallow half. The real control runs at connection time, on the resolved address, and it applies to redirects too — the classic bypass is a public hostname that responds with a 302 to the metadata service. A destination that passes registration can still be refused at delivery time if its name resolves somewhere blocked. WEBHOOK_ALLOW_PRIVATE_DESTINATIONS=true permits private and loopback destinations. It exists so a developer running a receiver on localhost, and an end-to-end test running one on 127.0.0.1, exercise the real delivery path rather than a bypassed one. Configuration validation refuses to start a production deployment with it set. If your local receiver is being refused, this is the setting; if you are reaching for it in production, the destination is the thing to change. Destinations must be HTTPS URLs of at most 2048 characters.

​
Deliveries are being retried

A delivery succeeds on a 2xx. Anything else — a 4xx, a 5xx, a connection failure, or no response within the 10-second whole-request budget — is a failure and is retried. Retries are exponential from a 30-second base, doubling, capped at 2 hours, for up to 20 attempts — roughly a day of failures in total. Then the delivery is marked failed and abandoned. Exponential rather than fixed, because the common failure is an endpoint that is down for a while, and a fixed interval turns that into a steady stream of pointless requests at a receiver that is already struggling. The 10-second budget covers connect, write and read. A receiver that has not answered in ten seconds is not going to answer usefully, and a slow one must not be able to hold a delivery worker open — that is how one customer’s broken receiver becomes every customer’s delivery backlog. Acknowledge fast and work afterwards: return 2xx as soon as you have verified the signature and durably enqueued the payload, then do the real work outside the request. An endpoint that keeps failing is eventually disabled. Re-enabling it clears the consecutive-failure count, so an operator who has fixed their receiver is not disabled again by the failures that preceded the fix.

​
Consumers must be idempotent

At-least-once delivery is a guarantee, not an accident. The same event can arrive more than once, and you must handle it. The unavoidable case: your receiver processes a delivery successfully, and the acknowledgement is lost — a connection reset, a timeout on the way back, a load balancer that gave up. TRUSTIVAN has no way to distinguish that from a receiver that never got the request, and the only safe assumption is that it did not arrive. So it retries. The alternative would be at-most-once, which drops events. For a security product, silently losing “a critical vulnerability appeared in production” is a far worse failure than delivering it twice. Deduplicate on TrustOwl-Delivery. Record the ID, in durable storage, at the point you finish processing; treat a delivery whose ID you have already recorded as a no-op and return 2xx. Use TrustOwl-Delivery rather than event_id if you want per-delivery semantics; use event_id if you want to process a domain event once regardless of how many endpoints it fanned out to. Ordering is not guaranteed either. A retried delivery arrives after events that were generated later, so make your handler tolerant of that: prefer fetching current state over reconstructing it from an event sequence.

​
Rotating a secret without dropping deliveries

The response carries the new secret — the only time it is returned — and previous_secret_expires_at, when the old one stops being used. The overlap is 24 hours. During the overlap every delivery is signed with both secrets, and the TrustOwl-Signature header carries both, space-separated. That is what makes rotation a change you can schedule rather than an outage: a secret that took effect the instant it was created would drop every delivery until your receiver was reconfigured. The roll, in order:
  1. Confirm your receiver accepts any signature in the header, not just the first. Do this before rotating, not during.
  2. Rotate. Store the new secret from the response immediately — there is no read-back path, and no endpoint that will ever show it again. Signing secrets are encrypted at rest rather than hashed, because TRUSTIVAN must reproduce them to sign, but nothing reads them out.
  3. Deploy the new secret to your receiver, within the 24 hours. If your receiver can hold two secrets, configure both and verify against either; otherwise swap to the new one, which is safe because deliveries are still signed with it.
  4. Watch GET /webhooks/deliveries for a few deliveries after the swap, confirming status: "delivered" and 2xx responses.
If you miss the window, deliveries start failing signature verification at your end and will be retried for about a day — long enough to notice and rotate again. Rotate again rather than trying to recover the old secret; it is not recoverable. webhooks:manage is required to rotate, and it is not grantable to an API key, so rotation is a human action through a browser session. An endpoint is a destination every future event is copied to, so a machine credential that could add or repoint one could stream your findings somewhere you never chose.

​
Where to go next

  • Credential handling — why webhook secrets are encrypted while API keys are hashed.
  • Scopes — webhooks:read and what it excludes.
  • Security model — the rest of the outbound-request boundary.