Webhook Reliability Playbook

Webhook Received, but Nothing Happened

Use this playbook when a webhook exists in Webhook Guard or your endpoint logs, but the expected application behavior did not happen. A 2xx response proves a specific HTTP boundary accepted the request; it does not prove the downstream workflow committed.

Written by TechYelp Engineering

processingqueueworkerHTTP 200
Platform Shopify
Pipeline area After HTTP acceptance
Typical impact A delivery looks successful but no order, inventory, fulfillment, or customer change appears downstream
Suggested owner Application owner with access to endpoint logs, queues, database records, and downstream API logs
Diagnostic complexity High
Recovery risk High if the original handler partially completed an external side effect

First five minutes

  • Identify which component returned the 2xx response.
  • Record the Shopify delivery ID, Webhook Guard event ID, destination endpoint, and downstream attempt number.
  • Find the application trace or queue message created from that exact request.
  • Check whether a downstream record already exists under another identifier.
  • Do not manually retry until partial processing is ruled out.

Evidence to collect

Evidence Where to find it What it proves What it does not prove
HTTP response boundary Shopify logs, Webhook Guard attempt timeline, or endpoint access logs Which component accepted the request and when That asynchronous processing, database commit, or ERP sync completed
Queue message Queue console, worker logs, or message audit table The event moved from request handling into asynchronous processing That a worker completed the business action
Worker execution record Application logs or tracing system filtered by event or business key A worker claimed the message and reached a known result That the database transaction committed unless commit evidence exists
Downstream API response ERP, OMS, WMS, or application outbound logs The target system accepted, rejected, rate-limited, or errored on the business command That local status was updated after the remote call

Focused failure-stage map

Webhook Guard intakeConfirm the received event, headers, payload, and topic.Integration matchConfirm the intended destination matched the topic.Downstream deliveryInspect status, response time, and downstream attempt number.Customer processingFind queue, worker, and database evidence outside Webhook Guard.Business stateVerify ERP, OMS, WMS, or database final state.

Fast decision tree

  1. Which system returned HTTP success: Shopify-to-Webhook Guard, Webhook Guard-to-destination, or the customer endpoint itself?
  2. Was the event durably recorded or queued before that success response? If no, the acceptance boundary is too early.
  3. Did a worker claim the queued work? If no, inspect backlog, visibility timeout, dead-letter queue, and worker availability.
  4. Did the worker call the downstream system? If no, inspect routing, validation, mapping, and swallowed exceptions.
  5. Did the downstream system accept and commit the business action? If no, recover by business object, not by blindly replaying payloads.

What HTTP 200 Proves

HTTP evidenceIt provesIt does not prove
Shopify-to-Webhook Guard 2xxWebhook Guard accepted Shopify's request.Your endpoint or ERP processed the event.
Webhook Guard-to-destination 2xxThe configured destination responded successfully.The destination worker committed database or ERP state.
Application API 2xx from ERPThe ERP API accepted that request boundary.Local sync status, inventory, fulfillment, and customer state are all correct.

Root-cause matrix

Acceptance boundary

Possible cause Evidence that supports it Evidence that eliminates it Corrective action
Endpoint returned 2xx before durable acceptance Access log shows 2xx but no event, queue message, or database record exists A durable event record or queue message exists with the same correlation ID Move the 2xx boundary after authentication and durable record or queue publication.
Signature or request validation result was ignored Invalid signature or malformed payload appears in logs but handler still returned success Verified signature and validated payload were recorded before enqueue Reject invalid requests before processing and record the rejection reason safely.

Async processing

Possible cause Evidence that supports it Evidence that eliminates it Corrective action
Queue publication or worker claim failed No message ID, growing backlog, worker crash, or dead-letter entry Worker trace shows successful claim and completion Fix queue publishing and worker health, then recover only known affected messages.
Database rollback after downstream side effect Remote ERP record exists but local sync status is missing or failed Local transaction commit and remote record are both present Reconcile local status to the remote result before considering replay.

Illustrative example — not a real customer incident

Example diagnosis

EvidenceObserved result
Incident evidenceWebhook Guard recorded the event.
Incident evidenceThe intended integration matched.
Incident evidenceThe customer endpoint returned HTTP 200.
Incident evidenceNo queue record or worker trace exists.
Incident evidenceNo ERP record exists.

Diagnosis

The downstream endpoint acknowledged the request but failed before durable application processing.

Why that diagnosis follows

Webhook Guard can prove downstream HTTP acceptance, but the missing queue, worker, and ERP evidence shows the workflow stopped after the HTTP boundary.

Safest next action

Fix the endpoint's durable-acceptance boundary, confirm idempotency, then recover only the affected event.

What not to do

Do not assume HTTP 200 proves the ERP operation completed.

Detailed diagnostic procedure

  1. Create a timeline with provider delivery time, Webhook Guard received time, destination attempt time, endpoint access log time, queue publish time, worker start time, and final state time.
  2. For each 2xx response, write down who returned it and what had been durably recorded at that instant.
  3. Search the queue and worker logs using the Shopify delivery ID, Webhook Guard event ID, order ID, and destination attempt number.
  4. If there is a worker error, classify it as validation, mapping, authentication, rate limit, timeout, downstream outage, or database rollback.
  5. If no worker evidence exists, treat the failure as handoff or observability failure and recover from a known source-of-truth record.

Immediate containment

  • Stop duplicate-producing workers or retries if side effects may still be in flight.
  • Preserve the original payload, response, and worker logs.
  • Identify all records processed in the same failed deployment window.

Permanent fix

  • Persist event and correlation IDs before acknowledging HTTP success.
  • Track separate states for received, forwarded, accepted, processing, committed, failed, and reconciled.
  • Alert when a 2xx delivery has no downstream completion record within the expected window.

Architecture improvement

  • Use a transactional outbox or durable queue between intake and slow business processing.
  • Make workers idempotent by business-operation key.
  • Add dead-letter handling with a recovery procedure that checks partial side effects first.

Safe recovery gate

  • I confirmed the last durable processing stage.
  • I checked queue, worker, transaction, and downstream records.
  • I checked for partial side effects.
  • I have an idempotent recovery key.
  • I can replay or reprocess only the affected event.

What Webhook Guard can and cannot prove

Can show

  • Whether Webhook Guard received the event and forwarded it to the configured endpoint.
  • The destination response status, response time, and attempt number.
  • Whether the event was classified as duplicate or invalid before forwarding.

Cannot prove

  • A downstream 2xx response does not prove customer application processing completed.
  • Webhook Guard cannot see internal queues, workers, database commits, or ERP calls unless those systems report back separately.

Webhook Guard evidence for this incident

Webhook Guard shows the matched integration and delivery-attempt timeline so you can identify the endpoint status, response time, and attempt number for the exact event.

Limitations

  • A destination HTTP 200 cannot prove worker or ERP completion.
  • Customer queues, worker execution, database commits, and ERP state remain outside Webhook Guard unless those systems report back.

Claim sources

Webhook Guard User Guide · Troubleshoot webhooks

User Guide

Open Delivery Attempts guidance

Production runbook

  1. Identify the 2xx boundary.
  2. Find durable handoff evidence.
  3. Trace worker and downstream processing.
  4. Check partial state.
  5. Recover with idempotency and verification.

Sources and last verification

Sources and verification