Webhook Reliability Playbook

Shopify Webhook Timeouts and Retries

Use this playbook when Shopify or Webhook Guard keeps retrying a webhook, response times approach Shopify's documented five-second delivery deadline, or slow processing risks duplicate side effects.

Written by TechYelp Engineering

timeoutretryasync processingacknowledgment
Platform Shopify
Pipeline area Delivery response and retry boundaries
Typical impact Delayed event processing, duplicate work, failed delivery metrics, and Admin API subscription deletion risk after Shopify-documented consecutive failures
Suggested owner Backend or platform engineer responsible for endpoint latency and queue reliability
Diagnostic complexity Moderate
Recovery risk High if failed acknowledgments caused repeated work before idempotency was added

Verified Shopify timeout and retry facts

Shopify documents HTTPS webhook delivery as a provider-to-intake request with a one-second connection timeout and a five-second timeout for the complete request. Any response outside the 200 range, including 3xx redirects, is treated as an error Shopify timeout docs.

If Shopify receives no response or an error, Shopify retries 8 times over the next 4 hours. After 8 consecutive failures, Shopify automatically deletes the subscription only if it was configured using the Admin API, and sends warning emails to the app emergency developer email address Shopify retry docs.

First five minutes

  • Record response time, status, delivery attempt number, and whether the retry came from Shopify, Webhook Guard, or the application worker.
  • Check whether the endpoint performs database work, ERP calls, or other slow operations before acknowledgment.
  • Confirm whether the request was durably recorded or enqueued before any 2xx response.
  • Identify duplicate side-effect risk before enabling manual retries.
  • Watch current failure rate and backlog so a recovery attempt does not amplify the incident.

Evidence to collect

Evidence Where to find it What it proves What it does not prove
Shopify delivery detail Shopify delivery logs Response status, response time, attempt count, and whether Shopify considered the delivery failed That downstream application processing completed
Webhook Guard attempt timeline Webhook Guard event detail Whether downstream retries came from Webhook Guard and what the endpoint returned That the endpoint worker avoided duplicate processing
Endpoint latency breakdown Application tracing or logs for signature verification, database, queue, and outbound calls Which operation consumed the request-response budget That retries are safe without idempotency evidence
Durable acceptance record Event table, queue message, or outbox record The event survived process crash after acknowledgment That final business processing succeeded

Focused failure-stage map

Shopify deliveryConfirm provider response time, status, and retry attempts.Durable acceptancePersist or enqueue before returning success.AcknowledgmentKeep the HTTP response inside Shopify's documented request deadline.Asynchronous processingMove ERP work behind a durable queue with idempotency.

Fast decision tree

  1. Which layer is retrying: Shopify to Webhook Guard, Webhook Guard to your endpoint, or your worker to the downstream system?
  2. Was a non-2xx, redirect, connection failure, or timeout observed? If yes, fix that response boundary first.
  3. Does the handler do slow business processing before durable acceptance and acknowledgment? If yes, move slow work asynchronously.
  4. Could the same event already have performed a side effect before the timeout? If yes, use the recovery gate before retry.
  5. After the endpoint is stable, recover only the affected events or reconcile current Shopify state.

Root-cause matrix

Request response

Possible cause Evidence that supports it Evidence that eliminates it Corrective action
Slow synchronous processing before acknowledgment Trace shows ERP calls, database locks, cold starts, or queue delays inside the webhook request Request returns after authenticate, validate, and durable enqueue only Move slow processing behind durable queue or outbox and return 2xx only after safe acceptance.
Redirect, non-2xx, or connection failure Delivery logs show 3xx, 4xx, 5xx, DNS, TLS, or connection errors Consistent 2xx status within the response deadline Fix the endpoint URL, TLS, routing, auth, or server error before replay.

Retry safety

Possible cause Evidence that supports it Evidence that eliminates it Corrective action
Retries repeat non-idempotent work Duplicate ERP orders, inventory changes, refunds, labels, or emails after repeated attempts Unique business-operation key allows only one side effect Add idempotency before retrying affected events.
Queue publication is not durable enough Endpoint returned success but no queue message or event record exists after crash Durable record exists before acknowledgment Make durable acceptance atomic and observable.

Illustrative example — not a real customer incident

Example diagnosis

EvidenceObserved result
Incident evidenceShopify delivery duration is 5.4 seconds.
Incident evidenceThe handler calls the ERP API synchronously.
Incident evidenceThe ERP call consumes most of the request time.
Incident evidenceA later Shopify attempt repeats the request.

Diagnosis

The endpoint performs slow downstream work before acknowledgment, causing provider retry risk.

Why that diagnosis follows

The provider-to-intake request exceeded Shopify's documented complete-request timeout because slow ERP work remained in the synchronous request path.

Safest next action

Authenticate, validate, durably persist or enqueue, acknowledge, and move ERP work to asynchronous processing with idempotency.

What not to do

Do not return success before durable acceptance, and do not keep the ERP call in the request path.

Detailed diagnostic procedure

  1. Collect delivery attempts from Shopify and Webhook Guard and label the retry layer for each attempt.
  2. Compare response times against Shopify's documented one-second connection timeout and five-second total request timeout.
  3. Trace the request handler and mark where authentication, validation, persistence, queue publication, acknowledgment, and slow business work happen.
  4. Inspect downstream side effects created during failed or timed-out attempts.
  5. After fixing latency, retry one known-safe event and verify both delivery status and business state.

Immediate containment

  • Temporarily pause manual replay and unsafe worker retries while duplicate risk is unknown.
  • Scale or restore the endpoint only after confirming it will not repeat side effects for old attempts.
  • Prioritize topics with Admin API subscriptions affected by repeated Shopify delivery failures, exhausted provider retries, or high business impact.

Permanent fix

  • Authenticate, validate enough to accept safely, durably record or enqueue, acknowledge, then process asynchronously.
  • Alert on response times approaching the deadline, retry exhaustion, and growing queue backlog.
  • Use idempotency keys for every external side effect.

Architecture improvement

  • Separate Shopify retry, Webhook Guard downstream retry, and application worker retry in logs and dashboards.
  • Store attempt numbers and correlation IDs across all layers.
  • Run reconciliation jobs for critical records after downtime.

Safe recovery gate

  • I identified which layer is retrying.
  • I confirmed whether the event was durably accepted before response.
  • I checked whether timed-out attempts caused side effects.
  • I added or verified idempotency.
  • I can test one known-safe event before resuming bulk recovery.

What Webhook Guard can and cannot prove

Can show

  • Webhook Guard downstream attempts, statuses, response times, and retry numbers.
  • The event history needed to identify affected deliveries before retry.

Cannot prove

  • It cannot prove that a timed-out customer endpoint did not still create a side effect.
  • It cannot make an unsafe downstream operation idempotent by itself.

Webhook Guard evidence for this incident

Webhook Guard response status, response time, and attempt number help distinguish downstream endpoint retry behavior from Shopify provider retry behavior.

Limitations

  • Attempt history cannot make a non-idempotent endpoint safe to retry.
  • Webhook Guard cannot prove whether a timed-out endpoint already created an external side effect unless downstream systems expose that evidence.

Claim sources

Verify webhook deliveries · Troubleshoot webhooks · Webhook Guard User Guide

User Guide

Open Delivery Attempts guidance

Production runbook

  1. Label the retrying layer.
  2. Find the slow or failing boundary.
  3. Move slow work after durable acceptance.
  4. Check duplicate risk.
  5. Recover with a narrow event set or reconciliation.

Sources and last verification

Sources and verification