Idempotency vs Deduplication

Scenario

The same Shopify order payload arrives twice, and the ERP creates two orders because the handler only checked recent request logs.

Operational problem

A deduplication check can suppress exact repeated deliveries, but it does not protect every side effect when workers retry or when two requests race.

Incorrect implementation

• Drop every payload with the same order ID, even if the order legitimately changed later.
• Deduplicate in memory or in short-lived logs.
• Create external side effects before acquiring a unique business-operation key.

Safer implementation

• Deduplicate delivery retries by stable delivery identifiers.
• Use an idempotency key for each business operation, such as create ERP order for Shopify order ID.
• Back the key with a database uniqueness constraint or transactional idempotency record.

Evidence and observability

• Shopify delivery ID, Shopify event correlation ID, business object ID, and operation key.
• Duplicate suppression result.
• External system ID linked to the idempotency record.

Deduplication vs idempotency

Identifier examples

• Shopify delivery-ID example: use the individual delivery identifier to identify repeated copies of the same delivery attempt.
• Shopify event-correlation example: use the event correlation identifier to group deliveries caused by the same merchant action.
• Business-operation-key example: create-erp-order:shopify-order-123 protects the ERP create operation even if the worker runs twice.
• Legitimate update example: the same Shopify order can emit different updates as fulfillment, payment, cancellation, refund, or address state changes.

Database uniqueness example

Use a durable uniqueness boundary around the business operation, not only around the inbound HTTP request. The exact syntax depends on your datastore, but the model is a unique operation key plus a transactional status update.

CREATE TABLE webhook_operations (
  operation_key TEXT PRIMARY KEY,
  source_event_id TEXT NOT NULL,
  status TEXT NOT NULL,
  external_system_id TEXT,
  created_at TIMESTAMP NOT NULL
);

Transaction-boundary example

• Start a transaction and insert the operation key before creating the external side effect.
• If the key already exists, load the existing result and do not create a second ERP order, fulfillment, refund, inventory movement, email, or shipping label.
• Commit the local operation state and external identifier together where possible.
• If the external system cannot participate in the transaction, store a pending state and reconcile by external identifier before retrying.

Why HTTP deduplication is not enough

• One HTTP delivery can enqueue two jobs if the handler races, retries internally, or writes to a queue before failing.
• Two workers can process the same queued message if visibility timeouts, crashes, or manual replay overlap.
• Deduplicating the inbound request does not prevent duplicate ERP orders unless the ERP create operation has its own idempotency key.

Pipeline context

Text equivalent: diagnose the event from source action, subscription, provider delivery, intake, routing, downstream delivery, customer processing, and final business state.

Related playbooks

Product visibility

Sources and last verification