Consider a request to create a record. The receiving system creates it, but the response never reaches the integration. Sending the same create request again could produce a second record. The retry policy needs to account for that uncertainty.

Identify repeated work

Depending on the receiving API, the design might use an idempotency key, an upsert with a stable identifier, or a check of the existing result before deciding what to do next. Each approach depends on the API's guarantees.

An idempotency key identifies an operation so the receiving system can recognize a retry. Reuse it for the same operation only when the API supports that behavior. A later, intentional update is a separate operation.

A lookup before creating a record can help reconcile an uncertain result, but another worker may act between the lookup and the write. Check the API's uniqueness and concurrency guarantees before treating that sequence as duplicate prevention.

Amazon Builders' Library: making retries safe with idempotent APIs

Decide which failures should be retried

Temporary failures and invalid data need different treatment. A retry may help with an unavailable service. A rejected field value usually needs investigation or correction.

For operations that are safe to retry, use bounded retries with increasing delays and jitter to spread requests out. Respect the receiving API's retry guidance. Repeated immediate attempts can add load to a service that is already struggling.

Amazon Builders' Library: timeouts, retries, and backoff with jitter

Make the retry behavior testable

Suppose an integration creates a target record for source order ORD-104. In a test environment, let the receiving system commit the create, then drop the response before the caller receives it. These are illustrative acceptance checks for a design that supports safe retries.

Illustrative lost-response acceptance test
Test stepExpected resultEvidence to inspect
Create succeeds, but its response is lostTreat the result as uncertain, rather than assuming the create failed.The target record exists. The integration retains the source ID and operation identity.
Recover the same operationUse the API's documented idempotency behavior, or reconcile the existing result. Keep one target record for ORD-104.Inspect the target records and recovery log. A successful retry status alone is insufficient.
The result cannot be established safelyHold the operation for review without issuing another blind create.The exception identifies ORD-104, the attempted operation, and the person responsible.
A later intentional change arrivesProcess it as a new operation against the same order.The existing target record reflects the change. The new operation has its own identity.

Give unresolved work an owner

When retries stop, someone needs enough context to act. Record the affected source ID, the attempted operation, and the failure. Agree who investigates, where corrections are made, and how to resume without blindly repeating completed work.

  • Test a repeated request and inspect the resulting records.
  • Test a lost response after the receiving system applies the change.
  • Test invalid data and confirm it reaches the person responsible for correction.
  • Test recovery after the retry limit is reached.

The ownership rules still come first. Decide which value should reach the other system before deciding how to retry its delivery.

Before you sync that field.