Architecture & Operations
A timeout is not a decline
Why transport failure and economic failure are different, and what evidence you need before acting again.
The rule to remember: a timeout tells you that a connection failed. It tells you nothing about the money. Until a provider status, a statement or a reconciliation shows a final outcome, the payout is in an unknown state, and the only safe next action is to find out which state it is in.
The job
A marketplace sends a seller payout of 1,500,000 COP through a bank-transfer provider. The request leaves at 14:02:11. At 14:02:41 the HTTP client gives up and raises a timeout. Somebody's code now has to decide what that timeout means, and the decision either protects the seller payout or sends it twice.
Before you start
- Written for engineering leads, heads of payments and operators who own payout or withdrawal flows.
- Assumes you send value to third parties through at least one external provider: a bank, a PSP or a local rail.
- Uses Orangepill payouts as the worked example. The reasoning applies to any integration where a request to a provider can move money.
- Maturity referenced here: payouts and idempotency keys are in production. Payout failover and Resolve are in development and are labelled where they appear.
Flow
- Payout sent Provider A
- Timeout Connection failed, outcome unknown
- Gather evidence Status query, provider reference, statement
- Completed Record it. Do not send again
- Failed, final A new attempt may be safe
- Still unknown Wait, block re-sending, escalate
The reasoning
A sequence that pays a seller twice
Nothing in this sequence is unusual. Each step is reasonable code on its own.
- 14:02:11 Your service sends the payout to Provider A.
- 14:02:12 Provider A accepts it and submits the transfer to the bank network. Your service does not see this.
- 14:02:41 Your HTTP client times out after 30 seconds. There is no response body.
- 14:02:42 A retry rule reads the timeout as a failure and sends the payout again with a fresh reference.
- 14:02:44 Provider A accepts the second request. The reference is new, so nothing looks like a duplicate.
- 14:03:05 Both transfers settle. The seller has 3,000,000 COP. Your records show one payout that failed and was retried.
Recovering the second 1,500,000 COP now depends on the seller agreeing to send it back. On most bank-transfer rails there is no undo once the transfer has settled.
Transport failure and economic failure
A transport failure means the request or the response did not make it. It describes the connection. An economic failure means the provider decided not to move the money. It describes the value. A timeout is a transport failure. "Beneficiary account closed", returned before the provider accepted anything, is an economic failure.
Signals that usually describe the money:
- An explicit rejection, with a reason, returned before the provider accepted the transfer.
- A provider status query that returns a final failed status for the transfer.
- A statement or settlement line that shows the debit, or shows the transfer returned.
Signals that only describe the connection:
- A timeout, a connection reset, or a 5xx response after the request was sent.
- No webhook, or a webhook that has not arrived yet.
- A status of
pendingorprocessingthat has lasted longer than you expected.
Why a blind retry duplicates value
The retry in the sequence above used a fresh reference, so Provider A had no way to connect it to the first request. Reusing the same reference helps only if that provider deduplicates on it for that endpoint and for long enough, which varies by provider. And a retry sent to a different provider is a new instruction as far as that provider is concerned.
Your own idempotency key protects your system from creating two payout records. It does not stop a provider from executing two transfers that it received as separate requests.
Unknown is a state, and it should stay one
Most duplicate payouts start with one line of code that maps a timeout to failed. From there the
product offers a retry button, the wallet shows the funds as available again, and a second payout looks like the
correct thing to do.
Keep the payout non-terminal. Do not release the value for other use. Show the user that the payout is processing. A payout in an unknown state costs time; a payout wrongly marked failed can cost the full amount.
Evidence before another attempt
Work through these in order. Stop as soon as one of them gives a final answer.
- Your own record. Did the payout get created, and under which key? With Orangepill, read
GET /v4/payouts/{payoutId}, or send the same request again with the sameidempotency_keyto get the existing payout back. - The provider reference. If the provider returned a transfer id before the connection broke, you can ask about that exact transfer. If it did not, you may have to search by your own reference, amount and time.
- A provider status query. A final status from the provider for that transfer. "Processing" is not an answer yet.
- Settlement or statement evidence. The bank statement or settlement file shows the debit or the return. On some rails this is the first point of certainty, and it can arrive a day later.
- Reconciliation. The provider's record and your ledger agree on what happened. Only when both show that no money moved is a new attempt safe, and it should be recorded as a new attempt linked to the first.
What Orangepill does with a payout timeout today
- Your request to Orangepill times out. Send it again with the same
idempotency_key. You get the existing payout back, not a second one. The status in that response is currently alwayspending, so read the payout to see its real state. - Orangepill's request to the provider times out. If the call times out, the connection resets
or the provider returns a 5xx, Orangepill does not mark the payout failed and does not send it again. The
payout stays
pendingwith noproviderPayoutIdand is flagged for operator attention. A 4xx rejection, or a connection refused before the request reached the provider, marks the payoutfailedwithINITIATE_FAILEDand releases the reservation. - The provider accepted the payout. Orangepill queries the provider's status for that transfer until it reports completed or failed. The interval starts at 30 seconds and doubles, up to 15 minutes, for at most 10 checks. That sequence can run longer than the one-hour reservation described below.
- The provider never gives a final answer. When the checks run out, the payout ends
failedwithRETRY_EXHAUSTED, although the provider never confirmed a failure. That code means Orangepill stopped asking. It is not a decline, and it is not evidence that the money did not move. Treat it as unknown and do not pay again until the provider or a statement confirms. - No automatic failover. A payout is not sent to another provider after a timeout or a failure. Payout failover is in development.
- Reservations expire. A wallet-funded payout records a reservation that expires after one hour.
If the payout is still
pendingorprocessingafter an hour, the reservation is released while the payout may still complete. (Today that reservation also does not reduce the wallet's available balance, so do not rely on it as a spending lock.)
Where Resolve fits In Development
The payouts that cannot safely continue on their own, the flagged and the exhausted ones, are the cases Resolve is being built for. The intended sequence is Investigate, Establish, Decide, Verify: collect evidence, establish whether the first attempt is final, choose a disposition (Continue, Wait, Remediate or Escalate) and confirm the result after the owning system acts.
Resolve is not available today and has no API. Today these payouts need a person on your side to gather the evidence above and decide.
What can go wrong
- A retry rule sends the payout again with a new reference. The provider sees two separate instructions and may execute both.
- The client generates a new idempotency key per attempt. A user clicks "retry" after a timeout, the client builds a new key, and Orangepill correctly creates a second payout. Derive the key from the business record you are paying, not from the request.
- The timeout releases the funds. The wallet shows the money as available, it is spent on something else, and then the first payout completes.
- The payout outlives its reservation. After one hour non-terminal, the reservation is released. Escalate before that point.
-
RETRY_EXHAUSTEDis treated as a decline. Someone resends the payout without confirming with the provider that the first one did not settle. - The payout is sent through another provider after the timeout. The first provider's transfer may still settle. See When payment failover is actually safe.
What Orangepill guarantees today
- One payout per
idempotency_keyin your tenant. A repeated key returns the existing payout. - A timeout, connection reset or 5xx when Orangepill sends a payout to the provider does not mark the payout failed. It is held for operator attention and not sent again automatically.
- After a provider accepts a payout, its final status is taken from querying the provider for that transfer.
pendingandprocessingare reported as non-terminal, not as failure.- A wallet-funded payout records a reservation. On completion it is consumed and a journal is posted; on failure it is released and nothing is posted.
What Orangepill does not guarantee
- Exactly-once execution across external providers. The idempotency key prevents duplicate payouts inside Orangepill only.
- Payout failover to another provider. It is in development.
- Holding funds indefinitely. Payout reservations currently expire after one hour, whatever the payout state.
- That a payout failed with
RETRY_EXHAUSTEDdid not move money. - Payout status webhooks. Payout status is read by polling.
- Automated handling of ambiguous payouts. Resolve is in development; today a person decides.
Production considerations
- Derive idempotency keys from the record being paid, store them before sending, and reuse them on every retry.
- Map timeouts, resets and 5xx responses to "unknown" in your own code. Never to
failed. - Alert on payouts that stay non-terminal longer than the rail normally takes, and well before the one-hour reservation expiry.
- Write the runbook before launch: who contacts the provider, which evidence is enough to call a payout failed, and who approves a new attempt.
- Keep provider references, statements and settlement files, and reconcile them against the ledger on a fixed schedule.
- Show "processing" to the end user while the outcome is unknown, and remove any retry button for that payout.
- Test the path in sandbox: a client-side timeout followed by a resend with the same key must return the same payout.
The implementation steps are in Handle a payout timeout safely. For how the runtime treats failed and unknown outcomes more generally, see Execution and Financial integrity.
Walk through your flow with us.
Bring the failure case that worries you most. We will show how the runtime handles it today.