The Silence After the Write



The charge succeeded. The customer never heard about it. Ten seconds later, they pressed Pay again.

Somewhere between those two events lies one of the hardest problems in distributed computing.


The Transit Gate Dilemma

  1. A phone sends a request to charge $5.
  2. The server validates, commits the ledger write, and moves the money.
  3. The train enters a tunnel—connectivity drops instantly.
  4. The 200 OK never reaches the client.
Server View
Transaction Committed (Succeeded)
Client View
Nothing Happened (Timeout)
SYSTEM ENTERED STATE: UNKNOWN

Retry risks a double-charge. Refusing the retry risks failing a transaction that never executed.

“Networks fail. The problem is not the broken pipe—it is that our architecture assumed the response would always arrive to tell us the truth.”

This Is How Architectural Dementia Begins

The failure at the transit gate is not unusual. It is one symptom of a larger architectural problem.

Modern systems execute business logic across networks that are inherently uncertain. Packets disappear. Connections time out. Services restart. Messages arrive twice. A caller can lose the response even though the operation itself succeeded.

Yet much of our software is still written as though every operation has only two outcomes:

  • SUCCESS
  • FAILURE

Distributed systems introduce a third: UNKNOWN.

When an architecture cannot represent that uncertainty, it pushes the problem somewhere else—into retries, duplicate side effects, manual reconciliation, fragmented logs, and eventually the mind of the engineer on call.

This is one of the conditions described as Architectural Dementia: a system progressively losing the ability to explain what it has done, why it did it, and whether an intent has already changed state.

The Retry Paradox

CLIENT                SERVER               LEDGER
  │                     │                    │
  │─── POST /charge ───>│                    │
  │                     │─── write $5 ──────>│
  │                     │                    │──[COMMIT ✓]
  │                     │<── success ────────│
  │      X [tunnel]     │                    │
  │ (connection drops)  │                    │
  ▼                     │                    │
TIMEOUT                 │                    │
  

The natural reaction is to retry. But a retry does not tell the server whether the customer means:

Case A: “Please complete the thing I already asked you to do.”
Case B: “Please execute a brand-new $5 charge.”

Those are two fundamentally different human intentions that produce an identical payload.

Give Every Intent an Identity

Introduce a unique execution identity: INTENT_ID = 7F3A...

  1. Mark intent IN_FLIGHT
  2. Execute charge
  3. Ledger COMMIT succeeds
  4. Process crashes — [silence]
  5. Mark SETTLED — (never reached)

Idempotency tells us that we have seen the intention before.

Reconciliation tells us what actually happened to it.

Comments

Popular posts from this blog

Show HN: mcp-gate – Ephemeral capability token proxy for LLM tool execution in Go

The Secret Handshake (Identification vs. Authentication)

Cross-border payments