The Most Dangerous Word in Agentic AI Is “Timeout”
Transaction Boundary Failures — and the infrastructure we need before agents are trusted with real-world actions
A customer asks an AI agent to upgrade an account.
The workflow looks harmless.
First, charge the card.
Then update the subscription ledger.
Then provision the new entitlement.
The agent calls the payment API.
The payment provider charges $500.
And then the connection times out.
What happened?
That sounds like a simple question. It isn't.
The agent knows that it did not receive a successful response.
It does not know that the charge failed.
Those are two very different things.
The payment may have failed before reaching the provider.
The payment may still be processing.
Or the payment may have completed perfectly and only the response was lost somewhere between the provider and the agent.
Now imagine what happens when we put an LLM in the middle of this uncertainty.
“The payment request timed out. I still need to complete the upgrade. Let me try again.”
That sentence sounds reasonable.
It could also charge the customer twice.
This is the problem I want to explore in the first article of this series: Transaction Boundary Failures in Agentic Systems.
And I suspect it will become one of the less glamorous—but much more important—problems we have to solve before autonomous agents are allowed to run serious businesses.
A timeout is not a failure
Most application code trains us to think in two states:
SUCCESS
FAILURE
Distributed systems have never been that simple.
Once a workflow crosses a network boundary, there is another state that matters enormously:
UNKNOWN
Suppose the charge succeeds, but the ledger write that follows it fails.
Reality now looks like this:
Payment provider $500 charged
Internal ledger No payment recorded
Subscription Not upgraded
Agent Unsure what happened
Money moved.
The application disagrees.
No database rollback can magically rewind the payment processor because the payment provider and our ledger do not share one ACID transaction. That partial-success condition is the heart of the transaction-boundary problem.
This isn't new.
Distributed-systems engineers have spent decades dealing with retries, idempotency, sagas, reconciliation, compensating transactions and durable execution.
What is new is the component we are inserting between failure and recovery:
a probabilistic reasoning engine whose primary objective is usually to finish the task.
That changes the risk.
We may be giving the wrong responsibility to the LLM
Much of today's agent architecture follows a pattern roughly like this:
Understand goal
↓
Choose tool
↓
Execute tool
↓
Observe result
↓
Reason about what to do next
That works beautifully when the tool is searching documentation.
It becomes more dangerous when the tool is:
charge_customer()
transfer_money()
place_trade()
delete_database()
provision_cluster()
purchase_inventory()
send_contract()
The LLM should be allowed to reason about what we intend to accomplish.
I don't think it should be allowed to independently decide whether an uncertain side effect should be executed again.
That distinction may become fundamental to production agent architecture.
LLM
│
│ proposes intent
▼
Transaction Safety Runtime
│
├── validates policy
├── assigns operation identity
├── records durable state
├── executes side effect
├── verifies result
├── reconciles uncertainty
└── compensates or escalates
│
▼
External systems
In other words:
Let the model reason. Let deterministic infrastructure commit.
Interestingly, recent agent-systems research is beginning to move in a similar direction. A 2026 preprint on Agentic Transaction Processing describes generated actions as proposals that should remain untrusted until a deterministic runtime admits and commits them. Another recent paper proposes reinterpreting ACID properties for agent execution itself. These are early research directions rather than settled industry standards, but the direction is worth watching.
The state machine needs one more state
Consider the payment again.
A safer runtime might model it as:
PREPARED
↓
DISPATCHED
↓
┌─────────────┐
↓ ↓
CONFIRMED UNKNOWN
↓ ↓
COMMITTING RECONCILING
↓ ↓
COMMITTED ┌───────────────┐
↓ ↓
CONFIRMED NOT_EXECUTED
↓ ↓
COMMITTING RETRY
And sometimes:
UNKNOWN
↓
RECONCILING
↓
MANUAL_REVIEW
That final path matters.
There will be situations in which the system cannot prove whether an irreversible operation happened.
The right behavior is not always to be clever.
Sometimes the safest automated decision is:
stop. preserve evidence. escalate.
Our working notes framed this nicely: unknown operations should carry an expected_resolution_by time, be reconciled against the external provider, retain unmatched callbacks or webhooks, and eventually escalate based on age, rather than merely counting failures.
That is very different from ordinary retry logic.
Idempotency helps—but agents expose its limits
Payment engineers already know one important defense: idempotency.
Stripe, for example, documents idempotency keys specifically so that a request interrupted by a connection error can be retried without creating the same object or performing the same update twice. It also compares parameters when a key is reused to prevent accidental misuse.
That sounds like the answer.
It is part of the answer.
But consider how an agent might retry.
First attempt:
{
"vendor": "ABC",
"amount": 10000,
"currency": "USD"
}
After seeing a timeout, the model reasons again and constructs:
{
"vendor_id": "vendor-481",
"amount": 10000,
"currency": "USD",
"memo": "September invoice"
}
Semantically, both requests might mean:
Pay ABC's September invoice once.
Technically, they are not necessarily the same request.
If each new reasoning pass generates a fresh operation identity, ordinary idempotency protection can be defeated by the agent itself.
That suggests a layer above API idempotency:
Semantic operation identity
Before the first side effect occurs, the runtime creates something like:
intent_id = PAY-INVOICE-93821
operation_id = OP-7F31...
idempotency_key = OP-7F31...
The model doesn't get to silently create a new identity because it reformulated the request.
If it wants to change the business intention, that should become a new explicitly validated transaction, not an accidental retry.
This is one area I think deserves considerably more research.
Durable execution doesn't eliminate the problem either
A durable workflow engine gets us much closer to safety because the workflow state is not dependent on an LLM remembering what happened.
But even here, the physics of distributed systems remains.
Temporal's documentation describes a particularly revealing failure:
an activity can complete successfully, the worker can crash before reporting completion, and the activity can subsequently be executed again.
Temporal therefore strongly recommends making write activities idempotent. It explicitly uses duplicate payment charges as an example of what can happen otherwise.
That gives us an important distinction:
Exactly-once workflow observation
≠
Exactly-once physical execution
The external world does not participate in our workflow engine's transaction log.
Which means durable orchestration, idempotency and reconciliation have to work together.
And then comes compensation
Suppose we know with certainty that the $500 charge succeeded.
But subscription provisioning failed permanently.
What now?
One option is compensation:
Charge → Refund
Reserve inventory → Release inventory
Create booking → Cancel booking
Ledger debit → Reversal entry
This is essentially the territory of saga patterns: a business transaction is decomposed into local transactions, with continuation or compensating actions used when later steps fail. AWS's saga guidance describes exactly this model.
But we should be careful with the word “rollback.”
A refund is not a rollback.
The customer may see both transactions.
Fees may already have been incurred.
Foreign-exchange rates may have moved.
A trade may have changed market exposure.
An email cannot really be unsent.
A deployed configuration may already have affected users.
So agents need more than:
do()
undo()
They need compensation semantics.
For every consequential action, we should know before execution:
Can it be retried?
Can it be verified?
Can it be compensated?
Is compensation guaranteed?
How long can uncertainty remain?
What evidence proves completion?
When must a human take over?
That looks less like prompt engineering and more like financial infrastructure.
Which is probably the point.
Accounting solved part of this problem long before AI existed
There is another idea worth borrowing—from double-entry accounting.
When accountants don't yet know the final destination of money, they don't simply pretend the transaction doesn't exist.
They can use an intermediate or suspense account.
Conceptually:
Customer
↓
Payments in transit
↓
Provider clearing
The unresolved transaction remains visible.
It doesn't disappear because the final step hasn't completed.
Our earlier research notes proposed exactly this for financial agent flows: represent unresolved money explicitly in a “payments in transit” position so the balance itself tells us the monetary exposure of transactions whose outcome has not yet been resolved.
That is a surprisingly powerful principle for agent systems:
Uncertainty should become state, not disappear into logs.
Perhaps we should apply the same principle beyond money.
deployments_pending_verification
orders_awaiting_confirmation
emails_delivery_unknown
cloud_resources_reconciling
contracts_signature_pending
If uncertainty has business consequences, make it first-class business state.
What should an agent actually be allowed to decide?
This leads to a broader architecture question.
We frequently hear that sensitive agent operations should have a “human in the loop.”
That sounds safe.
At scale, it can become another failure mode.
If a person must approve every $20 action, every infrastructure update, every reservation and every API retry, humans eventually become expensive OK buttons.
Then comes alert fatigue.
Then rubber-stamping.
Then the control exists mostly on paper.
A better model may be human over the loop.
For example:
Agent
decides WHAT it wants to accomplish
Runtime
decides WHETHER execution is permitted
and HOW transactional state moves
Human
defines boundaries
reviews exceptions
resolves ambiguity
handles conditions beyond policy
The drafts we started from make this distinction explicitly: the LLM and human can help choose the workflow, while deterministic infrastructure controls financial and security state transitions, escalating only when predefined responsibility boundaries are exceeded.
That architecture scales better than asking a human to babysit every tool call.
The agent should receive receipts, not memories
There is one more design principle I think we should explore.
LLM context should never become the authoritative transaction history.
Instead, every important action should produce a durable receipt:
operation_id
business_intent
request_hash
provider_reference
requested_at
confirmed_at
observed_status
verification_method
compensation_policy
evidence
current_state
When an agent resumes after a crash, model change, context truncation or handoff to another agent, it should not reconstruct reality from conversation history.
It should be given facts:
Payment OP-721:
Provider confirms: SETTLED
Ledger status: MISSING
Next permitted action: RECONCILE_LEDGER
Retry payment: PROHIBITED
Now the LLM can reason freely again.
But it reasons inside known transaction boundaries.
That is a much safer form of autonomy.
The test I want every serious agent platform to pass
Here's the experiment.
Take one ordinary agent workflow.
Then kill it at every possible boundary.
Before the API request leaves.
After it leaves but before the provider receives it.
After the provider commits but before the response returns.
After the response returns but before local persistence.
During the ledger update.
During compensation.
During reconciliation.
During agent handoff.
During workflow recovery.
Run every case repeatedly.
Then ask only one question:
Can this system prove that the real world is still in an allowed state?
Not:
“Did the agent finish?”
Not:
“Did the model give the right answer?”
Not even:
“Did our workflow complete?”
The real test for consequential agents is whether business invariants survive failure.
That could become an interesting benchmark of its own.
We may need an Agent Transaction Envelope
The more I think about this problem, the less comfortable I am with allowing consequential MCP or tool calls to be just:
{
"tool": "charge_customer",
"amount": 500
}
Perhaps every consequential invocation eventually needs something closer to:
Agent Transaction Envelope
Intent
Operation identity
Authority
Preconditions
Idempotency policy
Expected postconditions
Verification strategy
Compensation strategy
Resolution deadline
Audit evidence
Escalation policy
The LLM would propose the action.
The envelope would define the rules under which reality is allowed to change.
The transaction runtime would enforce them.
And only then would the tool execute.
Whether that eventually becomes a library, an MCP extension, an orchestration pattern or something closer to a protocol is an open question.
But I believe the problem is going to become unavoidable.
Better models make this problem more important, not less
It's tempting to assume that smarter agents will eventually reason their way out of these situations.
I think that's the wrong direction.
The more capable the model becomes, the more authority we will give it.
The more authority it has, the larger the blast radius when something goes wrong.
A model that can only summarize an invoice can make a bad summary.
A model that can pay the invoice can move money.
A model that can coordinate procurement, inventory, financing and logistics can create inconsistencies across an entire organization.
The answer isn't to make agents timid.
It is to build infrastructure strong enough that agents can safely become powerful.
So before asking:
How autonomous is our agent?
I think engineering teams should start asking:
What happens when the agent changes the real world, loses confirmation halfway through, and wakes up believing it still has work to do?
If we cannot answer that precisely, the agent probably isn't ready for production.
Not yet.
Transaction Boundary Failures is the first problem I want to investigate in this series on engineering for the next generation of agentic systems.
Next, I want to go deeper into the Agentic Transaction Boundary Model: operation identity, state machines, reconciliation, compensation, concurrency between agents, responsibility contracts, and a fault-injection benchmark that can tell us whether an agentic workflow is actually safe—not merely impressive in a demo.
Because eventually, “the agent tried its best” is not going to be an acceptable incident report.
Comments
Post a Comment