AI Agents Are Easy to Demo. Making Them Safe to Run a Business Is the Hard Part.

-



AI agents can now be built surprisingly quickly.

Connect an LLM to a few APIs, expose some tools through MCP, add a workflow, and within hours an agent can create tickets, modify cloud resources, update databases, send messages, provision infrastructure, or trigger business processes.

That is exciting.

But once agents start performing real-world actions, a much harder engineering problem appears:

What happens when an agent completes steps 1 and 2, but step 3 fails?

What if the network times out and we do not know whether an external API actually completed the operation?

What if two instances of an agent attempt to recover the same workflow?

What if an AI agent is authorized to inspect an environment but should not be allowed to modify production?

What if a low-risk operation can run automatically, while a high-risk operation needs human approval?

At that point, the problem is no longer primarily about prompting or LLMs.

It becomes a problem involving:

  • distributed systems

  • transaction management

  • idempotency

  • security

  • identity

  • authorization

  • policy

  • observability

  • failure recovery

  • human approval

  • auditability

These are the problems I wanted to explore when building Semantic Saga.

Introducing Semantic Saga

Semantic Saga is an open-source transactional control plane for AI-agent side effects.

Instead of allowing an agent to directly perform a sequence of independent operations, Semantic Saga coordinates those operations as a durable Saga.

The basic architecture is:

AI Agent / Peer Agent
        |
     MCP / A2A
        |
        v
Identity + RBAC
        |
        v
Policy & Governance
        |
        v
Semantic Saga Engine
        |
  -------------------
  |        |        |
Action A Action B Action C
  |        |        |
External business systems

Every concrete side effect is journaled before execution.

Each side effect is represented by a versioned action contract that defines:

  • what the operation does

  • its input and output schemas

  • semantic risk

  • retry behavior

  • approval requirements

  • how the operation can be compensated

If a later step fails, Semantic Saga can unwind previously completed work using the correct historical compensation contract.

Why Saga semantics matter for AI agents

Consider an employee onboarding agent.

It needs to:

  1. Create an employee identity.

  2. Create a Git repository.

  3. Provision a cloud environment.

  4. Register a service.

  5. Send onboarding notifications.

A typical agent implementation might simply execute five API calls.

But imagine step 4 fails.

Should the employee account remain?

Should the cloud infrastructure remain?

Should the repository remain?

What if step 3 timed out, but the cloud provider actually created the environment?

Simply asking the LLM to "try again" does not solve this problem.

Semantic Saga maintains durable knowledge of what was attempted, what completed, what may have completed, and how each operation should be reversed.

The important idea: AI needs a transaction control plane

Traditional databases provide ACID transactions inside a database.

AI agents increasingly operate across systems that do not share a database transaction:

GitHub
Cloud providers
Payment systems
CRM
Ticketing systems
Databases
Messaging systems
Internal APIs
External SaaS platforms
Other AI agents

There is no global database transaction across those systems.

The Saga pattern provides another model.

Each forward operation has a corresponding compensation operation.

Conceptually:

Forward workflow

Create account
      |
Create repository
      |
Provision infrastructure
      |
Register service
      |
      X failure


Compensation

Delete infrastructure
      |
Delete repository
      |
Disable/remove account

The important part is that compensation is not merely generated by an LLM.

It is defined as part of a versioned engineering contract.

What Semantic Saga currently provides

The project has evolved through multiple implementation phases.

MCP support

Agent hosts can invoke Semantic Saga through the Model Context Protocol.

MCP is the tool interface.

Semantic Saga provides tools for operations such as:

begin_saga
plan_saga_step
run_ready_steps
approve_saga_step
retry_saga_step
commit_saga
rollback_saga
get_saga
get_saga_timeline
verify_audit_chain

A2A agent interoperability

Semantic Saga can also operate as an A2A Protocol peer.

Other agents can discover it and delegate transactional work using structured commands.

This allows a workflow started by one authorized agent to be continued by another agent in the same organization.

Importantly:

MCP and A2A are just ingress protocols.

They both enter the same governed Saga engine.

Enterprise identity

Remote callers can be authenticated using OAuth/OIDC JWTs.

Identity includes:

Tenant
Principal
Principal type
Roles
Scopes

Tenant ownership becomes the organizational isolation boundary.

A workflow belonging to one tenant cannot be inspected by another tenant even if its saga ID becomes known.

RBAC

The project separates basic authorization from business governance.

Roles currently include:

viewer
operator
admin

For example:

Viewer
  -> inspect workflows and audit

Operator
  -> create and execute workflows

Admin
  -> full operational privileges

Immutable action contracts

This is one of the most important design decisions.

Every side-effecting action has a version.

For example:

create_service@1.0.0
create_service@2.0.0

When an operation begins, Semantic Saga persists:

action
action_version
definition_hash
definition_snapshot

Suppose an old Saga used version 1.

Weeks later, the organization deploys version 2 with different rollback logic.

Recovery of the old Saga still uses the historical version 1 contract.

It never silently compensates an old transaction using new semantics.

Durable workflow DAGs

Workflows can contain dependencies.

For example:

             Create account
                   |
        -------------------------
        |                       |
Create repository       Create cloud project
        |                       |
        -----------+-------------
                   |
              Deploy service
                   |
              Send message

Independent actions can run in parallel.

Dependent actions become ready only after their prerequisites complete.

Retry and idempotency

Transient failures need retries.

But retrying side effects carelessly can create duplicates.

Semantic Saga therefore maintains a stable persisted step identity across retries.

The same operation keeps the same idempotency identity.

External APIs still need to honor idempotency, but the coordinator provides a deterministic identity for them to use.

Human approval

Some actions should not be fully autonomous.

An action can require approval because:

  • its contract requires approval

  • tenant governance requires approval

  • its semantic risk crosses a threshold

A workflow can enter:

WAITING_APPROVAL

An authorized person can approve or reject it.

Recovery Required

This is another important safety principle.

Sometimes an external outcome is genuinely uncertain.

For example:

Agent -> Payment API

request sent
     |
network timeout
     |
???

The payment might have succeeded.

It might not have.

A dangerous system pretends it knows.

Semantic Saga can instead place the workflow in:

RECOVERY_REQUIRED

An operator reconciles the external system before forcing another attempt or rolling the transaction back.

Distributed execution safety

A durable database by itself does not make a coordinator horizontally scalable.

Multiple workers could otherwise process the same workflow.

Semantic Saga uses:

renewable leases
+
monotonically increasing fencing tokens
+
atomic sequencing
+
PostgreSQL SKIP LOCKED recovery

If worker A loses its lease and worker B takes ownership, worker A's old fencing token can no longer mutate the journal.

This prevents stale workers from corrupting the workflow after ownership changes.

Policy and governance

Authentication answers:

Who are you?

RBAC answers:

Are you allowed to call this capability?

Governance answers:

Should this specific operation be allowed right now?

Policies can evaluate:

tenant
principal
roles
action
resource
operation
risk
workflow phase
approval state
workflow budget
parallelism

A policy decision can return:

ALLOW

DENY

REQUIRE APPROVAL

The project supports both a deterministic built-in JSON policy engine and integration with Open Policy Agent.

Hard budgets

Agent workflows can also be constrained using budgets.

Examples:

Maximum steps per Saga
Maximum planned nodes
Maximum accumulated risk
Maximum parallel actions

This provides another control against runaway agent behavior.

OpenTelemetry

Distributed traces can follow an operation through:

Agent
  |
MCP or A2A
  |
Semantic Saga
  |
Governance
  |
Action
  |
External service

W3C trace context is propagated downstream.

Durable, tamper-evident audit

Every important control-plane decision generates audit evidence.

Examples include:

Saga creation
Action execution
Policy decisions
Human approvals
Retries
Recovery actions
Compensation
Commit
Rollback

Events are chained using SHA-256:

Event 1 hash
     |
     v
Event 2 previous_hash
Event 2 hash
     |
     v
Event 3 previous_hash

This makes accidental or unauthorized modification detectable.

The audit record deliberately avoids storing arbitrary business payloads and secret material.

What this project demonstrates beyond AI APIs

One reason I am publishing this project is that I think the skill profile required for production AI systems is changing.

LLMs are making certain parts of application development dramatically easier.

But the systems surrounding those models still require deep engineering.

Production agent infrastructure combines disciplines such as:

AI engineering
Distributed systems
Backend architecture
Cloud engineering
SRE
Security
OAuth/OIDC
Policy engines
Transactions
Concurrency
Observability
API design
Failure recovery
MCP
A2A

This project is also my way of demonstrating those capabilities through working software rather than only listing technologies on a résumé.

For engineering leaders and recruiters

If your organization is building:

  • AI infrastructure

  • agent platforms

  • autonomous workflow systems

  • MCP infrastructure

  • A2A systems

  • developer platforms

  • cloud control planes

  • distributed systems

  • payments or transaction infrastructure

  • reliability platforms

  • AI governance

  • AI safety infrastructure

I would be interested in connecting.

Roles where this type of work may be particularly relevant include:

Staff Engineer
Principal Engineer
AI Infrastructure Engineer
Agent Platform Engineer
Platform Architect
Distributed Systems Engineer
SRE / Reliability Engineer
AI Governance Engineer
Cloud Platform Engineer

Try Semantic Saga yourself

The project is open source:

GitHub

https://github.com/ananthaprakashb/semantic-saga-mcp

Python 3.10+ is supported.

Try the MCP server

pip install semantic-saga-mcp

semantic-saga-mcp

For durable local experimentation:

semantic-saga-mcp \
  --database ./semantic-saga.db \
  --actions ./examples/action_registry.json \
  --policy-mode json \
  --policy-file ./examples/governance_policy.json

Try A2A interoperability

pip install "semantic-saga-mcp[a2a]"

Then:

semantic-saga-a2a \
  --host 127.0.0.1 \
  --port 8100 \
  --a2a-public-url http://127.0.0.1:8100 \
  --database ./semantic-saga.db \
  --actions ./examples/action_registry.json

Agent discovery is available at:

http://127.0.0.1:8100/.well-known/agent-card.json

The repository also contains:

Action registry examples
Governance policy examples
Architecture documentation
Workflow documentation
Identity design
Distributed durability design
Audit and observability design
A2A documentation
Unit and integration tests
PostgreSQL concurrency tests

What comes next

There are still several interesting areas to explore.

Among them:

  • outbound A2A actions with durable remote task contracts

  • distributed event streaming and push notifications

  • operator/recovery console

  • stronger external audit anchoring

  • SDKs for application developers

  • finer-grained distributed scheduling

  • broader enterprise integrations

But the underlying principle will remain the same:

AI agents should not receive weaker transactional, security and operational guarantees simply because an LLM is making the decision.

As agents gain the ability to take real-world actions, we will increasingly need infrastructure that can answer:

Who initiated this operation?

Was it authorized?

Which policy allowed it?

What exactly changed?

Which action contract was used?

Can the change be reversed?

What happens after a partial failure?

What happens when the process crashes?

Can another agent safely continue?

Can an operator reconstruct what happened?

Those questions are where AI engineering meets serious production engineering.

That is the space I want to continue exploring and building in.

Comments

Post a Comment

Popular posts from this blog

Show HN: mcp-gate – Ephemeral capability token proxy for LLM tool execution in Go

-
Image
I built mcp-gate , a small Go reverse proxy for a problem I keep coming back to with LLM agents: How much authority should we actually give a model when it calls a tool? Repo: https://github.com/ananthaprakashb/mcp-gate A typical agent integration eventually ends up holding something powerful: an API key, service credential, OAuth token, or access to an MCP/tool server that can perform multiple operations. Even when the model is supposed to perform one very specific action, the credential it indirectly controls may authorize far more. I wanted the authorization boundary to look more like this: The model never receives the upstream API credential. Instead, the trusted orchestrator exchanges its gate credential for an ephemeral capability token authorizing one specific operation. For example: {   "route": "tickets",   "method": "POST",   "path": "/v1/tickets",   "ttl_seconds": 15 } The returned bearer token is HMAC-s...
Read more

Vital by ShareCapsule: Building a Local-First Daily Wellness System

-
Image
Most wellness apps begin with a category: meditation, exercise, sleep, journaling, or habit tracking. We started from a different question: What are the small actions that help someone support their day as a whole? That question gradually became Vital by ShareCapsule , a mobile-first, installable wellness PWA built around breathing, movement, learning, recovery, connection, and reflection. The goal is not to build another app that asks people to achieve perfect scores or maintain endless streaks. The goal is much simpler: Make small positive actions easy to start, repeat, and understand. Vital is currently built as a local-first Progressive Web App. It does not require an account for the core experience, and the current product stores activity history, routines, settings, wellness check-ins, gratitude metadata, learning reflections, and sleep check-ins in the browser rather than sending them to a central cloud database. Why we built it this way Modern wellness products often become com...
Read more

Recursion

-
Image
 Recursion, tail optimization and memoization are important piece of Functional programming. Before jumping into solving complex solutions, I slip on basic structure of recursion lot of times. What is base case and what is recursive step? When I ask most recursive person at home, got the answer as "until I need a charge". Seriously??? I don't know what algorithm it is!!???. It moves all around without any pattern until completely got drained. ☺ May be it is the algorithm... So finding the base case says when to stop.   Another programming example: We are given a denomination of coins/ currency notes and amount of value. We need to identify the minimum number of coins/ currency notes required for calculating that amount. For the amount $3.15, we need to pick up 12 quarters, 1 dime and 1 nickel. So total number of coins is 14 coins. The base case here -> if total of value is equal to the amount required. The recursive step -> pick a higher denomination available to re...
Read more