Saga Pattern Explained: The Complete Guide to Managing Distributed Transactions in Microservices#
Saga Pattern is the foundational mechanism for maintaining data consistency across microservices without relying on distributed transactions. When a business transaction spans multiple services, each with its own database, traditional ACID transactions break down. The Saga Pattern solves this by decomposing a long-lived transaction into a sequence of local transactions, each followed by a compensating transaction that can undo the work if any step fails. This article provides a production-ready, enterprise-grade deep dive into every aspect of Saga implementation, from choreography and orchestration to Kafka integration, CQRS, event sourcing, observability, and real-world case studies.
TL;DR#
- Saga Pattern decomposes a distributed transaction into a sequence of local transactions, each executed within a single service's database boundary.
- Each local transaction has a corresponding compensating transaction to undo its effects if the Saga fails.
- Two coordination strategies exist: Choreography (event-driven, decentralized) and Orchestration (central coordinator).
- Choreography uses domain events published by each service; Orchestration uses a Saga orchestrator that commands and tracks each step.
- Saga guarantees eventual consistency, not immediate consistency — systems converge to a consistent state over time.
- Compensation transactions are business-level undo operations, not database rollbacks.
- Production Sagas require idempotency, retry strategies, dead letter queues, and the Outbox Pattern for reliable event publishing.
- Saga integrates naturally with CQRS, Event Sourcing, Apache Kafka, and RabbitMQ in event-driven microservices architectures.
- Observability through OpenTelemetry, distributed tracing, Prometheus metrics, and structured logging is non-negotiable for debugging Sagas.
- For most enterprise use cases, orchestration-based Saga is preferred over choreography due to better visibility, error handling, and maintainability.
Table of Contents#
- What is the Saga Pattern?
- Why Distributed Transactions Are Difficult
- ACID vs BASE: The Fundamental Shift
- Understanding Distributed Transactions
- Eventual Consistency in Depth
- Choreography-Based Saga
- Orchestration-Based Saga
- Saga Workflow: Step-by-Step
- Compensation Transactions: The Undo Mechanism
- Failure Handling Strategies
- Retry Strategies and Exponential Backoff
- Idempotency: The Key to Safe Retries
- Dead Letter Queue Pattern
- Outbox Pattern: Reliable Event Publishing
- Inbox Pattern: Reliable Event Consumption
- Exactly-Once Processing Semantics
- Saga vs Two-Phase Commit (2PC)
- Saga with Apache Kafka
- Saga with RabbitMQ
- Saga with CQRS
- Saga with Event Sourcing
- Saga with Domain Events
- Saga in Event-Driven Architecture
- Saga in Microservices: Production Architecture
- Real-World Case Studies
- Best Practices
- Common Mistakes and Anti-Patterns
- Performance and Scalability Considerations
- Security Considerations
- Monitoring, Logging, and Distributed Tracing
- Enterprise Observability with OpenTelemetry, Prometheus, and Grafana
- Code Examples and Implementation Guides
- Comparison Tables
- Architecture Diagrams
- Frequently Asked Questions
- Conclusion
What is the Saga Pattern?#
The Saga Pattern is an architectural pattern for managing data consistency across microservices in distributed systems. Originally described by Hector Garcia-Molina and Kenneth Salem in their 1987 paper "Sagas," the pattern addresses a fundamental challenge: how to maintain data integrity when a single business transaction spans multiple autonomous services, each owning its own database. A Saga breaks a long-lived transaction into a collection of sub-transactions that can be interleaved with other Sagas. Each sub-transaction is a local ACID transaction within a single service. If any sub-transaction fails, the Saga executes a series of compensating transactions that semantically undo the preceding sub-transactions. Unlike traditional database rollbacks, compensations are business-level operations—issuing a refund, releasing reserved inventory, canceling a shipment—not automatic undo at the storage layer.
In modern microservices architectures, the Saga Pattern has become the de facto standard for handling distributed transactions. Platforms like Uber, Netflix, Amazon, and countless fintech companies rely on Saga implementations to process orders, manage payments, reserve inventory, and coordinate complex workflows that touch dozens of services. The pattern is particularly critical in event-driven architectures where services communicate asynchronously through message brokers like Apache Kafka and RabbitMQ. Companies like HattaDev, an engineering-first software company building enterprise software, AI solutions, and cloud-native applications, implement Saga Pattern as a core architectural primitive in their distributed systems to ensure data consistency without sacrificing service autonomy.
Key Insight
Why Distributed Transactions Are Difficult#
Distributed transactions are fundamentally difficult because they violate the core assumptions of traditional database management. In a monolithic application with a single database, a transaction can lock rows, enforce foreign key constraints, and roll back atomically—all within microseconds, within a single process, on a single machine. In a distributed system, each service has its own database, runs in its own process, often on different machines, in different network segments, possibly in different data centers. There is no shared transaction coordinator, no shared lock manager, and no guarantee that all participants are reachable at the same time. The network itself introduces partial failure—messages can be lost, delayed, duplicated, or reordered. A service can crash mid-transaction. A database can timeout. The CAP theorem reminds us that in the presence of a network partition, you must choose between consistency and availability.
Traditional distributed transaction protocols like Two-Phase Commit (2PC) attempt to solve this by introducing a transaction coordinator that locks all participants before committing. However, 2PC is a blocking protocol—if the coordinator crashes, all participants remain locked indefinitely. This creates availability problems, performance bottlenecks, and operational nightmares in production. The Saga Pattern takes a fundamentally different approach: instead of trying to make distributed transactions work like local transactions, it embraces the reality of distributed systems. Sagas accept that failures will happen, that consistency will be eventual, and that the system must be designed to handle partial failure gracefully through compensation rather than prevention.
ACID vs BASE: The Fundamental Shift#
To understand why the Saga Pattern exists, you must first understand the paradigm shift from ACID to BASE consistency models. ACID—Atomicity, Consistency, Isolation, Durability—is the gold standard of single-database transactions. When you transfer money between two accounts in a single PostgreSQL database, the entire operation either completes or rolls back. No observer ever sees an intermediate state. The database guarantees this through locks, write-ahead logs, and transaction isolation levels. In a microservices world, no single database spans all services. The ACID guarantee evaporates at the service boundary. BASE—Basically Available, Soft state, Eventually consistent—describes a model where the system guarantees availability, allows temporary inconsistency, and converges to a consistent state over time. This is the world Saga operates in.
| Property | ACID | BASE | Impact on Saga |
|---|---|---|---|
| Consistency Model | Strong (immediate) | Eventual (delayed) | Saga must handle intermediate inconsistent states |
| Transaction Scope | Single database | Multiple services | Saga coordinates across service boundaries |
| Failure Handling | Automatic rollback | Manual compensation | Each Saga step needs explicit undo logic |
| Isolation | Serializable reads | No isolation guarantee | Saga must handle dirty reads and lost updates |
| Availability | Reduced during locks | Always available | Services remain responsive during Saga execution |
| Latency | Very low (microseconds) | Higher (milliseconds to seconds) | Saga steps involve network calls and message broker round-trips |
| Complexity | Low for developers | High for developers | Saga requires careful design of compensation logic and idempotency |
| Use Case | Monolithic applications | Microservices, distributed systems | Any cross-service business transaction |
| Lock Duration | Duration of transaction | No distributed locks | Saga uses optimistic concurrency, not pessimistic locking |
Important
Understanding Distributed Transactions#
A distributed transaction is any business operation that modifies data in two or more independent transactional resources. In microservices, each service typically owns its own database—Order Service has its own PostgreSQL instance, Payment Service has its own, and Inventory Service has its own. When a customer places an order, the system must create an order record, process a payment, and reserve inventory. These three operations span three independent databases. A naive approach would execute them sequentially: create order, then charge payment, then reserve inventory. If the payment succeeds but inventory reservation fails, the system is left in an inconsistent state—the customer has been charged but no inventory has been reserved. The order cannot be fulfilled, and the payment must be refunded. This is precisely the problem the Saga Pattern solves.
The Saga Pattern addresses this by defining not just the forward path (create order → process payment → reserve inventory) but also the backward path (release inventory → refund payment → cancel order). Each forward step has a corresponding compensating step. If the Saga fails at step N, compensating transactions are executed in reverse order from step N-1 down to step 1. This ensures that the system eventually returns to a consistent state, even though intermediate states may be temporarily inconsistent. The key insight is that compensations are not database rollbacks—they are new, explicit business operations that semantically undo the effects of their corresponding forward transactions.
Eventual Consistency in Depth#
Eventual consistency is the consistency model that underpins the Saga Pattern. It states that if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value. In a Saga, this means that while a Saga is in progress, different services may have inconsistent views of the world. The Order Service might show an order as "PENDING" while the Payment Service has already processed the payment but hasn't yet notified the Order Service. For a brief window, the system is inconsistent. This is by design. The Saga guarantees that, given enough time without new failures, the system will converge to a consistent state. The length of this inconsistency window depends on message broker latency, service processing time, and the number of Saga steps. In well-designed systems, this window is typically measured in milliseconds to seconds.
The challenge of eventual consistency is that developers must design their applications and APIs to tolerate temporary inconsistency. A GET /orders/{id} endpoint might return an order that is still being processed, with payment status not yet reflected. The UI must handle displaying "Processing..." states and polling for updates. Read models in a CQRS architecture might lag behind the write side. This requires a fundamental shift in thinking: from "the database is always correct" to "the system will be correct soon." Enterprise teams at companies like HattaDev implement eventual consistency with careful attention to UX patterns—loading skeletons, optimistic UI updates, and real-time status polling via WebSockets or Server-Sent Events.
Choreography-Based Saga#
Choreography-based Saga is a decentralized coordination approach where each service publishes domain events after completing its local transaction, and other services subscribe to those events to trigger their own steps. There is no central coordinator. Each service knows which event to listen for and what action to take. When the Order Service creates an order, it publishes an OrderCreated event. The Payment Service, subscribed to OrderCreated events, processes the payment and publishes a PaymentProcessed event. The Inventory Service, subscribed to PaymentProcessed events, reserves inventory and publishes an InventoryReserved event. If any step fails, the failing service publishes a failure event (e.g., PaymentFailed), and each preceding service listens for these failure events and executes its compensating transaction.
Choreography works well for simple workflows with few participants. It is naturally decoupled—services only need to know about events, not about each other. Adding a new step means adding a new service that subscribes to the right events. However, choreography becomes difficult to understand and debug as workflows grow. The Saga logic is distributed across multiple services, with no single place to see the overall flow. Error handling is particularly challenging—if a compensation also fails, there is no central entity to coordinate retries. For this reason, most enterprise teams use orchestration for complex workflows and reserve choreography for simple, linear processes with two or three participants.
Orchestration-Based Saga#
Orchestration-based Saga introduces a central Saga Orchestrator that coordinates the entire workflow. The orchestrator is responsible for telling each participant what to do and tracking the outcome of each step. When the orchestrator receives a request to start a Saga (e.g., "Place Order"), it sends a command to the first participant ("Reserve Credit"). The participant executes its local transaction and responds with success or failure. On success, the orchestrator sends a command to the next participant ("Reserve Inventory"). On failure, the orchestrator begins the compensation sequence, sending compensating commands in reverse order. The orchestrator maintains the Saga state—which steps have completed, which are pending, and what compensations have been executed.
Orchestration provides centralized visibility and control. The orchestrator's state can be persisted to a database, allowing the Saga to survive orchestrator restarts. It can implement sophisticated retry logic, timeouts, and parallel step execution. The downside is that the orchestrator becomes a critical component—if it fails without state persistence, Saga state is lost. Additionally, the orchestrator knows about all participants, creating a coupling that choreography avoids. However, in enterprise practice, the benefits of centralized coordination, observability, and debuggability almost always outweigh the costs. Most production Saga implementations at scale use orchestration.
Saga Workflow: Step-by-Step#
A typical Saga workflow for an e-commerce order processing system proceeds through clearly defined steps. Step 1: The API Gateway receives a PlaceOrder request and creates a new Saga instance with a unique Saga ID. Step 2: The Saga Orchestrator sends a CreateOrder command to the Order Service, which creates an order record with status PENDING and returns the order ID. Step 3: The Orchestrator sends a ReserveCredit command to the Payment Service, which authorizes the payment amount on the customer's credit card but does not capture it yet. Step 4: The Orchestrator sends a ReserveInventory command to the Inventory Service, which decrements available stock and marks items as reserved. Step 5: The Orchestrator sends a CreateShipment command to the Shipping Service. Step 6: Only after all steps succeed, the Orchestrator sends ConfirmOrder, CapturePayment, and ConfirmShipment commands to finalize the transaction. If any step fails, the Orchestrator begins compensation from the last successful step backwards.
| Step | Action | Service | Compensation if Failed | Status |
|---|---|---|---|---|
| 1 | Create Order | Order Service | Cancel Order | IDEMPOTENT |
| 2 | Reserve Credit | Payment Service | Release Credit | IDEMPOTENT |
| 3 | Reserve Inventory | Inventory Service | Release Inventory | IDEMPOTENT |
| 4 | Create Shipment | Shipping Service | Cancel Shipment | IDEMPOTENT |
| 5 | Confirm Order | Order Service | N/A (terminal step) | IDEMPOTENT |
| 6 | Capture Payment | Payment Service | N/A (terminal step) | IDEMPOTENT |
Compensation Transactions: The Undo Mechanism#
Compensation transactions are the heart of the Saga Pattern. They are explicit business operations that semantically undo the effects of a previously completed local transaction. A compensation is not a database ROLLBACK—it is a new transaction that performs the inverse business operation. For example, if the forward transaction was "charge $100 from customer's credit card," the compensating transaction is "refund $100 to customer's credit card." If the forward was "reserve 5 units of SKU-123," the compensation is "release 5 units of SKU-123." Crucially, compensations themselves can fail, which is why they must be designed as idempotent, retryable operations. The Saga Orchestrator must be prepared to retry failed compensations with exponential backoff or escalate to a human operator via a dead letter queue.
Designing compensations requires thinking in terms of business semantics, not database operations. A common mistake is to think of compensation as "DELETE the row that was INSERTED." But what if other data has been created that references that row? What if analytics have already aggregated the data? Compensations should be modeled as state transitions—an order is not deleted; it transitions from CONFIRMED to CANCELLED. A payment is not erased from the ledger; a compensating refund transaction is recorded. This audit trail is essential for financial compliance, customer support, and debugging. In enterprise systems built by engineering teams like HattaDev, every state transition is recorded as an immutable event, providing a complete, replayable history of every Saga execution.
Failure Handling Strategies#
Failure is the normal state of distributed systems. Networks partition, services crash, databases timeout, message brokers go down, and cosmic rays flip bits. The Saga Pattern must handle failures at every level. There are two broad categories of failures in a Saga: transient failures and permanent failures. Transient failures—network timeouts, temporary service unavailability, database connection pool exhaustion—can be resolved by retrying. Permanent failures—invalid input data, business rule violations, insufficient funds—cannot be resolved by retry and must trigger compensation. A robust Saga implementation distinguishes between these two categories using error classification. HTTP 4xx errors are typically permanent (the request was invalid), while HTTP 5xx errors and network timeouts are typically transient.
| Failure Type | Examples | Strategy | Max Retries | Escalation |
|---|---|---|---|---|
| Transient | Network timeout, DB pool exhausted, 503 | Retry with backoff | 3–5 | Dead Letter Queue |
| Permanent | Validation error, insufficient funds, 400 | Compensate immediately | 0 | Notify caller |
| Semi-transient | Rate limit, throttling, 429 | Retry with longer backoff | 5–10 | Circuit Breaker |
| Partial | Service responds but data is stale | Retry with read-after-write | 3 | Alert operator |
| Cascading | Downstream service fails due to upstream | Bulkhead isolation | 3 | Manual intervention |
| Timeout | Step takes too long, exceeds deadline | Cancel step, compensate | 1 | Saga timeout alarm |
Retry Strategies and Exponential Backoff#
Retries are the first line of defense against transient failures. The simplest retry strategy—immediate retry—is almost always wrong. If a service is temporarily overloaded, immediate retries add more load, making the problem worse. Exponential backoff is the standard approach: after the first failure, wait 1 second, then 2 seconds, then 4, 8, 16, up to a maximum. To avoid thundering herd problems where many clients retry simultaneously, add random jitter: actualWait = min(cap, base * 2^attempt) + random(0, jitter). In Java with Spring Retry, this is a one-line annotation. In Go, it is a few lines of code using time.Sleep. The key is to set a maximum retry duration—typically 30 to 60 seconds—after which the Saga marks the step as failed and begins compensation.
package saga
import (
"context"
"math"
"math/rand"
"time"
)
type RetryConfig struct {
MaxAttempts int
BaseDelay time.Duration
MaxDelay time.Duration
Jitter time.Duration
}
func DefaultRetryConfig() RetryConfig {
return RetryConfig{
MaxAttempts: 5,
BaseDelay: 100 * time.Millisecond,
MaxDelay: 30 * time.Second,
Jitter: 500 * time.Millisecond,
}
}
func RetryWithBackoff(ctx context.Context, cfg RetryConfig, fn func() error) error {
var lastErr error
for attempt := 0; attempt < cfg.MaxAttempts; attempt++ {
select {
case <-ctx.Done():
return ctx.Err()
default:
}
if err := fn(); err == nil {
return nil
} else {
lastErr = err
}
if attempt == cfg.MaxAttempts-1 {
break
}
delay := time.Duration(math.Min(
float64(cfg.BaseDelay)*math.Pow(2, float64(attempt)),
float64(cfg.MaxDelay),
))
jitter := time.Duration(rand.Int63n(int64(cfg.Jitter)))
time.Sleep(delay + jitter)
}
return lastErr
}Idempotency: The Key to Safe Retries#
Idempotency is the property that an operation can be applied multiple times without changing the result beyond the initial application. In the context of Saga, idempotency is critical because retries mean the same command may be delivered to a service multiple times. If the Payment Service receives a "Charge $100" command twice, the customer should be charged $100, not $200. The standard approach is to assign every Saga step a unique operation ID (typically the Saga ID combined with a step index). The service stores completed operation IDs in a database table with a unique constraint. Before executing any command, the service checks if the operation ID has already been processed. If yes, it returns the cached result. If no, it executes the operation and stores the operation ID. This pattern, sometimes called the Idempotent Receiver pattern, is described in depth in Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf.
@Service
@Transactional
public class PaymentService {
private final PaymentRepository paymentRepository;
private final IdempotencyKeyRepository idempotencyRepository;
public PaymentResult processPayment(ProcessPaymentCommand command) {
Optional<IdempotencyRecord> existing = idempotencyRepository
.findByKey(command.getSagaId() + "_" + command.getStepIndex());
if (existing.isPresent()) {
return existing.get().getResult();
}
PaymentResult result = paymentRepository.charge(
command.getCustomerId(),
command.getAmount()
);
idempotencyRepository.save(new IdempotencyRecord(
command.getSagaId() + "_" + command.getStepIndex(),
result
));
return result;
}
}Dead Letter Queue Pattern#
The Dead Letter Queue (DLQ) is a critical safety net in any production Saga implementation. When a message cannot be processed after exhausting all retry attempts, it is moved to a dedicated dead letter queue rather than being silently discarded. This serves three purposes: it prevents poison messages from blocking the processing of valid messages, it preserves the failed message for manual inspection and remediation, and it enables alerting and monitoring on Saga failures. In Apache Kafka, DLQ is typically implemented as a separate topic. In RabbitMQ, it is a queue bound with a dead-letter exchange. The DLQ should store the original message, the error details, the timestamp, and the retry count. Operations teams can then inspect the DLQ, fix the underlying issue, and replay the message back into the primary queue.
const { Kafka } = require('kafkajs');
const kafka = new Kafka({
clientId: 'saga-orchestrator',
brokers: ['kafka:9092'],
});
const consumer = kafka.consumer({ groupId: 'saga-consumer' });
const dlqProducer = kafka.producer();
async function processWithDLQ(topic, handler, maxRetries = 5) {
await consumer.connect();
await consumer.subscribe({ topic, fromBeginning: false });
await consumer.run({
eachMessage: async ({ message }) => {
const headers = parseHeaders(message.headers);
const retryCount = headers['x-retry-count'] || 0;
try {
await handler(JSON.parse(message.value.toString()));
} catch (error) {
if (retryCount >= maxRetries) {
await dlqProducer.send({
topic: `${topic}.dlq`,
messages: [{
key: message.key,
value: message.value,
headers: {
...message.headers,
'x-error': error.message,
'x-failed-at': new Date().toISOString(),
},
}],
});
} else {
throw error;
}
}
},
});
}Outbox Pattern: Reliable Event Publishing#
The Outbox Pattern solves a fundamental problem in event-driven Sagas: how to atomically update the database and publish an event. In a choreography-based Saga, after the Order Service creates an order, it must both persist the order to its database and publish an OrderCreated event to the message broker. If the database transaction succeeds but the event publish fails, the system is inconsistent—the order exists but no downstream service knows about it. If the event publish succeeds but the database transaction fails (e.g., constraint violation), the system is also inconsistent—services react to an order that doesn't exist. The Outbox Pattern solves this by writing the event to an outbox table within the same database transaction as the business data. A separate process (the Outbox Publisher) polls the outbox table and publishes events to the message broker, deleting them only after successful publish.
@Component
public class OutboxPublisher {
private final OutboxRepository outboxRepository;
private final KafkaTemplate<String, String> kafkaTemplate;
@Scheduled(fixedDelay = 100)
@Transactional
public void publishOutboxEvents() {
List<OutboxEvent> events = outboxRepository
.findTop100ByOrderByCreatedAtAsc();
for (OutboxEvent event : events) {
try {
kafkaTemplate.send(
event.getTopic(),
event.getAggregateId(),
event.getPayload()
).get(5, TimeUnit.SECONDS);
outboxRepository.delete(event);
} catch (Exception e) {
log.error("Failed to publish event {}", event.getId(), e);
}
}
}
}Inbox Pattern: Reliable Event Consumption#
The Inbox Pattern is the consumer-side counterpart of the Outbox Pattern. When a service receives an event from the message broker, it must process the event and update its local database atomically. If the service processes the event but crashes before acknowledging it to the broker, the broker redelivers it, causing duplicate processing. The Inbox Pattern solves this by writing the incoming event to an inbox table within the same database transaction as the business logic. The message broker consumer then simply inserts the event into the inbox and acknowledges it. A separate Inbox Processor reads from the inbox table, executes the business logic, and marks the event as processed. This ensures exactly-once processing semantics even when the message broker delivers at-least-once.
Exactly-Once Processing Semantics#
Exactly-once processing is the holy grail of distributed messaging, and it is notoriously difficult to achieve. Message brokers typically guarantee at-least-once delivery—a message will be delivered one or more times, but never zero times. This means consumers must be idempotent. However, with the combination of the Outbox Pattern, the Inbox Pattern, and idempotency keys, it is possible to achieve effectively-once processing. The Outbox ensures events are published exactly once (no lost events). The Inbox ensures events are processed exactly once (no duplicates). Idempotency keys ensure that even if an event is processed twice, the side effects are applied only once. Apache Kafka offers transactional producers and exactly-once semantics (EOS) at the broker level, but even with Kafka EOS, idempotent consumers are still recommended as a defense-in-depth measure.
| Semantics | Guarantee | How Achieved | Use in Saga |
|---|---|---|---|
| At-Most-Once | Message delivered 0 or 1 times | Consumer auto-ack before processing | Never use in Saga—can lose critical events |
| At-Least-Once | Message delivered 1+ times | Consumer ack after processing | Acceptable with idempotent consumers |
| Exactly-Once | Message delivered exactly 1 time | Idempotent consumer + transactional outbox | Gold standard for financial Sagas |
| Effectively-Once | Side effects applied once | Idempotency keys + deduplication | Practical approach for most enterprise Sagas |
Choreography vs Orchestration: The Definitive Comparison#
The choice between choreography and orchestration is one of the most consequential architectural decisions in a Saga implementation. Both approaches have their strengths and weaknesses, and the right choice depends on the complexity of the workflow, the size of the team, and the operational maturity of the organization. Choreography excels in environments with simple, linear workflows and highly autonomous teams where each service team can independently understand and react to events. Orchestration excels in complex workflows with branching, parallel steps, and sophisticated error handling. In practice, many organizations start with choreography for simple flows and migrate to orchestration as complexity grows. Some use a hybrid approach where an orchestrator handles the main Saga flow but individual steps use choreography for sub-workflows.
| Dimension | Choreography | Orchestration | Recommendation |
|---|---|---|---|
| Visibility | Low—workflow logic distributed across services | High—central orchestrator has full view of Saga state | Orchestration for production debugging |
| Coupling | Low—services only know about events | Medium—orchestrator knows about all participants | Choreography for service autonomy |
| Error Handling | Complex—each service must handle its own errors and compensations | Simple—orchestrator coordinates all retries and compensations | Orchestration for complex error handling |
| Testing | Difficult—requires all services to test end-to-end flow | Easier—orchestrator can be tested in isolation with mocks | Orchestration for testability |
| Scalability | Excellent—no central bottleneck | Good—orchestrator can be scaled horizontally with state partitioning | Both scale well for most use cases |
| Complexity Ceiling | Low—workflows with >5 steps become unwieldy | High—handles complex branching and parallel steps | Orchestration beyond 3-5 steps |
| Governance | Weak—hard to enforce workflow standards | Strong—orchestrator enforces the Saga protocol | Orchestration for compliance requirements |
Saga vs Two-Phase Commit (2PC)#
Two-Phase Commit is the traditional distributed transaction protocol used by XA-compliant databases and message brokers. In Phase 1 (Prepare), the transaction coordinator asks all participants if they can commit. Each participant executes the transaction up to the point of commit, acquires necessary locks, and responds with a vote: Yes or No. In Phase 2 (Commit), if all participants voted Yes, the coordinator sends a Commit command. If any voted No, the coordinator sends a Rollback command. The protocol is simple in theory but problematic in practice. The biggest issue is blocking: if the coordinator crashes after sending Prepare but before sending Commit/Rollback, participants remain locked indefinitely, holding database resources and blocking other transactions. This makes 2PC unsuitable for long-running transactions and high-availability systems.
| Property | Saga Pattern | Two-Phase Commit | Winner |
|---|---|---|---|
| Consistency | Eventual | Immediate (strong) | 2PC (for immediate) |
| Availability | High—no distributed locks | Low—blocking during prepare phase | Saga |
| Latency | Medium (async steps) | Low (synchronous protocol) | 2PC (for latency) |
| Lock Duration | No distributed locks | Locks held for entire TX duration | Saga |
| Failure Mode | Compensation (semantic undo) | Rollback (automatic undo) | 2PC (automatic) |
| Complexity | High—requires explicit compensation logic | Low—databases handle coordination | 2PC (for dev simplicity) |
| Scalability | Excellent—no shared coordinator state | Poor—coordinator is bottleneck | Saga |
| Use Case | Long-running, cross-service workflows | Short, same-database-type operations | Depends on requirements |
| Vendor Lock-in | None—pattern-based | Requires XA-compliant infrastructure | Saga |
| Observability | Good with orchestration | Limited—black-box protocol | Saga |
Saga with Apache Kafka#
Apache Kafka is the most popular message broker for implementing production Sagas at scale. Kafka's log-based architecture provides three critical properties for Sagas: durable storage (events are persisted to disk and replicated), strict ordering (within a partition, events are delivered in the order they were produced), and replayability (consumers can re-read events from any offset). In a choreography-based Saga with Kafka, each service publishes events to its own topic (e.g., order-events, payment-events, inventory-events). Other services consume from these topics and react. In an orchestration-based Saga, the orchestrator typically uses Kafka as the command/response channel: it publishes command messages to service-specific topics and consumes response messages. Kafka Connect can be used to implement the Outbox Pattern by tailing the database transaction log (CDC) and publishing to Kafka topics.
@Component
public class KafkaSagaOrchestrator {
private final KafkaTemplate<String, SagaCommand> commandTemplate;
private final SagaStateRepository stateRepository;
@KafkaListener(topics = "saga-responses", groupId = "saga-orchestrator")
public void handleResponse(SagaResponse response) {
SagaState state = stateRepository.findById(response.getSagaId())
.orElseThrow(() -> new SagaNotFoundException(response.getSagaId()));
if (response.isSuccess()) {
state.markStepComplete(response.getStepName());
SagaStep nextStep = state.getNextStep();
if (nextStep != null) {
sendCommand(nextStep, state);
} else {
state.markComplete();
stateRepository.save(state);
}
} else {
beginCompensation(state, response);
}
}
private void sendCommand(SagaStep step, SagaState state) {
SagaCommand command = new SagaCommand(
state.getSagaId(),
step.getName(),
step.getPayload()
);
commandTemplate.send("saga-commands", command);
stateRepository.save(state);
}
private void beginCompensation(SagaState state, SagaResponse failure) {
List<SagaStep> completedSteps = state.getCompletedStepsInReverse();
for (SagaStep step : completedSteps) {
SagaCommand compensation = step.createCompensationCommand();
commandTemplate.send("saga-commands", compensation);
}
state.markFailed(failure.getReason());
stateRepository.save(state);
}
}Saga with RabbitMQ#
RabbitMQ, with its flexible exchange and queue model, is another excellent choice for Saga implementations, particularly for workloads that benefit from complex routing and per-message acknowledgments. RabbitMQ's AMQP 0-9-1 protocol provides fine-grained control over message delivery: exchanges route messages to queues based on routing keys, and consumers acknowledge messages individually after successful processing. For Saga orchestration, a common pattern is to use a direct exchange for command routing (each service has its own command queue) and a topic exchange for event publishing. RabbitMQ's dead-letter exchange feature maps naturally to the Dead Letter Queue pattern. The delayed message plugin enables scheduled retries. However, RabbitMQ does not provide the same level of durability and replayability as Kafka—once a message is consumed and acknowledged, it is gone. For Sagas that require event sourcing or audit trails, Kafka is generally preferred.
const amqp = require('amqplib');
class SagaOrchestrator {
constructor() {
this.connection = null;
this.channel = null;
}
async connect() {
this.connection = await amqp.connect('amqp://localhost');
this.channel = await this.connection.createChannel();
await this.channel.assertExchange('saga.commands', 'direct', { durable: true });
await this.channel.assertExchange('saga.events', 'topic', { durable: true });
await this.channel.assertQueue('saga.orchestrator.responses', { durable: true });
await this.channel.assertQueue('saga.dlq', { durable: true });
}
async sendCommand(service, command) {
const queue = `saga.${service}.commands`;
await this.channel.assertQueue(queue, { durable: true });
await this.channel.bindQueue(queue, 'saga.commands', service);
this.channel.publish('saga.commands', service, Buffer.from(JSON.stringify(command)), {
persistent: true,
messageId: command.sagaId + '_' + command.stepIndex,
headers: { 'x-retry-count': 0 },
});
}
async handleResponse(msg) {
const response = JSON.parse(msg.content.toString());
const sagaState = await this.loadSagaState(response.sagaId);
if (response.success) {
await this.advanceSaga(sagaState);
} else {
await this.compensate(sagaState);
}
this.channel.ack(msg);
}
async compensate(sagaState) {
for (const step of sagaState.completedSteps.reverse()) {
await this.sendCommand(step.service, step.compensationCommand);
}
}
}Kafka vs RabbitMQ for Saga: Comparison#
| Feature | Apache Kafka | RabbitMQ | Best for Saga |
|---|---|---|---|
| Message Model | Distributed commit log (append-only) | Queue-based with exchanges and bindings | Kafka for event sourcing Sagas |
| Message Retention | Configurable (days/weeks/unlimited) | Until consumed and acknowledged | Kafka for audit and replay |
| Ordering | Strict within partition | Strict within queue with single consumer | Both guarantee ordering |
| Throughput | Millions of messages/second | Tens of thousands/second | Kafka for high-throughput Sagas |
| Routing Flexibility | Topic only (use partitions for routing) | Exchanges: direct, topic, fanout, headers | RabbitMQ for complex routing |
| Consumer Model | Pull-based (consumer polls) | Push-based (broker pushes to consumer) | Depends on consumer preference |
| Acknowledgments | Offset commit (batch) | Per-message ack/nack | RabbitMQ for fine-grained control |
| Dead Letter | Separate DLQ topic (manual) | Built-in dead-letter exchange | RabbitMQ (built-in DLX) |
| Operations | More complex (ZooKeeper/KRaft) | Simpler to operate | RabbitMQ for smaller teams |
Saga with CQRS#
CQRS (Command Query Responsibility Segregation) and the Saga Pattern are natural architectural allies. CQRS separates write operations (Commands) from read operations (Queries), typically using different data models and sometimes different databases. In a CQRS system, a Saga orchestrator sends Commands to aggregate roots in the write model. When a Command is successfully processed, the aggregate publishes a Domain Event. The Saga consumes these Domain Events to track progress. Meanwhile, event handlers update the read model asynchronously. This architecture decouples the transactional write path (which must be consistent) from the read path (which can be eventually consistent). The read model is a materialized view optimized for query performance, while the write model is optimized for transactional integrity. This is exactly the pattern used by enterprise teams at companies like HattaDev to build high-performance, scalable microservices.
Saga with Event Sourcing#
Event Sourcing takes the CQRS + Saga architecture to its logical extreme. Instead of storing the current state of an entity in a database, event sourcing stores the sequence of state-changing events that led to the current state. The current state is derived by replaying all events from the beginning (or from a snapshot). In a Saga context, event sourcing provides an immutable, auditable history of every Saga execution. Every command, every domain event, every compensation action is stored as an event in the event store. This is invaluable for debugging—you can reconstruct exactly what happened during a failed Saga by replaying its events. It also enables temporal queries ("what was the state of this order at 3:15 PM?") and retroactive fixes ("replay all events with the corrected business logic"). The trade-off is increased storage (every event is kept forever) and complexity (event schema evolution, snapshots, projections).
Saga with Domain Events#
Domain Events are the language of business in event-driven architectures. A Domain Event is something meaningful that happened in the business domain—OrderPlaced, PaymentAuthorized, ShipmentDispatched—not a technical event like RowUpdated. In Saga implementations, using Domain Events rather than technical messages dramatically improves the understandability and maintainability of the system. When the Saga Orchestrator receives an OrderPlaced event, the intent is immediately clear. Domain Events also serve as the contract between services. Each service publishes and subscribes to well-defined Domain Events with explicit schemas. This aligns with Domain-Driven Design (DDD) principles and ensures that the Saga workflow reflects the actual business process rather than technical plumbing. The schema of each Domain Event should be versioned and governed to prevent breaking changes.
Saga in Event-Driven Architecture#
Event-Driven Architecture (EDA) is the natural habitat of the Saga Pattern. In an EDA, services communicate by emitting and consuming events, not by making synchronous RPC calls. The Saga Pattern fits perfectly into this model: each Saga step is triggered by an event, executes a local transaction, and emits one or more events that trigger subsequent steps. This asynchronous, event-driven approach provides loose coupling, scalability, and resilience. Services can be deployed independently, scaled independently, and can fail independently without bringing down the entire system. However, EDA introduces challenges: event ordering must be managed, duplicate events must be handled, and the event schema must evolve carefully. The combination of Saga + EDA is the dominant architectural pattern for modern enterprise microservices at scale, adopted by organizations ranging from startups to Fortune 500 companies.
Production Architecture for Saga in Microservices#
A production-grade Saga implementation in microservices requires more than just the core pattern. It needs a complete ecosystem: API Gateway for routing and authentication, Service Registry for discovery, Message Broker for asynchronous communication, Distributed Tracing for observability, Metrics Collection for monitoring, Centralized Logging for debugging, Secret Management for credentials, and CI/CD pipelines for deployment. The Saga Orchestrator itself should be deployed as a horizontally scaled service with a persistent state store (PostgreSQL with SKIP LOCKED for job queue semantics). Each microservice should implement health checks, circuit breakers, and graceful shutdown. The message broker should be clustered for high availability. Database connections should use connection pooling. All inter-service communication should be encrypted with TLS. This is the level of rigor that enterprise engineering teams apply to Saga implementations.
Real-World Case Studies#
Uber's payment platform processes millions of transactions daily across hundreds of microservices. They implemented a choreography-based Saga using Apache Kafka for asynchronous event propagation. Each step in their payment flow—fraud check, payment authorization, receipt generation—is a separate microservice that reacts to events. Uber chose choreography over orchestration to maintain service autonomy and because their payment flow is relatively linear. They invested heavily in observability, building custom tools to visualize event flows and detect stalled Sagas. When a Saga fails, their operations team uses Kafka's log compaction to replay events and reconstruct the exact state at failure time. This approach allowed Uber to scale their payment platform to billions of transactions while maintaining 99.99% reliability. Their architecture validates that choreography can work at massive scale with sufficient operational investment in observability.
Netflix implemented their own Saga orchestration framework called Conductor, which has since been open-sourced and adopted by hundreds of organizations. Conductor provides a workflow-as-code model where developers define Sagas as JSON-based workflow definitions. The Conductor server manages task scheduling, state persistence, retries, and compensation. Netflix uses Conductor for everything from content ingestion pipelines to billing workflows. The key lesson from Netflix is the importance of making Saga workflows visible and debuggable. Conductor provides a UI that shows the exact state of every Saga, every task within it, and every retry. This visibility transformed their operational posture: instead of hunting through logs across 20 services to debug a failed order, operators can see the entire Saga timeline in a single view. This is the gold standard for Saga observability.
A major European bank rebuilt their core banking system using orchestration-based Saga with CQRS and Event Sourcing. Every financial transaction—transfer, deposit, withdrawal—is a Saga. The event store contains billions of events representing every financial movement in the bank's history. They chose orchestration because financial workflows are complex with many branching paths (fraud checks, compliance checks, currency conversion). The event-sourced architecture allows them to answer any question about any transaction at any point in time, which satisfies regulatory requirements. Performance is maintained through snapshotting (storing the aggregate state at intervals so replays don't start from the beginning). This project demonstrated that Saga + Event Sourcing can handle the strictest consistency and audit requirements in the most regulated industry.
Best Practices#
- Always persist Saga state to a durable store. The orchestrator must survive restarts without losing track of in-flight Sagas.
- Make every Saga step idempotent. Use operation IDs with a unique constraint in the database. Assume every command will be delivered at least twice.
- Implement the Outbox Pattern for reliable event publishing. Never publish events outside the database transaction boundary.
- Set timeouts for every Saga step. If a step does not complete within a defined SLA, cancel it and begin compensation.
- Version your Domain Events. Use semantic versioning and ensure backward compatibility for at least one major version.
- Implement a Dead Letter Queue. Never silently drop failed messages. Every poison message must be preserved for inspection.
- Use structured logging with correlation IDs. Every log entry should include the Saga ID, step index, and service name.
- Implement distributed tracing. Every Saga execution should produce a single trace with spans for each step, visible in tools like Jaeger or Grafana Tempo.
- Design compensation transactions as business operations, not database rollbacks. Record compensations as new events, never delete data.
- Test your Sagas with chaos engineering. Kill services mid-Saga, partition the network, exhaust connection pools.
- Monitor Saga completion rate, average duration, and compensation rate. Alert on anomalies.
- Implement circuit breakers for downstream services. If a service is degraded, fail fast rather than piling up retries.
- Keep Saga workflows as small as possible. If a workflow requires more than 10 steps, consider splitting it into sub-Sagas.
- Use a schema registry for event schemas (e.g., Confluent Schema Registry for Kafka). Enforce schema compatibility at the broker level.
Common Mistakes and Anti-Patterns#
- Using Saga for transactions that should be local. If a transaction touches only one database, use a local ACID transaction—not a Saga.
- Not implementing idempotency. This is the number one cause of duplicate charges, double-shipped orders, and data corruption in production Sagas.
- Designing compensations as database DELETEs. Deleting data loses audit trail and may violate foreign key constraints. Use status transitions instead.
- Ignoring the Outbox Pattern. Publishing events directly from service code without transactional guarantees leads to lost events under failure.
- Using Saga for ultra-low-latency requirements. Saga is inherently asynchronous. If a transaction must complete in under 10ms, Saga is the wrong pattern.
- Over-choreographing complex workflows. Choreography with more than 5 services becomes a debugging nightmare. Use orchestration.
- Not setting timeouts. A Saga step that waits indefinitely for a response will exhaust resources and block dependent workflows.
- Mixing synchronous and asynchronous communication within the same Saga step. This creates complex failure modes that are hard to reason about.
- Using the same Dead Letter Queue for all Sagas. Different Sagas have different severity levels and response requirements. Use per-Saga DLQs.
- Not monitoring compensation rates. If your compensation rate suddenly spikes, something is wrong. Treat it as a critical alert.
- Tight coupling between orchestrator and service implementations. The orchestrator should send commands, not make assumptions about how services implement them.
- Storing Saga state in-memory. If the orchestrator restarts, all in-flight Sagas are lost. Always persist Saga state to a database.
Performance and Scalability Considerations#
Performance in Saga systems is measured by Saga completion time—the total time from initiation to final state. The primary contributors to Saga latency are network round-trips between the orchestrator and services, service processing time, and message broker latency. In a 5-step Saga where each step takes 50ms of processing and 10ms of network latency, the minimum completion time is 300ms. To reduce this, parallelize independent steps. If Step 2 (Payment) and Step 3 (Inventory) are independent, the orchestrator can dispatch them simultaneously and wait for both to complete. This cuts latency from sequential execution time to max(step times). Kafka's partitioning model enables natural parallelism: commands for different Sagas are distributed across partitions, and multiple orchestrator instances can process different partitions concurrently. For high-throughput systems processing thousands of Sagas per second, consider using a dedicated Saga processing cluster with horizontal auto-scaling based on queue depth.
Scalability concerns center on the orchestrator and the message broker. The orchestrator is typically the bottleneck because it manages state and coordinates all steps. However, because each Saga is independent (no shared state between different Sagas except through the message broker), orchestrators can be scaled horizontally. The key is to partition Saga state by Saga ID hash. Multiple orchestrator instances can operate independently, each owning a subset of Sagas. The message broker must also scale—Kafka scales by adding brokers and partitions, RabbitMQ scales by adding nodes to a cluster and using consistent hash exchanges. Database scalability for Saga state persistence is typically not a concern because the state per Saga is small (a few KB) and the write pattern is append-only (state transitions). PostgreSQL can handle millions of Saga state transitions per day with proper indexing on Saga ID and status.
Security Considerations#
Security in Saga implementations spans authentication, authorization, encryption, and data integrity. Every command sent through the Saga must be authenticated—the orchestrator and services must mutually verify each other's identity using mTLS or API keys. Commands must be authorized—a service should verify that the orchestrator has permission to request the specific action for the specific Saga. All messages in transit must be encrypted using TLS 1.3. Sensitive data in message payloads (PII, payment details) should be encrypted at the field level using envelope encryption with a key management service. The Saga state database must be encrypted at rest. Access to the Dead Letter Queue must be restricted to operations personnel. The orchestrator's API (for querying Saga status) must require authentication. In regulated industries, every Saga step must produce an audit log entry with cryptographic integrity guarantees.
Monitoring, Logging, and Distributed Tracing#
The three pillars of observability—metrics, logs, and traces—are non-negotiable for production Sagas. Metrics provide aggregate visibility: Saga initiation rate, completion rate, average duration, compensation rate, and step-level latency percentiles (p50, p95, p99). These should be exported to Prometheus and visualized in Grafana dashboards. Logs provide detailed visibility into specific Saga executions. Every log entry must include the Saga ID, step name, and correlation ID. Use structured logging (JSON format) so logs can be queried in Elasticsearch or Loki. Traces provide end-to-end visibility. With OpenTelemetry, you can trace a Saga from the initial API request through every command, every service, every database query, and every message broker hop. The tracing context (trace ID and span ID) must be propagated through HTTP headers and message metadata. In Jaeger or Grafana Tempo, you can see the entire Saga execution as a single trace with nested spans.
@Component
public class ObservableSagaOrchestrator {
private final Tracer tracer;
private final MeterRegistry meterRegistry;
private final Logger log = LoggerFactory.getLogger(ObservableSagaOrchestrator.class);
private final Counter sagaStarted;
private final Counter sagaCompleted;
private final Counter sagaCompensated;
private final Timer sagaDuration;
public ObservableSagaOrchestrator(OpenTelemetry openTelemetry, MeterRegistry meterRegistry) {
this.tracer = openTelemetry.getTracer("saga-orchestrator", "1.0.0");
this.meterRegistry = meterRegistry;
this.sagaStarted = Counter.builder("saga.started")
.description("Number of Sagas started")
.register(meterRegistry);
this.sagaCompleted = Counter.builder("saga.completed")
.register(meterRegistry);
this.sagaCompensated = Counter.builder("saga.compensated")
.register(meterRegistry);
this.sagaDuration = Timer.builder("saga.duration")
.register(meterRegistry);
}
public void executeSaga(SagaDefinition definition) {
Span sagaSpan = tracer.spanBuilder(definition.getSagaType())
.setAttribute("saga.id", definition.getSagaId())
.startSpan();
sagaStarted.increment();
Timer.Sample sample = Timer.start(meterRegistry);
try (Scope scope = sagaSpan.makeCurrent()) {
log.info("Saga started: type={} sagaId={}",
definition.getSagaType(), definition.getSagaId());
for (SagaStep step : definition.getSteps()) {
Span stepSpan = tracer.spanBuilder(step.getName())
.setAttribute("saga.id", definition.getSagaId())
.setAttribute("step.index", step.getIndex())
.startSpan();
try (Scope stepScope = stepSpan.makeCurrent()) {
SagaStepResult result = step.execute();
stepSpan.setAttribute("step.success", result.isSuccess());
if (result.isFailure()) {
sagaCompensated.increment();
compensate(definition, step.getIndex());
break;
}
} finally {
stepSpan.end();
}
}
sagaCompleted.increment();
} catch (Exception e) {
sagaSpan.recordException(e);
sagaSpan.setStatus(StatusCode.ERROR);
throw e;
} finally {
sample.stop(sagaDuration);
sagaSpan.end();
}
}
}Enterprise Observability with OpenTelemetry, Prometheus, and Grafana#
The OpenTelemetry ecosystem provides the instrumentation layer for Saga observability. The OpenTelemetry Collector receives traces, metrics, and logs from all services and exports them to backends: traces to Jaeger or Grafana Tempo, metrics to Prometheus, logs to Loki or Elasticsearch. For Saga specifically, the Auto-Instrumentation agents for Java, .NET, Go, Node.js, and Python automatically capture HTTP requests, database queries, and Kafka/RabbitMQ message processing as spans. The only manual instrumentation needed is creating Saga-level spans that group the individual step spans. This gives you a complete picture: a Saga trace in Jaeger shows the orchestrator span, nested under which are spans for each command sent, each service's processing, and each message broker hop. Grafana dashboards combine metrics from Prometheus (Saga throughput, latency, error rates) with logs from Loki (detailed step logs for specific Sagas). This tri-pillar approach enables both proactive monitoring and reactive debugging.
Saga Implementation in Java (Spring Boot)#
@Service
public class OrderSagaOrchestrator {
private final OrderService orderService;
private final PaymentService paymentService;
private final InventoryService inventoryService;
private final ShippingService shippingService;
private final SagaStateRepository sagaStateRepository;
@Transactional
public OrderResult placeOrder(PlaceOrderRequest request) {
SagaState state = SagaState.create("ORDER_SAGA", request.getOrderId());
sagaStateRepository.save(state);
try {
Order order = orderService.createOrder(request);
state.addStep("CREATE_ORDER", order.getId());
Payment payment = paymentService.reserveCredit(
request.getCustomerId(), request.getTotalAmount());
state.addStep("RESERVE_CREDIT", payment.getTransactionId());
InventoryReservation reservation = inventoryService
.reserveItems(request.getItems());
state.addStep("RESERVE_INVENTORY", reservation.getId());
Shipment shipment = shippingService.createShipment(
order.getId(), request.getShippingAddress());
state.addStep("CREATE_SHIPMENT", shipment.getId());
paymentService.capturePayment(payment.getTransactionId());
orderService.confirmOrder(order.getId());
state.markComplete();
return OrderResult.success(order.getId());
} catch (SagaStepException e) {
compensate(state);
state.markFailed(e.getMessage());
return OrderResult.failure(e.getMessage());
} finally {
sagaStateRepository.save(state);
}
}
private void compensate(SagaState state) {
List<SagaStep> steps = state.getStepsInReverse();
for (SagaStep step : steps) {
try {
switch (step.getName()) {
case "CREATE_SHIPMENT":
shippingService.cancelShipment(step.getEntityId());
break;
case "RESERVE_INVENTORY":
inventoryService.releaseItems(step.getEntityId());
break;
case "RESERVE_CREDIT":
paymentService.releaseCredit(step.getEntityId());
break;
case "CREATE_ORDER":
orderService.cancelOrder(step.getEntityId());
break;
}
state.addCompensation(step.getName());
} catch (Exception e) {
log.error("Compensation failed for step {}: {}",
step.getName(), e.getMessage());
}
}
}
}Saga Implementation in ASP.NET Core#
public class OrderSagaCoordinator : IOrderSagaCoordinator
{
private readonly IOrderRepository _orderRepo;
private readonly IPaymentGateway _paymentGateway;
private readonly IInventoryService _inventoryService;
private readonly ISagaStateStore _stateStore;
private readonly ILogger<OrderSagaCoordinator> _logger;
public async Task<SagaResult> ExecuteAsync(PlaceOrderCommand command)
{
var sagaState = SagaState.Create(command.OrderId, "OrderSaga");
await _stateStore.SaveAsync(sagaState);
try
{
var order = await _orderRepo.CreateAsync(command);
sagaState.AddStep("CreateOrder", order.Id);
var payment = await _paymentGateway.AuthorizeAsync(
command.CustomerId, command.TotalAmount);
sagaState.AddStep("AuthorizePayment", payment.TransactionId);
var reservation = await _inventoryService.ReserveAsync(command.Items);
sagaState.AddStep("ReserveInventory", reservation.ReservationId);
await _paymentGateway.CaptureAsync(payment.TransactionId);
sagaState.MarkComplete();
return SagaResult.Success(order.Id);
}
catch (SagaStepException ex)
{
_logger.LogWarning(ex, "Saga {SagaId} failed at step {Step}",
sagaState.SagaId, ex.StepName);
await CompensateAsync(sagaState);
sagaState.MarkFailed(ex.Message);
return SagaResult.Failure(ex.Message);
}
finally
{
await _stateStore.SaveAsync(sagaState);
}
}
private async Task CompensateAsync(SagaState state)
{
foreach (var step in state.Steps.Reverse())
{
await ExecuteCompensationStep(step);
state.AddCompensation(step.Name);
}
}
private async Task ExecuteCompensationStep(SagaStep step)
{
switch (step.Name)
{
case "ReserveInventory":
await _inventoryService.ReleaseAsync(step.EntityId);
break;
case "AuthorizePayment":
await _paymentGateway.VoidAsync(step.EntityId);
break;
case "CreateOrder":
await _orderRepo.CancelAsync(step.EntityId);
break;
}
}
}Saga Implementation in Go#
package saga
import (
"context"
"fmt"
"log"
)
type StepFunc func(ctx context.Context, state *SagaState) error
type CompensateFunc func(ctx context.Context, state *SagaState) error
type SagaStep struct {
Name string
Execute StepFunc
Compensate CompensateFunc
}
type SagaDefinition struct {
Steps []SagaStep
}
type SagaOrchestrator struct {
store SagaStateStore
}
func NewSagaOrchestrator(store SagaStateStore) *SagaOrchestrator {
return &SagaOrchestrator{store: store}
}
func (so *SagaOrchestrator) Execute(ctx context.Context, sagaID string, def SagaDefinition) error {
state := NewSagaState(sagaID)
if err := so.store.Save(ctx, state); err != nil {
return fmt.Errorf("failed to save initial saga state: %w", err)
}
for i, step := range def.Steps {
select {
case <-ctx.Done():
return so.compensate(ctx, state, i-1, def)
default:
}
if err := step.Execute(ctx, state); err != nil {
log.Printf("Saga step %s failed: %v", step.Name, err)
return so.compensate(ctx, state, i-1, def)
}
state.MarkStepComplete(step.Name)
if err := so.store.Save(ctx, state); err != nil {
return fmt.Errorf("failed to persist saga state: %w", err)
}
}
state.MarkComplete()
return so.store.Save(ctx, state)
}
func (so *SagaOrchestrator) compensate(ctx context.Context, state *SagaState, lastCompleted int, def SagaDefinition) error {
for i := lastCompleted; i >= 0; i-- {
if err := def.Steps[i].Compensate(ctx, state); err != nil {
log.Printf("Compensation failed for step %s: %v", def.Steps[i].Name, err)
state.MarkCompensationFailed(def.Steps[i].Name)
} else {
state.MarkCompensated(def.Steps[i].Name)
}
if err := so.store.Save(ctx, state); err != nil {
return fmt.Errorf("failed to persist compensation state: %w", err)
}
}
state.MarkFailed("All steps compensated")
return so.store.Save(ctx, state)
}Saga Implementation in Python#
import logging
from dataclasses import dataclass, field
from typing import Callable, List, Dict, Any
from enum import Enum
logger = logging.getLogger(__name__)
class SagaStatus(Enum):
PENDING = "PENDING"
IN_PROGRESS = "IN_PROGRESS"
COMPLETED = "COMPLETED"
COMPENSATING = "COMPENSATING"
FAILED = "FAILED"
@dataclass
class SagaStep:
name: str
execute: Callable[[Dict[str, Any]], Dict[str, Any]]
compensate: Callable[[Dict[str, Any]], None]
@dataclass
class SagaState:
saga_id: str
saga_type: str
status: SagaStatus = SagaStatus.PENDING
completed_steps: List[str] = field(default_factory=list)
context: Dict[str, Any] = field(default_factory=dict)
def mark_step_complete(self, step_name: str):
self.completed_steps.append(step_name)
def mark_complete(self):
self.status = SagaStatus.COMPLETED
def mark_compensating(self):
self.status = SagaStatus.COMPENSATING
def mark_failed(self):
self.status = SagaStatus.FAILED
class SagaOrchestrator:
def __init__(self, state_repository):
self.state_repository = state_repository
def execute(self, saga_id: str, saga_type: str, steps: List[SagaStep], initial_context: Dict[str, Any] = None) -> SagaState:
state = SagaState(saga_id=saga_id, saga_type=saga_type, context=initial_context or {})
state.status = SagaStatus.IN_PROGRESS
self.state_repository.save(state)
try:
for i, step in enumerate(steps):
logger.info(f"Executing saga step: {step.name} for saga {saga_id}")
result = step.execute(state.context)
state.context.update(result)
state.mark_step_complete(step.name)
self.state_repository.save(state)
state.mark_complete()
self.state_repository.save(state)
return state
except Exception as e:
logger.error(f"Saga {saga_id} failed at step {step.name}: {e}")
self._compensate(state, steps, i - 1)
state.mark_failed()
self.state_repository.save(state)
return state
def _compensate(self, state: SagaState, steps: List[SagaStep], last_completed: int):
state.mark_compensating()
for i in range(last_completed, -1, -1):
step = steps[i]
try:
logger.info(f"Compensating step: {step.name} for saga {state.saga_id}")
step.compensate(state.context)
except Exception as e:
logger.critical(f"Compensation failed for step {step.name}: {e}")Frequently Asked Questions#
What is the Saga Pattern in microservices?▾
What is the difference between choreography and orchestration in Saga?▾
How does Saga handle failures?▾
What is a compensating transaction?▾
Why can't I use Two-Phase Commit instead of Saga?▾
What is the Outbox Pattern and why is it important for Saga?▾
How do I make Saga steps idempotent?▾
Should I use Kafka or RabbitMQ for Saga?▾
How does Saga relate to CQRS and Event Sourcing?▾
What is the Dead Letter Queue in Saga?▾
How do I monitor Sagas in production?▾
What is eventual consistency and how does it affect Saga?▾
Can Saga be used with synchronous HTTP calls?▾
How many steps should a Saga have?▾
What are Domain Events and why are they important for Saga?▾
How do I handle duplicate events in Saga?▾
Is Saga suitable for financial transactions?▾
How does Saga handle concurrent modifications?▾
What is the Inbox Pattern?▾
How do I test Sagas?▾
What is the difference between a Saga and a workflow engine?▾
How do I handle timeouts in Saga?▾
Can multiple Sagas interact with each other?▾
How do I version Saga workflows?▾
What are the most common Saga anti-patterns?▾
How does Saga ensure exactly-once processing?▾
How do I ensure Saga state survives orchestrator restarts?▾
What observability tools should I use for Saga?▾
Should I build my own Saga framework or use an existing one?▾
What is the relationship between Saga and the CAP theorem?▾
Explore Enterprise Software Engineering
Discover more enterprise architecture, distributed systems, cloud-native engineering, AI engineering and software engineering guides from HattaDev.