The Problem: One Action, Two Systems
Imagine an e-commerce service processing a new order. It needs to perform two actions:
1. Save the order in the database.
2. Publish an OrderCreated event so that payment, inventory, and notification services can react.
A straightforward implementation might look like this:
order = Order.create!(order_params)
message_broker.publish("OrderCreated", order)
This looks correct, but it contains a dangerous distributed-systems problem known as the dual-write problem.
What happens if the database update succeeds, but the message broker becomes unavailable before the event is published?
The order exists, but the other services never hear about it. Inventory is not reserved, payment is not started, and no confirmation email is sent.
Reversing the order of operations does not solve the problem. If we publish the event first and the database operation later fails, other services may process an order that does not actually exist.
Why a Database Transaction Is Not Enough
A database transaction can make multiple database changes succeed or fail together. However, a normal transaction cannot also guarantee that an external message broker operation succeeds.
The database and message broker are separate systems. Each system can succeed or fail independently.
The Transactional Outbox Pattern solves this by temporarily treating the event as database data.
How the Transactional Outbox Pattern Works
Instead of publishing the event directly, the application performs two database operations inside the same transaction:
1. Save the business record.
2. Save the event in an outbox table.
Application Database
orders
+----+----------+---------+
| id | customer | status |
+----+----------+---------+
| 42 | Alice | pending |
+----+----------+---------+
outbox_events
+----+--------------+-----------+
| id | event_type | processed |
+----+--------------+-----------+
| 81 | OrderCreated | false |
+----+--------------+-----------+
Because both records are stored in one database transaction, they either succeed together or fail together.
Database.transaction do
order = Order.create!(order_params)
OutboxEvent.create!(
event_type: "OrderCreated",
aggregate_id: order.id,
payload: {
order_id: order.id,
customer_id: order.customer_id
}
)
end
A separate background worker regularly reads unprocessed events from the outbox table and publishes them to the message broker.
OutboxEvent.pending.find_each do |event|
message_broker.publish(event.event_type, event.payload)
event.update!(processed_at: Time.current)
end
If the broker is temporarily unavailable, the event remains in the outbox and can be retried later. The important business event is no longer lost.
The Pattern in Four Steps
Step 1: The application receives a request, such as creating an order.
Step 2: It stores both the order and its corresponding event in one database transaction.
Step 3: A background publisher reads pending outbox records.
Step 4: The publisher sends each event to the message broker and marks it as processed.
The Duplicate Message Problem
The Outbox Pattern prevents lost events, but a message may occasionally be published more than once.
For example, the publisher could successfully send an event and then crash before marking it as processed. When the worker restarts, it may publish the same event again.
For this reason, consumers should be idempotent. Processing the same event multiple times should have the same effect as processing it once.
return if ProcessedEvent.exists?(event_id: event.id)
Database.transaction do
reserve_inventory(event.order_id)
ProcessedEvent.create!(event_id: event.id)
end
Every event should therefore include a unique identifier. Consumers can store processed identifiers and ignore duplicates.
Important Production Considerations
Retry failures: Use retry delays and exponential backoff when the broker is unavailable.
Lock records: When multiple workers publish events, use row locking or another claiming mechanism so they do not process the same record simultaneously.
Monitor the backlog: Alert the team when the number or age of pending outbox events becomes unusually high.
Clean old records: Archive or delete successfully published events after an appropriate retention period.
Design for ordering: If event order matters, include an aggregate identifier and sequence number.
When Should You Use It?
The Transactional Outbox Pattern is useful when an application must update its database and reliably notify another system about that change.
Common examples include creating an order and starting payment, updating inventory and notifying a warehouse, registering a user and sending a welcome email, or changing an account and updating an analytics pipeline.
It may be unnecessary for simple applications where occasional message loss is acceptable or where all operations occur inside one database.
Final Takeaway
The main idea is simple:
Do not try to reliably write to two independent systems at the same time.
Write the business data and event to the same database transaction first. Then publish the saved event asynchronously.
The Transactional Outbox Pattern does not remove every distributed-systems challenge. You still need retries, monitoring, cleanup, and idempotent consumers. However, it gives your application a reliable foundation for event-driven communication without introducing complex distributed transactions.