Skip to content

Grokem · https://www.grokem.com/programming/software-architecture/event-driven-architecture/transactional-outbox-pattern-polling-vs-debezium-cdc-for-microservice-events

Transactional outbox pattern: polling vs Debezium CDC for microservice events

The transactional outbox removes the dual write between a service database and a broker, but you still choose how outbox rows reach the broker. This comparison judges a polling publisher, Debezium CDC, the Debezium MongoDB router and Spring Modulith listeners on stated criteria.

Krishna Sharma

Technical Writer

Published
  • 10 min read
  • 0 sec

Tested with
  • Spring Modulith2.0
  • MongoDB4.0
  • Debezium outbox event router SMT
  • Debezium MongoDB outbox event router SMT
  • Apache Kafka
Your progress
  1. Transactional outbox pattern: polling vs Debezium CDC for microservice events10 min(finished)
    0%

A microservice that updates its database and then sends an event to a broker writes to two systems that share no transaction. The transactional outbox pattern removes that gap: you write the event to an outbox table in the same database transaction as the state change, and a separate mechanism delivers it to the broker.

This article compares four ways to build the delivery half, then recommends one per situation. It targets experienced engineers who already accept the pattern and need to pick a relay. Versions: Spring Modulith 2.0 and MongoDB 4.0 (the minimum for multi-document transactions in the MongoDB option). The Debezium pages do not list a version for the router SMTs in the instructions you rely on here, so check the release you run against the official Debezium documentation.

The problem the outbox solves

A dual write happens when an application writes to two different systems, for example a database and a message broker. A failure in one of the two operations can leave the data inconsistent.

There are two failure shapes:

  • The database update commits but the event is never sent. Downstream services never learn about the change.
  • The event is sent but the database transaction rolls back. Downstream services act on a change that never happened.

The outbox pattern pushes the atomicity into the database. You update your state and insert an outbox row in one transaction. If the state update fails, the event is not written. The outbox therefore stays in sync with the database, and every committed change has a matching event, with some delay.

The delay is the cost. A second mechanism must move rows from the outbox to the broker, and that mechanism decides most of your operational trade-offs.

The options and how they are judged

You compare four approaches:

  1. Polling publisher: a process reads the outbox table, sends events to the broker, and removes rows once delivered.
  2. Debezium with the outbox event router SMT for relational databases: a connector captures changes in the outbox table and the SMT turns each row into a routed event.
  3. Debezium with the MongoDB outbox event router SMT: the same idea for an outbox collection.
  4. Spring Modulith application events: in-process event publication between modules, with transactional listeners.

The criteria, applied to each option below:

  • Atomicity with the business change
  • Ordering of events
  • Delivery guarantee and duplicate handling
  • Operational footprint
  • Database fit
  • Outbox growth and write load
MICROSERVICEOne database transactionBusiness tableOutbox tableALTERNATIVESPolling publisherDebezium connectorwith outbox eventrouter SMTKafka topicoutbox.event.orderConsumerIdempotent,deduplicate by event id
Two relay options deliver the same outbox rows to the broker.

Option 1: polling publisher

The relay is a process that asynchronously monitors the outbox table. When it sees a new event, it delivers it to the broker. After successful delivery, it can remove the row. You can write it as another thread inside the original microservice or run it as a separate application.

You create the table in the same database as your business data and write to both in one transaction. The columns below match the layout the Debezium router expects by default, so you keep the option of moving to CDC later without changing the schema. The PostgreSQL types are the ones shown in the Debezium documentation.

The transaction matters more than the table. If the state update fails, the insert rolls back with it, so no event leaks out for a change that did not happen.

Strengths

  • Smallest dependency set: your service, your database and your broker client.
  • You control the batching, retry and deletion policy.
  • Works with any database that supports transactions.

Weaknesses

  • Retries after a failure or timeout can send a message twice, so delivery is at-least-once.
  • Order is your responsibility. You must send events in the order the service updated the database. An outbox table that stores a timestamp and a sequence number gives you something to sort on.
  • Every write adds an outbox insert, which can mean a lot of traffic on the table.
  • If you run several relay instances, how they share work and keep order is not described in the documentation. Design and test it yourself.
sql
CREATE TABLE orders (
    id bigint PRIMARY KEY,
    status varchar(50) NOT NULL
);

CREATE TABLE outboxevent (
    id uuid NOT NULL PRIMARY KEY,
    aggregatetype varchar(255) NOT NULL,
    aggregateid varchar(255) NOT NULL,
    type varchar(255) NOT NULL,
    payload jsonb
);

BEGIN;

INSERT INTO orders (id, status) VALUES (1, 'ENTERED');

INSERT INTO outboxevent (id, aggregatetype, aggregateid, type, payload)
VALUES (
    '4d47e190-0402-4048-bc2c-89dd54343cdc',
    'Order',
    '1',
    'OrderCreated',
    '{"id": 1, "customerId": 123}'::jsonb
);

COMMIT;

Option 2: Debezium outbox event router (relational)

With Debezium you do not write the relay. You configure a connector to capture changes in an outbox table and apply the outbox event router SMT. The connector should capture changes in the outbox table only. It can capture more than one outbox table only if every table has the same structure.

By default the SMT assumes these columns: id, aggregatetype, aggregateid, type and payload.

  • id is the unique event ID. In the outbox message it becomes a header, and you can use it to remove duplicate messages.
  • aggregatetype contributes to the topic name. In the Debezium example, an Order aggregate produces the topic outbox.event.order.
  • aggregateid is the event key, which gives an ID for the payload.
  • type and payload describe the event and carry its content.

Debezium's raw change event wraps the row in before, after, source and op fields. The SMT replaces that envelope with the payload as the message value, the aggregateid as the key and the event id as a header. Consumers see a business event, not a table change.

The configuration option names are on the official Outbox Event Router page, and you should copy them from there. They are not reproduced here because the options are numerous and version specific.

Strengths

  • No custom relay code to write, deploy or keep alive.
  • Changes come from the database change stream, so ordering follows the order of the captured changes.
  • Default routing by aggregate gives you one topic per aggregate type.

Weaknesses

  • You operate a connector and the platform it runs on.
  • Consumers still need deduplication. The event ID header exists for that purpose.
  • The outbox table still receives an insert for every write.
  • The SMT is not compatible with the Debezium MongoDB connector.

Option 3: Debezium MongoDB outbox event router

MongoDB users run a separate SMT, the MongoDB outbox event router, which works with the Debezium MongoDB connector only. It follows the same flow: capture an outbox collection, apply the SMT.

The default outbox collection document has these fields: _id (objectId), aggregatetype (string), aggregateid (objectId), type (string) and payload (object).

The key constraint is transactional. Operations on your business collections and the insert into the outbox collection must run in one multi-document transaction. MongoDB has supported those since version 4.0, and this article pins that version as the floor. Without a multi-document transaction you get the inconsistency the pattern exists to prevent.

The Debezium page says that a future configuration is planned to store outbox events as a sub-document of an existing collection. That would allow an ACID update without multi-document transactions. It is planned, not available, so do not design around it.

Strengths

  • A documented route for MongoDB services, with the same default routing model as the relational SMT.

Weaknesses

  • Requires multi-document transactions on the write path.
  • An outbox collection is the only supported storage shape today.
  • The same collection-structure rule applies: several outbox collections only if they share one structure.

Option 4: Spring Modulith application events

Spring Modulith addresses a neighbouring problem: how application modules inside one application interact. Its events page recommends publishing a domain event through Spring's ApplicationEventPublisher instead of calling another module's bean. This guide targets Spring Modulith 2.0.

Publication is synchronous by default. That gives a simple consistency model: the order change and the dependent work both succeed or both fail. It also widens the transaction boundary, so a non-critical listener can fail the whole transaction.

Moving the listener to @Async with @TransactionalEventListener runs it after commit. That decouples the transactions but creates the exact risk the outbox addresses. If the listener fails, the event publication is lost unless each listener has its own safety net, and the system can fail before the method is even called.

Spring Modulith offers @ApplicationModuleListener as a shortcut for the declaration of an asynchronous listener that runs in its own transaction. The page also points to Spring Data's application event publication mechanism for an aggregate-driven approach.

This option is an in-process pattern. The documentation on this page does not describe delivery to an external broker or a relay. Read the Spring Modulith events page for how event publications are persisted and externalized before you count it as an outbox.

java
@Service
@RequiredArgsConstructor
public class OrderManagement {

  private final ApplicationEventPublisher events;
  private final OrderInternal dependency;

  @Transactional
  public void complete(Order order) {
    // State transition on the order aggregate go here
    events.publishEvent(new OrderCompleted(order.getId()));
  }
}

@Component
class InventoryManagement {

  @Async
  @Transactional(propagation = Propagation.REQUIRES_NEW)
  @TransactionalEventListener
  void on(OrderCompleted event) { /* … */ }
}

Side-by-side summary

Criterion Polling publisher Debezium router (relational) Debezium MongoDB router Spring Modulith events
Atomic with business change Yes, one transaction Yes, one transaction Yes, needs multi-document transaction Within one application only
Ordering You implement it Follows captured changes Follows captured changes Not an outbox concern
Delivery guarantee At-least-once Consumer must deduplicate Consumer must deduplicate Listener failure can lose publication
Operational footprint Your own relay Connector and its platform Connector and its platform None beyond the application
Database fit Any transactional database Relational outbox table MongoDB 4.0 or later Application-level
Outbox growth Delete after delivery Table still receives an insert per write Collection still receives an insert per write Not applicable

The ordering cell for the Debezium options reflects that the outbox row is captured from the database change stream. The Debezium pages shown here do not give a formal ordering guarantee, so verify ordering for your connector and topic partitioning.

Duplicates and rollbacks apply to every option

No relay removes the need for idempotent consumers. When the relay delivers an event and then hits a failure or timeout, it retries, so the broker can receive the event more than once. Delivery is therefore at-least-once.

Track processed messages in the consuming service. The Debezium event id header gives you a stable deduplication key. Deduplication is also good practice without an outbox, because a consumer that fails mid-message sees the same message again after a restart.

Two more rules from the pattern guidance:

  • Do not send an event notification when the transaction is rolled back. The outbox insert guarantees this because it rolls back with the state change.
  • If one business operation spans several services that each update a data store, use the saga orchestration pattern. An outbox protects one service's database, not a cross-service transaction.

What we compared

  • Atomicity: the event must commit or roll back together with the business change.
  • Ordering: events must leave in the order the service updated the database.
  • Delivery guarantee: at-least-once delivery requires idempotent consumers, so check what the option gives you for deduplication.
  • Operational footprint: code you write and run yourself versus a connector platform you operate.
  • Database fit: relational outbox table, MongoDB outbox collection with multi-document transactions, or in-process only.
  • Outbox growth and write load: every business write adds an outbox insert, so plan for the traffic.

Recommendation

Pick by situation. **Relational database and Kafka already in place:** use Debezium with the outbox event router SMT. You avoid writing and operating a relay, you get routing by aggregate, and the event ID header supports deduplication. Use the default outbox table columns so you need almost no mapping. **Small system, no connector platform, any transactional database:** use a polling publisher inside the service or as a small separate application. Keep the same table layout as the Debezium default so you can switch to CDC later. Budget time for ordering, retry and deletion logic, which you own. **MongoDB services:** use the Debezium MongoDB outbox event router, and only after you confirm the deployment supports multi-document transactions (MongoDB 4.0 or later). Do not use the relational SMT with the MongoDB connector. **Module-to-module events inside one Spring application:** use Spring Modulith application events and `@ApplicationModuleListener` for decoupling. Treat it as an outbox only after you confirm in the Spring Modulith documentation how event publications are persisted and delivered. A plain `@Async` `@TransactionalEventListener` loses the event if the listener fails.

Caveats

  • The Debezium pages shown here do not state a formal ordering guarantee for routed events. Verify ordering for your connector, key and topic partitioning.
  • The router SMT configuration option names are not reproduced in this article. Copy them from the official Debezium pages for the version you run.
  • A connector with the outbox SMT should capture changes in the outbox table only, and several outbox tables are allowed only with identical structure.
  • The MongoDB planned support for sub-document outbox events is a stated plan, not a feature you can use.
  • Spring Modulith's events page, as used here, does not describe delivery to an external broker. Check the page for persistence and externalization details.
  • The outbox table can see an insert for every write. Plan retention or deletion so the table does not become a hot spot.
  • An outbox does not coordinate transactions across services. Use saga orchestration for that.

Frequently asked questions

Does the transactional outbox give exactly-once delivery?

No. The relay can retry after a failure or timeout, so delivery to the broker is at-least-once. Make consumers idempotent, for example by tracking the event ID.

Do I need Debezium to implement the outbox pattern?

No. A separate process, such as a thread in the service or a standalone application, can poll the outbox table, send events and remove delivered rows. CDC tools and Kafka connectors are an alternative that avoids writing that process.

Can I use the relational outbox event router with MongoDB?

No. The relational SMT is not compatible with the MongoDB connector. Use the MongoDB outbox event router, and write business and outbox documents in one multi-document transaction (supported since MongoDB 4.0).

Which columns does the Debezium outbox table need by default?

id (uuid), aggregatetype, aggregateid and type (each varchar 255, not null) and payload (jsonb). You can change which column supplies a value through the SMT options listed on the official page.

Why not publish from a transactional event listener after commit?

If the listener fails, or the system fails before the method is invoked, the event publication is lost unless each listener has its own safety net. An outbox row written in the original transaction survives such failures.

What happens to the outbox table size?

The outbox can receive an insert for every database write, which can mean a lot of traffic. In a polling design you remove rows after delivery, so plan a retention or cleanup policy for the approach you choose.

References

From Grokem: https://www.grokem.com/programming/software-architecture/event-driven-architecture/transactional-outbox-pattern-polling-vs-debezium-cdc-for-microservice-events

Finished this article?

How did this strike you?

Be the first to react – it is anonymous.

Community

Questions about this?

Ask in Questions and help and get help from other readers.

Ask a questionBrowse Questions and help

Comments

No comments yet. Share what worked for you, or ask a question.

Keep learning

Picked for this article

Related reading, discussions and learning paths on the same subject.

All in Event-Driven Architecture

From the community

Learning paths

Learning pathIntermediate

SQL for data work

From removing duplicates and window functions to modelling for reporting, cohort analysis, indexes and query plans: the SQL a working analyst or engineer uses every week, each lesson with queries that were run.

8 lessonsabout 90 minStart

Learning pathBeginner

Measurement that stands up to questions

Plan what to measure, tag campaigns consistently, hand data to your tags properly, find out why events go missing, understand attribution and know what privacy law asks of analytics.

6 lessonsStart

Choose what we may use. You can change this at any time with “Privacy choices” at the foot of every page. Details are in the Cookie policy and the Privacy policy.

Strictly necessary

Remembers your theme, the lessons you marked as done and this choice. It stays on your device, tracks nothing and cannot be switched off.

Always on
What exactly is collected, and why

Google Analytics

Collects
The pages you view, your device and browser type, your language, how you reached the site and your approximate location (country and city, worked out from your IP address).
Why
To understand which pages and lessons are used, so we can improve them.
Stored
Two cookies, _ga and _ga_<id>, for up to 2 years. Google acts for us; the data is processed by Google, including outside your country.

Grokem’s own counts

Collects
The pages you open and how long you stay, how far you scroll, where you came from, your device type, browser, screen size, language and time zone. Your IP address is turned into a code that changes every day; it is never stored as it is.
Why
To see how long lessons are read and where readers leave. Not shared with anyone.
Stored
A random visitor ID, a visit ID and the time of your last page view in your browser’s storage (gr_vid, gr_sid, gr_last). No name or email is attached.