Solution Pattern: Using Change Data Capture for Stack Modernization

This solution pattern brings an architectural solution for scenarios where services integration must happen through data integration and cause no impact to the existing stack.

The architecture demonstrates how Change Data Capture (CDC) design pattern and event-driven architectures supports the extension of existing capabilities with no changes to legacy apps where new features can be delivered by cloud-native microservices and can deliver with zero impact new search capabilities through the integration of legacy and new services with specialized search index services.

Contributors: Bernard Tison (Red Hat), Karina Varela

1. Use cases

Common use cases that can be addressed with this architecture are:

  • Legacy services need to be extended or enhanced but, it is not a possibility to implement the changes in the existing application;

  • New applications or services must leverage existing data from an existing stack of services;

  • The organization seeks to move towards cloud environments and comply to best practices on cloud-native architectures but must also guarantee that any production-active legacy system data is also synchronized to the new services;

  • The organization seeks to move towards cloud environments and comply to best practices on cloud-native architectures but must also guarantee that any production-active legacy system data is also synchronized to the new services;

  • Legacy applications are now extended and complemented by capabilities delivered by new microservices or services (e.g. search index, cache) but the data should always be synchronized.

2. The story behind this solution pattern

The solution pattern is backed by the story of a retail store that has been active in the market for more than fifteen years. This retail store uses a single application to manage the inventory, catalog and sales during their daily operations.

We are a leading retail store acting in the market for over 15 years. We managed to start digital technologies adoption many years ago by acquiring a solution for our inventory and sales registrations. Our biggest challenge is to join competition with high-tech competitors.
— Alex Tizon - CEO
About company current situation

Here are some characteristics of the existing technology stack:

  • The technology has been acquired many years ago, and no source code is available.

  • The application, application server and databases are installed and running in an on premise environment.

  • There are domain experts who know how to keep the inventory and catalog list updated.

  • There are operations running 24/7 using the sales application, stopping the application impacts the money income directly.

It’s getting hard to compete with more high-tech companies, but the technical team does not have the required skills to boostrap the adoption of cloud-native technologies to enhance the organization online services.

Our first step into modernization is developing a new cashback system for our customers. Adding to that, our current service lacks good searching capabilities so we need better ways for customers to find what they need.
— Avery Smith- CTO
On modernization goals

3. Technical overview

This solution pattern builds on top of an event-driven architecture in order to support the extension of the legacy stack. The architecture includes new microservices, event streaming, event processing and search indexing tools.

In respect to the story goals and targeted use cases, it’s recommended to consider adopting an Enterprise Integration Pattern for data integration, more specifically, adopting the Change Data Capture (CDC) pattern.

This solution requires no source code changes in the existing services. The core concept builds on data integration between legacy and new services through usage of asynchronous events. A short description of this solution key concept is:

Relevant changes to data persisted in the tracked databases (e.g. delete/insert/update) are captured and published as events. Then, external services react to events and execute necessary operations.

The integration happens like this:

  1. Using Debezium, the database becomes an event stream. Since data changes are directly tracked, the legacy application code won’t require changes.

  2. The captured data changes are pushed to topics in a Kafka broker.

  3. The services that offers that extra capabilities can then subscribe to relevant topics and use the events to obtain the information needed to execute its logic.

For detailed architecture diagrams please check the In Depth Architecture section.

See below a simplified representation of the solution:

simplified tech usage
Figure 1. Simplified representation of the integration between the legacy application and the new technology stack.