Traditional systems often enforce strict transactional consistency, which can impede application performance and compromise business logic. For instance, upon the completion of a ride-sharing service at the end-user’s destination, the system should reliably capture this real-world event. The capture should be performed regardless of any potential operational disruptions affecting dependent services (e.g., invoicing).

In such scenarios, transactional applications, which typically encompass a number of steps in a user workflow, make all the steps undone when an error is returned. This prevents any data from being changed, and causes the loss of real-world intent of end-users, and results in an adverse user experience and a deviation from the genuine business process, potentially leading to customer dissatisfaction.

The concept of employing the Event Mesh as a central, reliable hub for dispatching commands, as events, lies at the heart of this solution. This approach aligns closely with the Command Query Responsibility Segregation (CQRS) pattern, which distinctly categorizes commands and queries. Commands, in this context, are modeled as asynchronous events, designed to undergo processing by the Event Mesh. On the other hand, queries are synchronous operations, safe to retry, ensuring no loss of data integrity due to transient errors.

The primary responsibility of the Event Mesh is twofold. Firstly, it persists the incoming events, thereby maintaining a record of changes in the system’s state. Secondly, it routes these events to their respective endpoints, ensuring that the appropriate microservices are notified and can subsequently update their internal states based on the event data.

The mesh’s inherent resilience is further bolstered by its built-in retry strategies (linear or exponential backoff), which it employs when encountering operational failures. This mechanism ensures that the system retries the operation until it succeeds, thus mitigating the risk of data loss or system disruption due to transient issues.

By integrating the Event Mesh into the system architecture, several architectural benefits are achieved:

The event-driven flow enables eventual consistent collaboration and state synchronization between services, fostering a resilient, scalable, and developer-friendly system architecture.

A usual flow may look like:

The diagram illustrates the example flow of events between the applications, the Knative Event Mesh, and the datastores which persist settled state of the system.

A good way of thinking about the Event Mesh and its persistent queue backend is the Work Ledger analogy. Like in the olden days, the clerk kept his to-do work in the Work Ledger (e.g. a tray for paper forms). Then he was picking the next form, and processing it, making changes within the physical file cabinets. In case of rare and unexpected issues (e.g. invoicing dept being unavailable), the clerk would just put the data back onto the Work Ledger to be processed later.

The Event Mesh is processing the data in very similar fashion. The data is held in the Event Mesh only until the system successfully consumes it.

The Event Mesh pattern could be mistaken for Event Sourcing, as both are Event-Driven approaches (EDA) to application architecture. However, Event Mesh has few improvements over the shortcomings of Event Sourcing approach.

The data is held in the Event Mesh only until the system successfully consumes it, settling the data in various datastores to a consistent state. This effectively avoids the need to keep the applications backward compatible with all the events ever emitted. Introducing breaking changes in the event schema is as easy as making sure to consume all the events of given type from the Event Mesh. This also works for the systems which can’t have downtime windows. The applications could have short-lived backward compatible layers for such situations. When all the events are processed, the backward compatible code may be removed simplifying the maintenance.

Because in the long-term, the regular datastores are the source of truth for the system, all traditional techniques for application maintenance apply well. It is also, easier to understand for developers, as it avoids sophisticated event handlers logic, and reconciling into the read database abstraction.

Worth pointing out are the differences from the Service Mesh pattern. The Service Mesh pattern is intended to improve the resilience of synchronous communications which return the response. The Service Mesh effectively raises the uptime of the dependent endpoints by retrying and backoff policies. The uptime can’t be raised to 100%, though, so Service Mesh still can lose the messages.

The table below captures the key differences:

One of the strengths of an Event Mesh architecture is its ability to integrate seamlessly with legacy systems, making them more resilient and adaptable. Legacy applications can be retrofitted to produce and consume events through lightweight adapters. For instance:

Traditional systems often rely on synchronous calls and transactions, which can cascade failures across components. Replacing these with asynchronous event-driven communication reduces dependencies and makes the system Eventually Consistent.

For example, invoicing and notification services in a ride-sharing platform can process events independently, ensuring that downtime in one service does not block the entire workflow.

Retry mechanisms provided by the Event Mesh guarantee that transient failures are handled gracefully without data loss.