Architecture of Event-Driven Microservices

Quiz

We can send and receive event messages using Event-Driven Microservices (EDM). Producers generate and publish events to notify other services when actions and changes occur in the system. At the same time, other services consume these events to take relevant actions to updates in real-time.

The main advantages of EDM systems are their asynchronous communication and loosely coupled design. Services subscribe to and react to events as these occur rather than waiting to request data at the time of need. EDM is loose coupling to keep independence between microservices. So, we can operate and scale any services individually without impacting each other.

EDM has loosely coupled architecture with event-driven basics. EDM can address the limitations of traditional approaches. We will discuss these solutions in this chapter.

REST-Driven Approach

We call components one another and wait for responses in the REST-based synchronous system −

But there are some problems with this approach that include the following −

Dependence on Responses − If module 1 waits for a response from Module 2. Then it can have delays if Module 2 is slow and unavailable.
Service Downtime − If a service goes down. Then its dependent services are disrupted.
Network Latency − Data transfer can have delays for large payloads requiring pagination.
Heavy Load Impact − Services under heavy load may respond slowly. So it reduces performance.

We use EDM solutions when systems become less efficient due to these synchronous connections.

Event-Driven Microservices Example

Suppose we have a reporting application GIB API used to send sales reports to the government. This application fetches all sales data from another API. In the beginning, with a low volume of transactions, we built the API using a simple REST approach.

But when the data grows, the volume of transactions increases unexpectedly. The REST approach has a problem, so we need an event-driven solution.

Solution 1: Converting to Event Messaging

We apply the event-driven approach for efficient report generation. This plan is easy as given below −

Generate events for new transactions − An event is published whenever a new transaction item is created.
Asynchronous data fetching − Related data is fetched and transformed into report content.
Storing and retrieving reports − Report data is stored in a relational database (PostgreSQL). Report strings are concatenated and saved as files for later access.

Gathering DATA via Async Event Messaging

Through this asynchronous event messaging. The reporting application can process data without the need for constant direct REST calls.

There is another performance issue after converting the process to an asynchronous model. The reporting API is still overloading the transaction API because it requests transaction details for each item.

Solution 2: Fat Event

We can use the fat event approach to resolve the above issue. Each event message has all relevant transaction details. We embed the data directly within each event. So it eliminates the need for additional REST calls. It creates a fully asynchronous EDM architecture.

{
   "identifier": 2,
   "detail": "sale transaction item",
   "created_date": 4587997988425,
   "last_modified_date": 1845298016540
}

Complete Event-Driven Architecture

The entire system is more responsive with fat events. We have all necessary information within events. Each service can operate independently without depending on other services in real-time.

Solution 3: Outbox Pattern

We consider the data processed as financial transactions. Keeping accuracy is important. This required a reliable method to ensure all events are delivered and processed without any data loss. Here, we applied the outbox pattern.

What is the Outbox Pattern?

We first saved event messages in a database table in the outbox pattern, instead of publishing event messages directly. A job then batches these events and sends these periodically.

Outbox Pattern Workflow

Event Generation − The business module publishes an event.
Persisting Events − The event service stores the event message in a relational database (RDBMS).
Triggering the Job − A scheduler initiates the Send Event Messages job.
Publishing Messages − The event service retrieves the stored events and publishes these using message brokers like RabbitMQ.

Advantages of the Outbox Pattern

Persistent Events − It has ACID properties because of events saved in a relational database, so data is persistence.
Recovery of Lost Events − If an event fails to publish. It can be resent from the database.

Disadvantages of the Outbox Pattern

Increased Complexity − Since it adds another layer to manage within the architecture.
Delayed Publishing − Events may be published at intervals rather than in real time.

Benefits of EDM Architecture

Given below are some of the benefits of using the EDM architecture −

Loosely Coupled Structure − Each microservice is independent, so it is flexible and modular.
Complete Isolation − Each microservice can function without synchronous calls to other services.
Asynchronous Processing − It improves overall system responsiveness and efficiency.
Performance Gain − Services can react to events as these occur.

The main advantage of EDM is loose coupling. It preserves the independence of microservices.

EDM has a Single Point of Failure

If you use a message broker like RabbitMQ in EDM, it can have a point of failure.

But you can solve this using one of these techniques −

Cluster Configuration − You can set up RabbitMQ as a cluster to prevent single points of failure.
Durable Queues − You can configure queues to persist.
Message Persistence − You can store messages for later retrieval, even in case of a temporary broker outage.

Other instances in the cluster can take over, if an error occurs. So the persisted messages can be recovered.

Duplicate Event Messages

An event may be published more than once, sometimes. But you can use the following technique to prevent issues from duplicate messages −

Idempotent Consumers − Consumers should be used to handle duplicate events. It should first check if these have already processed an event before taking action.

In microservice architecture, you need to maintain loose coupling for effective communication. You can use event-driven (EDM) approaches to reduce dependency on direct REST calls. EDM avoids the issues of synchronous communication, like waiting for responses and handling service outages.

We set up a RabbitMQ cluster with persisted messages. Services can publish events without requiring immediate acknowledgment. Fat events have all needed data to improve efficiency. The outbox pattern has reliable event delivery, even in cases of network issues and temporary outages.

Conclusion

EDM is used to keep microservices independent and loosely coupled. You can handle real-time data in EDM with efficiency and scalability. But, if microservices become interdependent. Then the architecture risks transforming into a distributed monolith. So you should keep loose coupling using an event-driven design.

Print Page