Log Aggregation

Problem Statement

Microservice architecture structures an application as a set of loosely coupled microservices and each service can be developed independently in agile manner to enable continous delivery/deployment. Requests often span multiple services. Each service instance write some information in its log file in a standardized format. These logs can be info, error, warning or debug logs. How to analyze and troubleshoot application problems using these logs.

Solution

We can use a centralized logging service which aggregates the logs from each service. User should be able to search and analyze the logs provided by this logging service. User should be able to configure alerts when certain type of messages appear in logs.

Corelation ID

When first microservice receives a call, it should generate a corelation id which then can be passed to downstream services. This corelation id should be logged across all microservices. It will help to track the information spanning multiple services.

Searchable Logs

As logs should be placed at a centralized location, following diagram showcase how to use Kafka, LogStash and Kibana to aggregate logs and search the indexed logs using required filters.

Microservices generates logs, which are published using kafka log appender which then output the log messages to kafka cluster. LogStash ingests the messages from kafka, transforms the messages and publish to elastic search container. Now kibana provides a visual interface to search/read indexed logs from elastic search container and provides required filters.