- Microservices Design Patterns Tutorial
- Microservices Design Patterns - Home
- Microservices Design Patterns - Overview
- Decomposition Design Patterns
- Decompose by Business Capability
- Decompose by Subdomain
- Decompose by Strangler
- Integration Design Patterns
- API Gateway
- Aggregator
- Proxy
- Client Side UI Composition
- Chain Of Responsibilities
- Branch
- Database Design Patterns
- Database per Service
- Shared Database per Service
- Command Query Responsibility Segregator
- Saga
- Aysynchronous Messaging
- Event Sourcing
- Observability Design Patterns
- Log Aggregation
- Performance Metrics
- Distributed Tracing
- Health Check
- Cross Cutting Concern Design Patterns
- External Configuration
- Service Discovery
- Circuit Breaker
- Blue Green Deployment
- Useful Resources
- Quick Guide
- Useful Resources
- Discussion
Log Aggregation
Problem Statement
Microservice architecture structures an application as a set of loosely coupled microservices and each service can be developed independently in agile manner to enable continous delivery/deployment. Requests often span multiple services. Each service instance write some information in its log file in a standardized format. These logs can be info, error, warning or debug logs. How to analyze and troubleshoot application problems using these logs.
Solution
We can use a centralized logging service which aggregates the logs from each service. User should be able to search and analyze the logs provided by this logging service. User should be able to configure alerts when certain type of messages appear in logs.
Corelation ID
When first microservice receives a call, it should generate a corelation id which then can be passed to downstream services. This corelation id should be logged across all microservices. It will help to track the information spanning multiple services.
Searchable Logs
As logs should be placed at a centralized location, following diagram showcase how to use Kafka, LogStash and Kibana to aggregate logs and search the indexed logs using required filters.
Microservices generates logs, which are published using kafka log appender which then output the log messages to kafka cluster. LogStash ingests the messages from kafka, transforms the messages and publish to elastic search container. Now kibana provides a visual interface to search/read indexed logs from elastic search container and provides required filters.
