Java Microservices - Centralized Logging (ELK Stack)

Quiz

Introduction

As microservices become the norm for building large-scale, distributed applications, operational challenges increase-especially in monitoring and troubleshooting. Each microservice may run on separate hosts, containers, or clusters, generating logs in different formats and locations.

Centralized Logging is a critical design pattern in microservices architecture. It allows the aggregation of logs from all services into a single searchable system, enabling faster diagnostics, alerting, and auditing.

One of the most popular solutions for centralized logging is the ELK Stack, which stands for Elasticsearch, Logstash, and Kibana.

Why Centralized Logging?

Logging in Monolith vs Microservices

Sr.No.	Feature	Monolith	Microservices
1	Log Location	Single location	Multiple services, containers, and hosts
2	Troubleshooting	Easier (single log file)	Harder (correlating across services)
3	Log Format	Uniform	Varies across services
4	Access	Simple	Complex in distributed environments

Problems Without Centralized Logging

Logs are scattered across nodes and services.
Difficult to trace a request end-to-end.
Inconsistent logging formats.
No support for full-text search or visualization.
Troubleshooting becomes time-consuming and error-prone.

What is the ELK Stack?

The ELK Stack is an open-source collection of tools designed to collect, analyze, and visualize logs in real time.

Sr.No.	Component	Role
1	Elasticsearch	Distributed search and analytics engine
2	Logstash	Data processing pipeline for log ingestion
3	Kibana	Visualization tool for dashboards and queries

The ELK Stack is often extended with Beats (e.g., Filebeat) for lightweight data shipping.

ELK Stack Architecture in Microservices

Architecture Overview

Filebeat reads logs from microservices.
Logstash parses, filters, and ships logs.
Elasticsearch indexes and stores logs.
Kibana lets you search and visualize log data.

Benefits of Centralized Logging

Sr.No.	Benefit	Description
1	Single View of All Logs	Unified access to logs from all microservices
2	Faster Troubleshooting	Trace errors across services using filters/search
3	Enhanced Security	Logs are protected centrally instead of scattered files
4	Analytics & Dashboards	Kibana enables real-time metrics and visualizations
5	Auditing and Compliance	Historical logs can be retained and searched
6	Scalability	Elasticsearch handles high-volume log data

Integrating Spring Boot with ELK

Configure Log Output Format (JSON)

Use logstash-logback-encoder −

Maven Dependency

<dependency>
   <groupId>net.logstash.logback</groupId>
   <artifactId>logstash-logback-encoder</artifactId>
   <version>7.4</version>
</dependency>

logback-spring.xml

<configuration>
   <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
      <destination>localhost:5000</destination>
      <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
   </appender>

   <root level="INFO">
      <appender-ref ref="LOGSTASH" />
   </root>
</configuration>

This sends structured JSON logs to Logstash via TCP.

Setting Up the ELK Stack

Install via Docker Compose

version: '3.7'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.10.0
    environment:
      - discovery.type=single-node
    ports:
      - "9200:9200"

  logstash:
    image: docker.elastic.co/logstash/logstash:8.10.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5000:5000"

  kibana:
    image: docker.elastic.co/kibana/kibana:8.10.0
    ports:
      - "5601:5601"

Sample Logstash Configuration (logstash.conf)

input {
  tcp {
    port => 5000
    codec => json
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "microservices-logs"
  }
}

Viewing Logs in Kibana

Access Kibana at http://localhost:5601
Go to "Discover" → Select microservices-logs index
Use filters like−
- level: ERROR
- serviceName: order-service
- @timestamp > now-1h

You can also create −

Real-time dashboards
Alerts for errors
Graphs for log frequency over time

Correlating Logs with Trace IDs

To trace a request across services−

Use a correlation ID or trace ID
Pass it via HTTP headers or message brokers
Include it in every log entry

Best Practices

Sr.No.	Best Practice	Reason
1	Use structured JSON logging	Easier parsing and searching
2	Implement correlation IDs	Trace requests across services
3	Use log levels wisely	Avoid flooding Elasticsearch with DEBUG logs
4	Configure log retention policies	Save storage and meet compliance
5	Secure access to Kibana	Prevent unauthorized data exposure
6	Monitor Elasticsearch health	Avoid index overloads

Alternatives to ELK Stack

Sr.No.	Tool	Description
1	EFK Stack	ELK + Fluentd (instead of Logstash)
2	OpenSearch	Fork of Elasticsearch/Kibana maintained by AWS
3	Datadog, Splunk	Paid observability platforms
4	Grafana Loki	Lightweight logging solution, integrates with Prometheus

Real-World Use Cases

Sr.No.	Company	Use of Centralized Logging
1	Netflix	Observability of thousands of microservices
2	Airbnb	Analyses structured logs to detect production issues
3	Uber	Tracks end-to-end request latency with centralized logs
4	LinkedIn	Uses structured logging for internal debugging

Challenges and Limitations

Sr.No.	Challenge	Solution
1	High storage usage	Implement log rotation and TTL
2	Parsing unstructured logs	Use Logstash filters or structured formats
3	Elasticsearch overload	Use ILM (Index Lifecycle Management)
4	Complex configuration	Use managed services (e.g., AWS OpenSearch)

Conclusion

Centralized Logging with ELK Stack is essential for managing modern, distributed microservices systems. It brings together logs from all services into one place, enabling −

Real-time monitoring
Faster root-cause analysis
Improved security and auditing
Scalable observability

By integrating Spring Boot with Logstash, and visualizing logs in Kibana, teams gain a powerful toolkit to ensure operational excellence in microservices architectures.

Print Page