ELK Stack Tutorial: Get Started with Elasticsearch, Logstash, Kibana, & Beats

The ELK Stack is a powerful collection of open-source tools for searching, analyzing, and visualizing log data in real time. Originally consisting of Elasticsearch, Logstash, and Kibana, the stack has evolved to include Beats, forming what's now called the Elastic Stack. This comprehensive solution enables organizations to centralize logging, monitor systems, and gain insights from massive datasets across healthcare, finance, IT, and other industries.

Components of ELK Stack

• Elasticsearch A distributed search and analytics engine that stores, indexes, and searches data. It provides fast, scalable full-text search capabilities and serves as the central data store.

• Logstash A data processing pipeline that ingests data from multiple sources, transforms it through filters, and outputs it to destinations like Elasticsearch. It handles parsing, enrichment, and normalization.

• Kibana A web-based visualization platform that creates interactive dashboards, charts, and reports from Elasticsearch data. It provides a user-friendly interface for data exploration and analysis.

• Beats Lightweight data shippers that collect specific types of operational data from servers and send it to Elasticsearch or Logstash. Examples include Filebeat (log files), Metricbeat (system metrics), and Packetbeat (network traffic).

Features

• Near Real-Time (NRT) Search Data becomes searchable within seconds of indexing

• Full-Text Search Advanced search capabilities with relevance scoring

• JSON Document Store Schema-free document storage with automatic field mapping

• Geolocation Support Built-in geographic data analysis and visualization

• REST API Interface HTTP-based API for easy integration with applications

• Multi-Language Support Text analysis in multiple languages

Installation Prerequisites

Before installing ELK Stack on Ubuntu, ensure your system meets these requirements:

System Updates

sudo apt update

Java Installation

Elasticsearch and Logstash require Java 8 or Java 11:

sudo apt install default-jre

Nginx Installation

Nginx serves as a reverse proxy for Kibana:

sudo apt install nginx -y

Installing Elasticsearch

Add Repository

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Install and Configure

sudo apt-get update && sudo apt-get install elasticsearch

Edit the configuration file:

sudo nano /etc/elasticsearch/elasticsearch.yml

Configure network settings:

network.host: localhost
discovery.type: single-node

Set JVM heap size:

sudo nano /etc/elasticsearch/jvm.options

-Xms512m
-Xmx512m

Start Service

sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch
sudo systemctl status elasticsearch

Verify Installation

curl -X GET "localhost:9200"

Installing Kibana

sudo apt install kibana

Configure Kibana:

sudo nano /etc/kibana/kibana.yml

server.port: 5601
server.host: "localhost"
elasticsearch.hosts: ["http://localhost:9200"]

Start the service:

sudo systemctl start kibana
sudo systemctl enable kibana
sudo ufw allow 5601/tcp

Installing Logstash

sudo apt install logstash
sudo systemctl start logstash
sudo systemctl enable logstash

Configuration files are located in /etc/logstash/conf.d/ for custom input, filter, and output pipelines.

Installing Beats (Metricbeat Example)

sudo apt install metricbeat

Configure Metricbeat:

sudo nano /etc/metricbeat/metricbeat.yml

setup.kibana:
  host: "localhost:5601"

output.elasticsearch:
  hosts: ["localhost:9200"]

Set up dashboards and start:

sudo metricbeat setup
sudo systemctl start metricbeat
sudo systemctl enable metricbeat

Comparison of ELK Stack Components

Common Use Cases

• Application Monitoring Track errors, performance metrics, and user behavior across distributed applications

• Security Analytics Detect threats, analyze attack patterns, and monitor compliance through log correlation

• Infrastructure Monitoring Monitor server health, resource utilization, and network performance

• Business Intelligence Analyze website traffic, user engagement, and e-commerce metrics for data-driven decisions

Advantages and Disadvantages

Conclusion

The ELK Stack provides a comprehensive solution for centralized logging, real-time monitoring, and data visualization. While it requires careful planning and resource allocation, its powerful search capabilities and flexible architecture make it invaluable for organizations needing to analyze large volumes of data and gain actionable insights from their systems and applications.