How to Deploy Elasticsearch on Kubernetes?

In today's world, data is the lifeblood of businesses. Whether it is customer data, sales data, or product data, the ability to store, search, and analyze it effectively can make a significant difference in a company's success. Elasticsearch is a distributed search and analytics engine designed for speed and scalability.

It can process millions of queries per second and handle petabytes of data with ease. Kubernetes, on the other hand, is an open-source container orchestration system that automates application deployment, scaling, and management.

Setting up the Environment

Installing the Necessary Tools (kubectl, helm)

Before deploying Elasticsearch on Kubernetes, you will need to have kubectl and helm installed on your local machine. Kubectl is a command line interface (CLI) tool used to communicate with the Kubernetes cluster and manage resources such as deployments, services, and pods.

Helm is a package manager for Kubernetes that simplifies the process of installing applications by providing pre-configured charts.

Configuring the Kubernetes Cluster for Elasticsearch Deployment

Once kubectl and helm are installed on your local machine, you will need to configure your Kubernetes cluster for Elasticsearch deployment. This involves creating a namespace for Elasticsearch resources and setting up persistent storage. To create a namespace using kubectl −

kubectl create namespace elasticsearch  

This will create a new namespace named `elasticsearch`.

Next step is configuring persistent storage which allows data to be stored outside of an individual pod so that it remains available even if the pod fails or restarts due to maintenance or scaling needs. You should install an external storage provider such as NFS Storage Class provisioner or Cloud provider’s block storage class which offers provisioned block volumes.

Creating an Elasticsearch Deployment

Creating a YAML file for deployment configuration

Before deploying Elasticsearch on Kubernetes, you will need to create a YAML file that defines the configuration of your Elasticsearch deployment. This YAML file is used by Kubernetes to understand how you want to deploy ElasticSearch.

Here's an example of what your YAML file might look like:

apiVersion: apps/v1 
kind: Deployment metadata: 
name: elasticsearch spec: 
selector: matchLabels: 
app: elasticsearch replicas: 1 
template: metadata: 
labels: app: elasticsearch 
spec: containers: 
- name: elasticsearch image: docker.elastic.co/elasticsearch/elasticsearch:{VERSION} 
ports: - containerPort: 9200 
name: http - containerPort: 9300 
name: tcp  

This YAML file contains the specification for a single replica of an Elasticsearch Container running in a Kubernetes Pod.

Defining resources, replicas, and other settings in the YAML file

In addition to defining the replica count and image tag for your Elasticsearch deployment, you can also specify various resource requirements such as CPU and memory limits. By default, Kubernetes will assign default resource requests and limits for each Container based on its known characteristics. To define custom resources limits or requirements for your ElasticSearch deployment add following lines in container spec−

resources: requests : 
memory : "8Gi" cpu : "2" 
limits : memory : "16Gi" 
cpu : "4"  

You can also configure various other settings such as environment variables, command line arguments, liveness probes etc.

Deploying the Elasticsearch instance using kubectl apply command

Once you have created the YAML file, you can deploy it to your Kubernetes cluster using kubectl apply command−

kubectl apply -f elasticsearch.yaml  

This command will create a new deployment with the configuration specified in the YAML file.

You can view the status of your Elasticsearch deployment by running `kubectl get deployments` and `kubectl get pods`. Congratulations!

You have successfully created and deployed an Elasticsearch instance on Kubernetes. The next step is to configure Elasticsearch settings such as node, index patterns, mappings etc.

Configuring Elasticsearch

Setting up index patterns and mappings

Once you have successfully deployed your ElasticSearch cluster on Kubernetes, the next step is to configure it to fit your data needs. One important aspect of this configuration is setting up index patterns and mappings. An index pattern is a way to define how your data will be organized in ElasticSearch, while mapping specifies the fields and their data types that will be stored in each index.

To create an index pattern, you can use Kibana or the ElasticSearch REST API. Kibana makes it easier for people who are not familiar with the API or programming to configure their ElasticSearch instance.

To create an index pattern in Kibana, go to the management tab and select "Index Patterns". Then click on "Create Index Pattern" and enter a name for your pattern.

Configuring node settings for optimal performance

Elasticsearch nodes are responsible for indexing and storing documents as well as searching through them when a query is executed. To ensure optimal performance of your Elasticsearch deployment on Kubernetes, it's important to configure node settings correctly. One important consideration when configuring nodes is memory allocation.

By default Elasticsearch allocates 1GB of memory per node which may not be enough if you are working with large datasets. To adjust this setting simply edit the jvm.options file on each node and change -Xms1g and -Xmx1g to higher values.

Another consideration when configuring nodes is the number of shards per index. Elasticsearch indexes can be split into multiple shards for better performance.

Deploying Kibana on Kubernetes

Installing Kibana via Helm chart

Kibana is an open-source data visualization and exploration tool that can be used to visualize Elasticsearch data. It is often deployed alongside Elasticsearch to provide a user-friendly interface for interacting with the data. In order to deploy Kibana on Kubernetes, we will use the Helm package manager.

Helm is a powerful tool that allows us to deploy pre-configured packages (called charts) onto our Kubernetes cluster. In order to install Kibana using Helm, we first need to add the Elastic repository to our Helm configuration −

bash helm repo add elastic https://helm.elastic.co    

Once the Elastic repository has been added, we can install the Kibana chart −

bash helm install kibana elastic/kibana  

This will deploy a single instance of Kibana onto our Kubernetes cluster, which can be accessed via a web browser at http://:5601.

Configuring Kibana to connect to deployed ElasticSearch Cluster

In order for Kibana to interact with Elasticsearch, it needs to know how to connect to it. By default, the Helm chart will configure Kibana with some sensible defaults that should work out-of-the-box. However, if you need more control over how Kibana connects to Elasticsearch (for example if you have configured your Elasticsearch deployment for high availability), you may need to edit some configuration files.

The easiest way to do this is by using ConfigMaps in Kubernetes. First, we'll create a new ConfigMap called `kibana.yml` −

bash kubectl create configmap kibana-yml --from-file=kibana.yml    

Next, we'll tell the Elastic chart that we want it to use our custom `kibana.yml` file for configuration −

bash helm upgrade kibana elastic/kibana --set configMap=kibana-yml  

At this point, Kibana should be configured to connect to our Elasticsearch deployment.

Conclusion

In this article, we have discussed the importance of deploying Elasticsearch on Kubernetes and provided a detailed guide on how to do so. We have covered the prerequisites for deploying Elasticsearch on Kubernetes, set up the environment, created an Elasticsearch deployment with kubectl apply command, configured Elasticsearch for optimal performance settings and deployed Kibana on Kubernetes using Helm chart. Additionally, we have discussed how to monitor ElasticSearch cluster using Prometheus.

Deploying Elasticsearch on Kubernetes can bring many benefits to organizations such as scalability, automated management and increased efficiency. By following the steps outlined in this article, you can successfully deploy your own ElasticSearch cluster on Kubernetes.