Data Structure
Networking
RDBMS
Operating System
Java
MS Excel
iOS
HTML
CSS
Android
Python
C Programming
C++
C#
MongoDB
MySQL
Javascript
PHP
Physics
Chemistry
Biology
Mathematics
English
Economics
Psychology
Social Studies
Fashion Studies
Legal Studies
- Selected Reading
- UPSC IAS Exams Notes
- Developer's Best Practices
- Questions and Answers
- Effective Resume Writing
- HR Interview Questions
- Computer Glossary
- Who is Who
How to use ansible for setting up production elasticsearch servers with cluster
In this article, we will learn how to use Anisble to configure and install production Elastic Search cluster on CentOS7 which ensures that the Elasticsearch nodes will be secure from outside network .we will use VPN services to connect to the clusters.
Elasticsearch is a very popular open source search, server which can be used for real-time distributed search and analysis of the data for better performance, stability and for scalability we need the Elasticsearch to be deployed across multiple servers as a cluster.
Prerequisites
We need a minimum of three CentOS 7 server environments with private networking since the Elasticsearch cluster needed is 3 master nodes and if we are planning for a master and a data node, then we need additional servers for the data nodes also. please make sure that we have unique Ansible inventory host names.
Assuming that we have already configured VPN server and able to connect each server using the VPN and the interface name is “tun0” or we can change the Elasticsearch listen on a different interface which we can make the appropriate changes in the site.yml file.
Download the Ansible-Elasticsearch Playbook
Elasticsearch provides an Ansible role which can be used to set up an Elasticsearch cluster, for that we needed to add it to our ansible-tinc playbook and define the host groups and assign them the appropriate roles to those groups.
Goto the ansible-tinc folder in the server
# cd ~/ansible-tinc
We needed to clone the ansible-elasticsearch roles which are in Elastic’s Github repository
# cd roles # git clone https://github.com/elastic/ansible-elasticsearch Initialized empty Git repository in /root/ansible-elasticsearch/.git/ remote: Counting objects: 1192, done. remote: Total 1192 (delta 0), reused 0 (delta 0), pack-reused 1192 Receiving objects: 100% (1192/1192), 167.42 KiB | 86 KiB/s, done. Resolving deltas: 100% (640/640), done. # mv ansible-elasticsearch elasticsearch
Updating the Master playbook in Ansible site.yml
We need to map all the three different Elasticsearch roles to three different Anisble host groups which will allow us to create the dedicated master, dedicated data and master Elasticsearch nodes.
Mapping Elasticsearch Dedicated Master Role of Group
$ cd ~/ansible-playbook
$ vi site.yml
- hosts: elasticsearch_master_nodes
roles:
- { role: elasticsearch, es_instance_name: "node1", es_config: { discovery.zen.ping.unicast.hosts: "node01, node02, node03", network.host: "_tun0_, _local_", cluster.name: "production", discovery.zen.ping.multicast.enabled: false, http.port: 9200, transport.tcp.port: 9300, node.data: false, node.master: true, bootstrap.mlockall: true } }
vars:
es_major_version: "2.x"
es_version: "2.2.1"
es_heap_size: "2g"
es_cluster_name: "production"This configuration will create dedicated master nodes roles in the group where the values node.master: true and node.data: false will defines the same.
Now the hosts which are configured in the elasticsearch_master_nodes are configured as dedicated master Elasticsearch nodes.
Mapping Elasticsearch Master/Data Role in Group
At the bottom of the master playbook site.yml, map the master-eligible and data elasticsearch role to the elasticsearch_master-data_nodes group by adding the below configuration.
- hosts: elasticsearch_master_data_nodes
roles:
- { role: elasticsearch, es_instance_name: "node1", es_config: { discovery.zen.ping.unicast.hosts: "node01, node02, node03", network.host: "_tun0_, _local_", cluster.name: "production", discovery.zen.ping.multicast.enabled: false, http.port: 9200, transport.tcp.port: 9300, node.data: true, node.master: true, bootstrap.mlockall: true } }
vars:
es_major_version: "2.x"
es_version: "2.2.1"
es_heap_size: "2g"
es_cluster_name: "production"This will create data nodes for master-eligible the values node.master: true and node.date: true will defines the same.
Set es_version to the same value which is configured for dedicated master roles.
Mapping Elasticsearch Dedicated Data Role to Group
We need to add the below configuration to master playbook site.yml for dedicated data elasticsearch role for the elasticsearch_data_nodes group.
- hosts: elasticsearch_data_nodes
roles:
- { role: elasticsearch, es_instance_name: "node1", es_config: { discovery.zen.ping.unicast.hosts: "node01, node02, node03", network.host: "_tun0_, _local_", cluster.name: "production", discovery.zen.ping.multicast.enabled: false, http.port: 9200, transport.tcp.port: 9300, node.data: true, node.master: false, bootstrap.mlockall: true } }
vars:
es_major_version: "2.x"
es_version: "2.2.1"
es_heap_size: "2g"
es_cluster_name: "production"This role will create dedicated data nodes as it is configured to the nodes with the value node.master − false and node.data: true.
Save the configuration files and exit
Updating the Host Inventory File
To create Elasticsearch nodes, simply add the host to the hosts files in Ansible inventory files
# vi hosts [vpn] node01 vpn_ip=10.0.0.1 ansible_host=10.2.2.1 node02 vpn_ip=10.0.0.2 ansible_host=192.168.100.100 node03 vpn_ip=10.0.0.3 ansible_host=192.168.100.101 node04 vpn_ip=10.0.0.4 ansible_host=192.168.100.102 [removevpn]
Now we have to add three groups which are mapped in the site.yml file
[elasticsearch_master_nodes] [elasticsearch_master_data_nodes] [elasticsearch_data_nodes]
The total hosts file in the Anisble inventory should look like this
[vpn] node01 vpn_ip=10.0.0.1 ansible_host=10.2.2.1 node02 vpn_ip=10.0.0.2 ansible_host=192.168.100.100 node03 vpn_ip=10.0.0.3 ansible_host=192.168.100.101 node04 vpn_ip=10.0.0.4 ansible_host=192.168.100.102 [removevpn] [elasticsearch_master_nodes] node01 node02 node03 [elasticsearch_master_data_nodes] [elasticsearch_data_nodes] node04
Creating Elasticseach Cluster
As we have already configured all the setting required for the Elasticsearch cluster servers, below is the command to create Elasticsearch cluster
# ansible-playbook site.yml
After the playbook run completly the elasticsearch cluster should be up and running.
Verifying the Elasticsearch Cluster Status
From any of the three servers we needed to run the below command
# curl -XGET 'http://localhost:9200/_cluster/state?pretty'
Output:
Cluster State:
{
"cluster_name" : "production",
"version" : 8,
"state_uuid" : "SgTyb0vNTTu2rdKPrc6tkQ",
"master_node" : "OzqNzte9RYWSXS6OkGhveA",
"blocks" : { },
"nodes" : {
"OzqMzte9RYWDXS6OkGhveA" : {
"name" : "node02-node1",
"transport_address" : "192.168.100.2:9300",
"attributes" : {
"data" : "false",
"master" : "true"
}
},
"7bohaaYVTee$HvSgBFp-2g" : {
"name" : "node04-node1",
"transport_address" : "192.168.100.4:9300",
"attributes" : {
"master" : "false"
}
},
"cBat9IgPQwKU_GPF8L3Y1g" : {
"name" : "node03-node1",
"transport_address" : "192.168.100..3:9300",
"attributes" : {
"master" : "false"
}
},
...If we are able to see the above output, then Elasticsearch cluster servers are working fine and running.
As we have completed the configuration and set up we can configure Elasticsearch cluster running in a healthy state, as Elasticsearch has many more features and configuration options which are not covered for more information for advance features you can refer the official documentation.
213 Views
