Checking the Cluster Health in Cassandra

Introduction

Apache Cassandra is a highly-scalable, high-performance distributed database that is designed to handle large amounts of data across many commodity servers. As a result, it's important to keep an eye on the health of your Cassandra cluster to ensure that it's running smoothly and that there aren't any issues that could impact performance or availability. In this article, we'll go over the different ways to check the health of your Cassandra cluster and what to look for to identify potential issues.

Understanding Cassandra Cluster Health

Before diving into how to check the health of your Cassandra cluster, it's important to understand what exactly we mean by "cluster health." A healthy Cassandra cluster should have all nodes up and running, with no dropped or stale nodes. Additionally, there should be a relatively even distribution of data across all nodes, and there should be no issues with replication or compaction.

Using nodetool to check cluster health

One of the most common ways to check the health of a Cassandra cluster is by using the nodetool command-line tool. This tool is included with Cassandra and provides a variety of information about the cluster and its nodes. One of the most useful commands for checking cluster health is nodetool status. This command will show the status of all nodes in the cluster, including whether they are up or down, and how much data they are currently storing.

<div class="code-mirror  language-sql" contenteditable="plaintext-only" spellcheck="false" style="outline: none; overflow-wrap: break-word; overflow-y: auto; white-space: pre-wrap;">$ nodetool <span class="token keyword">status</span>
Datacenter: datacenter1
<span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span>
<span class="token keyword">Status</span><span class="token operator">=</span>Up<span class="token operator">/</span>Down
<span class="token operator">|</span><span class="token operator">/</span> State<span class="token operator">=</span>Normal<span class="token operator">/</span>Leaving<span class="token operator">/</span>Joining<span class="token operator">/</span>Moving
<span class="token comment">--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack</span>
UN  <span class="token number">10.0</span><span class="token number">.0</span><span class="token number">.1</span>  <span class="token number">1.21</span> GB    <span class="token number">256</span>          <span class="token number">67.6</span><span class="token operator">%</span>             <span class="token number">456789</span>abcdef01234567890abcdef0123  rack1
DN  <span class="token number">10.0</span><span class="token number">.0</span><span class="token number">.2</span>  <span class="token number">1.21</span> GB    <span class="token number">256</span>          <span class="token number">67.6</span><span class="token operator">%</span>             <span class="token number">456789</span>abcdef01234567890abcdef0123  rack1
</div>

The status command will show the status of all nodes in the cluster, including whether they are Up or Down. In the example above, the first node is Up and the second node is Down. Additionally, the command will show the load and tokens for each node, as well as the percentage of data that the node owns.

Another useful command provided by nodetool is nodetool ring. This command will show the token distribution for the cluster and can help identify if there is an uneven distribution of data among the nodes.

<div class="code-mirror  language-sql" contenteditable="plaintext-only" spellcheck="false" style="outline: none; overflow-wrap: break-word; overflow-y: auto; white-space: pre-wrap;">$ nodetool ring
Datacenter: datacenter1
<span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator">=</span>
Address         Rack        <span class="token keyword">Status</span> State   <span class="token keyword">Load</span>            Owns                Token                                      
<span class="token number">10.0</span><span class="token number">.0</span><span class="token number">.1</span>        rack1       Up    Normal  <span class="token number">1.21</span> GB          <span class="token number">67.6</span><span class="token operator">%</span>               <span class="token operator">-</span><span class="token number">9223372036854775808</span>                        
<span class="token number">10.0</span><span class="token number">.0</span><span class="token number">.2</span>        rack1       Up    Normal  <span class="token number">1.21</span> GB          <span class="token number">67.6</span><span class="token operator">%</span>               <span class="token operator">-</span><span class="token number">3074457345618258603</span>                        
<span class="token number">10.0</span><span class="token number">.0</span><span class="token number">.3</span>        rack1       Up    Normal  <span class="token number">1.21</span> GB          <span class="token number">67.6</span><span class="token operator">%</span>               <span class="token number">3074457345618258602</span>
</div>

Monitoring Replication and Compaction

In addition to monitoring the overall health of the cluster and individual nodes, it's also important to keep an eye on replication and compaction. Replication is important to ensure that data is being properly replicated across all nodes in the cluster, while compaction is important to keep the data on each node organized and efficient.

To monitor replication, you can use the nodetool command nodetool statusgossip which will show the current state of gossip and the replication factor for each keyspace.

<div class="code-mirror  language-sql" contenteditable="plaintext-only" spellcheck="false" style="outline: none; overflow-wrap: break-word; overflow-y: auto; white-space: pre-wrap;">$ nodetool statusgossip
Gossip active  : <span class="token boolean">true</span>
Thrift active  : <span class="token boolean">false</span>
Native Transport active: <span class="token boolean">true</span>
<span class="token keyword">Load</span>          : <span class="token number">1.21</span> GB
Generation <span class="token keyword">No</span> : <span class="token number">1596282421</span>
Uptime        : <span class="token number">1</span>d:<span class="token number">19</span>h:<span class="token number">54</span>m:<span class="token number">33</span>s
Heap Memory <span class="token punctuation">(</span>MB<span class="token punctuation">)</span>       : <span class="token number">606.36</span> <span class="token operator">/</span> <span class="token number">3441.00</span>
<span class="token keyword">Off</span> Heap Memory <span class="token punctuation">(</span>MB<span class="token punctuation">)</span>   : <span class="token number">1.51</span>
<span class="token keyword">Data</span> Center   : dc1
Rack          : r1
Exceptions    : <span class="token number">0</span>
<span class="token keyword">Key</span> Cache     : entries <span class="token number">6</span><span class="token punctuation">,</span> size <span class="token number">4.11</span> KB<span class="token punctuation">,</span> capacity <span class="token number">100</span> MB<span class="token punctuation">,</span> hit rate <span class="token number">0.000</span><span class="token punctuation">,</span> recent hit rate <span class="token number">0.000</span><span class="token punctuation">,</span> <span class="token keyword">save</span> period <span class="token operator">in</span> seconds <span class="token number">3600</span>
<span class="token keyword">Row</span> Cache     : entries <span class="token number">0</span><span class="token punctuation">,</span> size <span class="token number">0</span> bytes<span class="token punctuation">,</span> capacity <span class="token number">0</span> bytes<span class="token punctuation">,</span> hit rate NaN<span class="token punctuation">,</span> <span class="token keyword">save</span> period <span class="token operator">in</span> seconds <span class="token number">0</span>
</div>

The statusgossip command will show the current state of gossip, the load on the cluster, and the generation number. It will also show the uptime, heap memory, and off-heap memory for each node. Additionally, it will show the data center and rack for each node, as well as any exceptions that have occurred.

To monitor compaction, you can use the nodetool command nodetool compactionstats. This command will show the current compaction status for each table in the cluster, including the number of completed and active compactions, as well as the total bytes compacted.

<div class="code-mirror  language-sql" contenteditable="plaintext-only" spellcheck="false" style="outline: none; overflow-wrap: break-word; overflow-y: auto; white-space: pre-wrap;">$ nodetool compactionstats
pending tasks: <span class="token number">0</span>
compaction <span class="token keyword">type</span>: Major
compaction history:
</div>

Note ? In this example there is no compaction history.

The compactionstats command will show the number of pending tasks, the compaction type (major or minor), and the compaction history for each table.

Real-life examples

A real-life example of monitoring cluster health in Cassandra would be a scenario where you are running a Cassandra cluster that is being used to store user data for a web application. In this case, you would want to regularly check the health of the cluster to ensure that all nodes are up and running, and that there is an even distribution of data across the nodes. Additionally, you would want to check the replication and compaction status to ensure that data is being properly replicated and compacted.

Another example would be for an e-commerce website where the Cassandra cluster is used to store product and order information. In this case, it would be important to monitor the load on each node, as well as the number of completed and active compactions, to ensure that the cluster can handle the high volume of read and write requests from the website.

Conclusion

Monitoring the health of your Cassandra cluster is an important aspect of maintaining a high-performing and available database. By understanding the key metrics to look for and using the tools provided by Cassandra, like nodetool, you can keep a close eye on the overall health of your cluster and identify potential issues before they become a problem. With proper monitoring, you can ensure that your Cassandra cluster is running at its best and your data is always available to your application.