Here with another blog will share the knowledge of Indexer Bucket Rebalancing. Every other company upgrades day by day whether it can be by Employees or the Equipment’s. Here we are focusing into Splunk, Big Data, Data Analysis and there can be many more. Putting a tools i.e. like splunk into a firm and building an architecture for the employees to work on is easy but maintaining is more far a difficult scenario. Let’s see there can be servers not running, indexer might be down, Forwarders not sending the data and so on. On this Blog we will be covering one of the issues that can arise while maintaining the architecture of your Splunk Environment. So let’s take an example of our firm

We were successful in building an Splunk Architecture that consist of 4 searchhead , 3 indexers, Heavy forwarder, Universal Forwarder Deployment Server and Cluster master. For few months it was working smoothly but once we were getting loads of data, eventually splunk started to slow down. Then we came up with 2 more indexers to tackle the issues. We thought now we are again on track with 5 indexers in environment can be more efficient but system doesn’t seems too. Still we were facing same issues that we had with 3 indexers. Then figured out that new indexer were not in a sync and all the load was getting into old indexers.

Missing steps like “INDEXER BUCKET REBALANCING” cause to this failure.

So what is Indexer bucket rebalancing? 

Rebalancing the bucket copies into index clustering. In simpler term – Balancing the data equally in all the indexers.
In splunk technical term we refer data by buckets. All the data ingested in splunk are in form of buckets.

You balance the distribution of bucket copies across the set of peer nodes.

Here’s the graphical representation of indexer before indexer rebalancing –


As you can see Data or Buckets in indexers were not uniform. These lead to problems like:

  1. Higher load on existing indexers
  2. Poor utilization of indexers
  3. Incomplete searches
  4. Node Failure
  5. Random node selection by indexer replication
  6. Node detention on reaching max available storage.
After indexer balancing, we see uniformity in all indexers.

 

Now if we see our indexers, All indexers are in use.
Some Advantages of rebalancing are –
  • Even data and search load distribution
  • Lower storage requirement per node
  • Improves search performance
How did we achieve this?

There are 2 types of Indexer cluster rebalancing-
1. Primary Rebalancing
2. Data Rebalancing 
Primary Rebalancing
Primary Rebalancing does not move searchable copies to different peer nodes. Because of this limitation, primary rebalancing is unlikely to achieve a perfect balance of primaries.
To Perform Primary rebalancing on Master
Run this query On Master Node

  • curl –k –u admin:pass –request POST https://localhost:8089/services/cluster/master/control/control/rebalance_primaries
All you have to do is run this query and done.
Data Rebalancing
There are 3 ways how we can perform data rebalancing in splunk.
  1. By User Interface
  2. By CLI
  3. By editing server.conf file
Method 1 – By User Interface
– Login to Indexer Master Instance
– Navigate to Setting > Indexer Clustering
– Navigate to Data Rebalance

– Set the Threshold Value 
– Select that index you want to rebalance
– Click on “Start”

Note :- A rebalance threshold value of 1.00 means that rebalancing will continue until the cluster is fully balanced, with each peer having the same number of copies. The default value is 0.90, which means that rebalancing will continue until each peer is within 90% of a perfect balance.

Method 2 – By CLI (On Indexer Master)

Enter Following command on CLI, where Indexer Master is installed
  • # splunk edit cluster-config -mode master -rebalance_threshold 0.95 -auth admin:your_SplunkWeb_password

Note :- 0.95 is the rebalance_threshold value , Default value is 0.90.

Then Restart the session to Reflect Changes.
Method 3- By Editing server.conf (On Index Master)
Edit the server.conf file from this directory $SPLUNK_HOME/etc/system/local/

Add rebalance_threshold  = 0.9 to [clustering] stanza
Then Restart the same Instance to reflect the changes

If you are still facing issue regarding Indexer Bucket Rebalancing Feel free to Ask Doubts in the Comment Box Below and Don’t Forget to Follow us on 👍 Social Networks, happy Splunking >😉