We were successful in building an Splunk Architecture that consist of 4 searchhead , 3 indexers, Heavy forwarder, Universal Forwarder Deployment Server and Cluster master. For few months it was working smoothly but once we were getting loads of data, eventually splunk started to slow down. Then we came up with 2 more indexers to tackle the issues. We thought now we are again on track with 5 indexers in environment can be more efficient but system doesn’t seems too. Still we were facing same issues that we had with 3 indexers. Then figured out that new indexer were not in a sync and all the load was getting into old indexers.
Missing steps like “INDEXER BUCKET REBALANCING” cause to this failure.
So what is Indexer bucket rebalancing?
Rebalancing the bucket copies into index clustering. In simpler term – Balancing the data equally in all the indexers.
In splunk technical term we refer data by buckets. All the data ingested in splunk are in form of buckets.
You balance the distribution of bucket copies across the set of peer nodes.
Here’s the graphical representation of indexer before indexer rebalancing –
As you can see Data or Buckets in indexers were not uniform. These lead to problems like:
- Higher load on existing indexers
- Poor utilization of indexers
- Incomplete searches
- Node Failure
- Random node selection by indexer replication
- Node detention on reaching max available storage.
- Even data and search load distribution
- Lower storage requirement per node
- Improves search performance
- curl –k –u admin:pass –request POST https://localhost:8089/services/cluster/master/control/control/rebalance_primaries
- By User Interface
- By CLI
- By editing server.conf file
Note :- A rebalance threshold value of 1.00 means that rebalancing will continue until the cluster is fully balanced, with each peer having the same number of copies. The default value is 0.90, which means that rebalancing will continue until each peer is within 90% of a perfect balance.
- # splunk edit cluster-config -mode master -rebalance_threshold 0.95 -auth admin:your_SplunkWeb_password
Note :- 0.95 is the rebalance_threshold value , Default value is 0.90.