Latest Updates


Solving KV Store Errors in Splunk

There are number of reasons exists which leads to Splunk throwing up errors with KVstore. In our Splunk environment we recently got multiple KVstore errors popped up as shown below. The solving process made us realized that KVstore error can be due to anonymous issues and same error message for different reason will get pop up make it confusing for us. So to get rid of it other than pre-existing solution on internet one need to check every aspects related to KVstore directory thoroughly.

Search peer AVP-RHEL-SPLK_SH1 has the following message: KV Store changed status to failed. Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded:; the following nodes did not respond affirmatively: failed with No route to host, failed with No route to host.

Below are the 2 procedure we tried in order to resolve errors. Out of which first one was already documented, but in our case errors showing was due to some other reason. In some cases the error message popping up are same.

Procedure 1st: See the below steps to solve SSL related issue.
Step 1:  Go to bin directory of Splunk Search Head
  • cd $SPLUNK_HOME/bin
Step 2: Check status of KV store by using the following command.
  • ./splunk show kvstore-status -auth :     or  #./splunk show kvstore-status   (later it will ask for id and pass)

Step 3: Check the FQDN (Fully Qualified Domain Name) of your server by using the following command.
  • hostname –fqdn

Step 4: Now create a new SSL certificate in the directory called $SPLUNK_HOME$/etc/auth.
Run the below command to create an SSL certificate for this server by putting FQDN value you copied through previous step. This will generate “ pem”  file in response
  • ./splunk createssl server -cert 3072 -d /opt/splunk/etc/auth -n server -c

Step 5: Now restart Splunk by bin directory and again run command to check KVstore status as listed above it will be showing ready if it was related to SSL certificate issue

Unfortunately, we didn’t succeed as there was no SSL issue, but you can follow this for any SSL issue generating KV store error in Splunk.

Procedure 2nd - Steps to solve mongodb related issue.

Step 1: Open the CLI of this Search Head. Go to Kvstore directory
  • cd $SPLKUNK_HOME/var/lib/splunk/kvstore
Step 2:  Run below command to change ownership if it is misplaced by any chance in KVstore or any other directory in it.

In our case some mongodb directory files inside /kvstore/mongodb were in root ownership.
It got changed to Splunk, and all error message resolved 😊
  • chown -R splunk:splunk /opt/splunk/var/lib/kvstore/mongodb

If you are still facing issue regarding this topic Feel free to Ask Doubts in the Comment Box Below and Don't Forget to Follow us on 👍 Social Networks, happy Splunking >😉
Solving KV Store Errors in Splunk Reviewed by Avotrix.Author on Thursday, July 30, 2020 Rating: 5

No comments:

All Rights Reserved by Avotrix © 2018 - 2019
Developed and created by Avotrix

Contact Form


Email *

Message *

Powered by Blogger.