There are number of reasons exists which leads to Splunk throwing up errors with KVstore. We recently got splunk kv store errors popped up as shown below. The solving process made us realized that KVstore error can be due to anonymous issues and same error message for different reason will get pop up make it confusing for us. So to get rid of it other than pre-existing solution on internet one need to check every aspects related to KVstore directory thoroughly. 

splunk kv store errors 1

Search peer AVP-RHEL-SPLK_SH1 has the following message: KV Store changed status to failed. Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: 192.168.0.142:8191; the following nodes did not respond affirmatively: 192.168.0.144:8191 failed with No route to host, 192.168.0.143:8191 failed with No route to host.

splunk kv store errors 2

Below are the 2 procedure we tried in order to resolve errors. Out of which first one was already documented, but in our case errors showing was due to some other reason. In some cases the error message popping up are same.

Procedure 1st: See the below steps to solve SSL related issue.
Step 1:  Go to bin directory of Splunk Search Head

  • cd $SPLUNK_HOME/bin

Step 2: Check status of KV store by using the following command.

  • ./splunk show kvstore-status -auth :     or  #./splunk show kvstore-status   (later it will ask for id and pass)
splunk kv store errors 3

OR

splunk kv store errors 4


Step 3: Check the FQDN (Fully Qualified Domain Name) of your server by using the following command.

  • hostname –fqdn
splunk kv store errors 5


Step 4: Now create a new SSL certificate in the directory called $SPLUNK_HOME$/etc/auth.
Run the below command to create an SSL certificate for this server by putting FQDN value you copied through previous step. This will generate “ pem”  file in response

  • ./splunk createssl server -cert 3072 -d /opt/splunk/etc/auth -n server -c
splunk kv store error


Step 5: Now restart Splunk by bin directory and again run command to check KVstore status as listed above it will be showing ready if it was related to SSL certificate issue

splunk kv store errors 7

Unfortunately, we didn’t succeed as there was no SSL issue, but you can follow this for any SSL issue generating KV store error in Splunk.

Procedure 2nd – Steps to solve mongodb related issue.

Step 1: Open the CLI of this Search Head. Go to Kvstore directory

  • cd $SPLKUNK_HOME/var/lib/splunk/kvstore

Step 2:  Run below command to change ownership if it is misplaced by any chance in KVstore or any other directory in it.

In our case some mongodb directory files inside /kvstore/mongodb were in root ownership.
It got changed to Splunk, and all error message resolved 😊

  • chown -R splunk:splunk /opt/splunk/var/lib/kvstore/mongodb
splunk kv store errors 8


If you are still facing issue regarding splunk kv store errors Feel free to Ask Doubts in the Comment Box Below and Don’t Forget to Follow us on 👍 Social Networks, happy Splunking >😉