Search peer AVP-RHEL-SPLK_SH1 has the following message: KV Store changed status to failed. Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: 192.168.0.142:8191; the following nodes did not respond affirmatively: 192.168.0.144:8191 failed with No route to host, 192.168.0.143:8191 failed with No route to host.
Below are the 2 procedure we tried in order to resolve errors. Out of which first one was already documented, but in our case errors showing was due to some other reason. In some cases the error message popping up are same.
Procedure 1st: See the below steps to solve SSL related issue.
Step 1: Go to bin directory of Splunk Search Head
Step 2: Check status of KV store by using the following command.
Step 3: Check the FQDN (Fully Qualified Domain Name) of your server by using the following command.
Step 4: Now create a new SSL certificate in the directory called $SPLUNK_HOME$/etc/auth.
Run the below command to create an SSL certificate for this server by putting FQDN value you copied through previous step. This will generate “ pem” file in response
Step 5: Now restart Splunk by bin directory and again run command to check KVstore status as listed above it will be showing ready if it was related to SSL certificate issue
Unfortunately, we didn’t succeed as there was no SSL issue, but you can follow this for any SSL issue generating KV store error in Splunk.
Procedure 2nd – Steps to solve mongodb related issue.
Step 1: Open the CLI of this Search Head. Go to Kvstore directory
Step 2: Run below command to change ownership if it is misplaced by any chance in KVstore or any other directory in it.
In our case some mongodb directory files inside /kvstore/mongodb were in root ownership.
It got changed to Splunk, and all error message resolved 😊
If you are still facing issue regarding splunk kv store errors Feel free to Ask Doubts in the Comment Box Below and Don’t Forget to Follow us on 👍 Social Networks, happy Splunking >😉