Latest Updates


Solving Splunk KV Store Errors

Splunk environment can be affected with Errors related to KV store. Along with these errors, there were several Warnings related to KV store and Buckets. It directly impacts to our clustering and make it unstable.
Note: Here we are facing issue with our search head cluster
Below is snip to show error message:

Steps to Solve KV Store Error:

1. To solve this issue first we have to synchronize KV store on all the members of Search-Head cluster.

Resync the KV Store:

When a KV store member fails to transform its data from all the write operations, then it might be stale. To resolve this issue, you must resynchronize the member.
Ø Identify the Stale Member
                         Login to each Member from Putty and Run this command
                          ./splunk show kvstore-status.
This will return the summary of KV store members, as well as information about every other member in the KV store cluster.

Look at the replicationstatus field and identify if any members that have neither "KV store captain" nor "Non-captain KV store member" as values. Then consider it as stale and need to Re-Sync
                                    (Make Sure All the members show the correct information.)

Ø Follow these steps to resync the members. 
·         Determine which node is currently the search head cluster captain, by running below command in any of the Sh’s.
/opt/splunk/bin/splunk show shcluster-status.
·         Login to SH Cluster captain and run splunk resync kvstore .
·         Use the splunk show kvstore-status command to verify that the cluster is resynced.

On cluster members, individually perform following steps.
·         Stop the splunk on search head on each member.
·         Run the command splunk clean kvstore --local.
·         Restart the search head. This triggers the initial synchronization from other KV store members.
·         Run the command splunk show kvstore-status to verify synchronization.

2. Check and change the Permission of splunk.key file if required on each instance.
Go to path /opt/splunk/var/lib/splunk/kvstore/mongo/   and check the permission of the file by command ll or ls –lrth, on each Instance having the error.

Ø  Then change the permission of the file to read only.
            By command: - chmod 400 splunk.key

Restart the Splunk on each instance to reflect changes.

If you are still facing issue regarding this topic Feel free to Ask Doubts in the Comment Box Below and Don't Forget to Follow us on 👍 Social Networks, happy Splunking >😉
Solving Splunk KV Store Errors Reviewed by Avotrix.Author on Tuesday, February 18, 2020 Rating: 5

No comments:

All Rights Reserved by Avotrix © 2018 - 2019
Developed and created by Avotrix

Contact Form


Email *

Message *

Powered by Blogger.