Implement Split Indexing in Splunk

“This usecase is related to customer query, where customer wants to divide data from single source into two internal teams for dashboarding and analysis purpose. Due to some internal team data exchange security policy, data needs to be hidden so that every team could see data related to their work only.”

Team A is handling “error data” and Team B is handling “Info data”

Solution:
We have one field log_level and it has two values info and error. Now, what we want to do is we need to send info logs to index=info and error logs to index=error.

Data onboarded for this usecase , contains two fields log_level and msg.

Note: Msg field contains, data as per info and error.

Note: Split indexing should be done at index time.

Create props.conf – Create sourcetype for data to be indexed and for transforms configurations.

• [split_csv]                                             
• TRANSFORMS-index = routing_to_index          //Transforms.conf initialisation 

Create transforms.conf – To route data to alternate index based on log_level field value.

• [routing_to_index]
• DEST_KEY = _MetaData:Index    // send the portion of data to index mentioned in FORMAT{error}
• REGEX = ^error                            //It will search for regex pattern (error keyword) and inde
• FORMAT = error                          //mention index name where error log to be store

Create inputs.conf – To monitor the onboarded data file.

• [monitor:///opt/split_testing.csv]
• index = info
• sourcetype = split_csv
• disabled = false

Overall output will be shown as following.

 

Info messages will go to index=info. It will show only info logs from log_level field as error log already split to error index.

Error message will go to index=error