In this blog we will practically perform how to use Preamble and Sedcmd parameter in Splunk using props.conf.
PREAMBLE REGEX
Preamble is an introductory statement. It is used to ignore preamble lines by writing regular expression that matches preamble lines.
It is used to remove the lines that occur before the structure data during index time.
Default: Splunk does not set preamble regex by default.
Syntax
PREAMBLE_REGEX = <REGEX>
SEDCMD
It is used to remove/replace/substitute data only at index time. It is used to protect private or sensitive data by deleting or encrypting it. For example passwords, credit card number etc.
Splunk does not set it by default.
Syntax
SEDCMD-<class-name>= s/regex/replacement/flags.
SEDCMD is used in case for Replace/Substitute/Masking.
s – It is used for replacement.
Regex– It is the regular expression that captures words/string which needs to be removed/replaced.
Replacement-It is used to replace the match regex.
Flag– g to replace all the matches ,or a number to replace specific match.
Masking
It is used to hide the actual data using modified content like characters or numbers.
Task to be achieve:
This is the sample file in which we will use Preamble regex and sedcmd parameter in props.conf while ingesting data into splunk.
1. We need to remove preamble lines which is highlighted in red box using Preamble regex.
2. Replacement using SECMD – First Line which is underlined in red is the id field which will be replaced by space.
3. Masking using SECMD – second line underlined in red is the password field which has to be Masked.
4. We need to remove the last two lines which is highlighted in red box using SEDCMD.
Sample file –
This is how sample file will be ingested in Splunk after applying PREAMBLE regex and SEDCMD parameter in Props.conf.
In props.conf we have written our Preamble and SEDCMD parameters as below:
Explanation of props :
1.In PREAMBLE_REGEX write the regex string that matches your data string which needs to be removed. As we see below
Preamble lines to be removed:
PREAMBLE_REGEX=(\<\?xml.*)|(\<SymCLI_ML.*)|(\<Symmetrix.*)|(\Symm_Info.*)|(\<symid.*)
Note:- the regex written in above parameter should be matching with the whole preamble lines that needs to be removed.
2. SEDCMD attribute is used to replace id field with space. id is the sed class name, s is used for replacement then regex matching the id field is written g to replace all the matches.
SEDCMD-id=s/<?id.*>//g
Note: These id fields are replaced by space using SEDCMD-id. ?id.* is regex to match full id line . If we don’t mention anything in replacement it will replace with space by default.
- To mask password 123-897 with xxx-xxx with SEDCMD.
SEDCMD-password=s/(\d{3}-\d{3})/xxx-xxx /g
SEDCMD attribute is used for masking password. class name of sedcmd is password, s is used for replacement then regex matching the password is written, xxx-xxx is written to replace the matching regex and g to replace all the matches.
- To remove the last lines with SEDCMD, by default it will remove by space.
Last lines in data are:
To remove last lines we can use below sedcmd syntax.
SEDCMD-remove=s/(?ims)^\s{2}\<\/Symmetrix\>.*\<\/SymCLI_ML\>.*$//g
Note: SEDCMD attribute is used to remove the last two lines, remove is the sed class name, s is used for replacement then regex matching the last two lines is written and g to replace all the matches.
If you are still facing issue in Preamble and sedcmd parameter , Feel free to Ask Doubts in the Comment Box Below and Don’t Forget to Follow us on 👍 Social Networks, happy Splunking >😉
