Introduction :
Anomaly detection is the identification of rare events, items, or observations which are suspicious because they differ significantly from standard behaviors or patterns. Anomalies in data are also called standard deviations, outliers, noise, novelties, and exceptions.
Definition :
Anomalous command in splunk is used to detect the unusual or unpredicted events in the data , once you run ‘anomalies’ command in search you can able to find out the unexpectedness field in the interesting field list on the bottom left side of the search and reporting app. You can add it to the selected field lists to verify the information about the values in the event.
Syntax :
anomalies [<anomalies-options>…][field]
Anomalous options:
Threshold=<num> – It calculates the unusual values in the event. The <num> denotes the number to represents limits,if the value is greater than threshold value, the event is considered as anomalous.
labelonly=<bool> – <bool> denotes Boolean characters (True or false). If lableonly=true, gives results of all events and if labelonly=false, unusual events less than threshold limit are removed from the result. Default is False.
Normalize=<bool> – normalize=false , if the event fields containing numeric data values should be considered as categorized .
Default is True.
Maxvalues=<num>-It finds the maximum previous events while analyzing the unexpectedness of a field value. By default the calculation uses the previous 100 events for the comparison
Default is 100.
Field=<field> – field name from an event which has to be analyzed for unexpectedness.
Default is _raw.
Denylist=<filename> – File name is the name of the file (eg: csv file..) , there is list of events which are expected and is not required for us in the respective csv file is added to the denylist.
denylistthreshold=<num> – <num> represents limits in number, A similar score can be created while comparing the incoming events with denylisted events. We can provide a threshold limit for which if the score is greater than the given threshold score, the event is marked as unexpected. Default is 0.05.
Example 1:
Query
index=main sourcetype=csv | anomalies threshold=0.01 by City |search City=*
Result :
Explanation :
Here we extract a sample set of data from the local main index and sourcetype=csv as a main search , after piping provided a sub search; anomalies threshold=0.01 by city anomalies command ( detect anomalies ) along with the ‘threshold value 0.01 (if unexpectedness field in the event found greater than the threshold limit 0.01 then that event is anomalous), by City specifies the user field to separate the results for anomaly detection using ‘by clause’. Search City=* gives the results of all anomalous detected city list.
Example 2:
Query
index=main sourcetype=csv| anomalies threshold=0.9 maxvalues=10 labelonly=true by Gender| sort -unexpectedness| eval threshold=0.9| eval score=if(unexpectedness>=threshold, “anomalous”, “not_anomalous”)| table score, Gender ,unexpectedness |dedup Gender
Result :
Explanation :
Here again we are extracting data from index main and sourcetype csv where anomalies threshold =0.9 and set maxvalues =10 to find the maximum previous values denotes the size of the event numbers (value=10 mean it will calculated resulted values from 0 to 9 labelonly=true by Gender means when labelonly is set true ,no events were removed. ‘By Gender’ specifies the Gender field to separate the results for unexpectedness using ’by clause’. sort the unexpectedness field in the descending order. and eval threshold=0.09 used to calculate the expression of the threshold value by creating a field called threshold, eval score = if ( unexpectedness >= threshold, “anomalous”, “not anomalous”),track the anomalous value by applied condition using eval command (if unexpectedness is >=threshold limit; the result shows the event is anomalous, otherwise not anomalous and the final result displayed on newly created field ‘score’. Then finally search uses the table command to display results separately in the specified fields.
If you are still facing an issue, feel free to Ask Doubts in the Comment Section Below and Don’t Forget to Follow us on 👍 Social Networks.
| Happy Splunking 😉
