What is url toolbox in splunk?
It is an app in splunk base which is also known as UTBOX. It only needs to be deployed on Splunk Search Heads and the bundles will automatically be sent to your Splunk Indexers.
What is the use of Url toolbox?
UTBox for Splunk specially created for URL manipulation.It converts the complicated urls into the simple one.
Lets start with the installation of the app.
Step1: Install the app from the splunk base.
Link : https://splunkbase.splunk.com/app/2734/
As soon as you install the app you will get the additional lookups added into Lookup definition.
NOTE:URL Toolbox isn’t a custom search command, you get access to all its power via macros (so remember your ticks)! One of the most commonly used macros in URL Toolbox is called ut_parse_extended(2). It parses your URL and passes the data to multiple different fields prefaced with ut_.
Bringing two fields into the ut_parse_extended macro. The first is the URL, which is pretty straightforward, but the second is a field called “list.” The URL Toolbox—that “list” field is the catalog of different top level domain.
Note:There are a couple of common lists that exist in the world (including an official one from IANA), but if we’re trying to differentiate the domain from the top level domain (TLD), the most popular source of truth is from Mozilla. Mozilla’s list of TLDs not only has “classic” TLDs like .com and .co.uk (which is bizarrely missing from IANA), but it will also include items like .edu.tj. The important takeway is that you need to use eval to make a field called “list” with the value “mozilla” or “*” (which searches all of the TLD lists available) before you actually call ut_parse_extended.
- ut_parse(url, list) or ut_parse_extended(url, list)
which uses a list to extract the following fields: ut_port, ut_domain, ut_tld, ut_domain_without_tld, ut_subdomain, ut_subdomain_count and ut_subdomain_parts
index="test2" sourcetype=tool
| eval list="mozilla"
| ut_parse_extended(url,list)
You will get the required fields.
index="test2" sourcetype=tool
| eval list="mozilla"
| ut_parse_extended(url,list)
|table url ut_domain
|dedup ut_domain
In this example We then use the ut_parse_extended(url, list) macro to parse the URL based on the Mozilla TLD list.
What is Shannon entropy?
Shannon’s Entropy is simply the “amount of information” in a variable.
- ut_shannon(word) :
Return the shannon entropy of the given word
In this example we are using ut_shannon which calculate the level of entropy in the field “ut_domain” .
Query:
index="test2" sourcetype=tool
|eval list="mozilla"
|ut_parse_extended(url,list)
| ut_shannon(ut_domain)
| stats count by url ut_shannon
The score is pretty high, which makes sense since there is a high variety of frequency over those data.
The words of low characters distribution get a low score.
If you are still facing issue regarding url toolbox in splunk Feel free to Ask Doubts in the Comment Box Below and Don’t Forget to Follow us on 👍 Social Networks, happy Splunking >😉
