How to automatically split a mixed syslog stream into clean, searchable, CIM-compliant sourcetypes — web logs and audit logs — using transforms and props.

What is Sourcetype Renaming?

Sourcetype renaming in Splunk is the process of changing the sourcetype of incoming data based on specific patterns or conditions in the raw data.

Why Do We Need It?

When data comes from a single syslog stream (like zscaler:zia:syslog), it may contain multiple types of logs mixed together:

  • Web/Proxy logs (HTTP traffic, URLs, user activity)
  • Audit logs (admin actions, configuration changes, login attempts)
  • DNS logs
  • Firewall logs

Problem: All logs arrive with the same sourcetype zscaler:zia:syslog, making it difficult to:

  • Search specific log types efficiently
  • Apply different field extractions for different log types
  • Create targeted dashboards and reports
  • Manage data retention policies per log type

Solution: Use transforms to inspect the raw data and rename the sourcetype based on content patterns.

How Does It Work?

 

Why We Did This for Zscaler

In our case:

  1. Single syslog input receives all Zscaler ZIA logs as zscaler:zia:syslog
  2. Mixed content: Web logs and audit logs arrive in the same stream
  3. Different field structures:
    • Web logs have fields like url=, requestmethod=, useragent=
    • Audit logs have fields like recordid=, preaction=, postaction=
  4. Need separation to apply correct field extractions and searches

Sourcetype renaming solves this by automatically routing:

  • Web traffic logs → zscalernss-web
  • Admin audit logs → zscalernss-audit

Configuration Order & Flow

Order of Execution:

  1. Base Sourcetype (zscaler:zia:syslog) – Receives raw syslog data
  2. transforms.conf – Defines regex patterns to detect log types
  3. props.conf for renamed sourcetypes – Extracts fields after renaming
 

Configuration Flow:

Step-by-Step Configuration

Step 1: Configure transforms.conf (Pattern Detection)

Location: $SPLUNK_HOME/etc/apps/<your_app>/local/transforms.conf

Purpose: Define regex patterns to identify different log types

Explanation:

  • zscaler_zia_web_syslog:
    • Looks for web-specific fields: url=, requestmethod=, useragent=, status=, responsesize=
    • If found → rename sourcetype to zscalernss-web
  • zscaler_zia_audit_syslog:
    • Looks for audit-specific fields: recordid= followed by digits + preaction= or postaction=
    • If found → rename sourcetype to zscalernss-audit

Why these patterns?

  • Web logs always contain URL and HTTP-related fields
  • Audit logs always contain recordid and action fields
  • These patterns are mutually exclusive (no overlap)

Step 2: Configure Base Sourcetype in props.conf

Location: $SPLUNK_HOME/etc/apps/<your_app>/local/props.conf

Purpose: Define how to process incoming syslog data and apply transforms

Line-by-line Explanation:

  • [zscaler:zia:syslog]: Base sourcetype that receives raw syslog data

  • SHOULD_LINEMERGE = false: Don’t merge multiple lines into one event

    • Each line is a separate log event

  • LINE_BREAKER = ([\r\n]+): Break events on newline characters

    • Tells Splunk where one event ends and another begins

  • TIME_PREFIX = ^: Look for timestamp at the beginning of line

    • ^ means start of line

  • TIME_FORMAT = %a %b %d %H:%M:%S %Y: Define timestamp format

    • Example: Thu Nov 20 17:43:52 2025
    • %a = Day name (Thu)
    • %b = Month name (Nov)
    • %d = Day (20)
    • %H:%M:%S = Time (17:43:52)
    • %Y = Year (2025)

  • MAX_TIMESTAMP_LOOKAHEAD = 50: Search first 50 characters for timestamp

    • Improves parsing performance

  • TRANSFORMS-02-zia_web = zscaler_zia_web_syslog: Apply web log transform

    • 02 = execution order (runs first)
    • Calls the transform defined in transforms.conf

  • TRANSFORMS-04-zia_audit = zscaler_zia_audit_syslog: Apply audit log transform

    • 04 = execution order (runs second)
    • Calls the transform defined in transforms.conf

Why this order matters:

  • Transforms execute in numerical order (02 before 04)
  • If web pattern matches, event is renamed and audit transform is skipped
  • If web pattern doesn’t match, audit transform is tried
  • This prevents double-processing and conflicts

Step 3: Configure zscalernss-audit Sourcetype (After Renaming)

Purpose: Extract and normalize fields for audit logs after sourcetype has been renamed

[zscalernss-audit]
FIELDALIAS-action = event.action AS action
FIELDALIAS-Action2 = event.action AS Action
FIELDALIAS-category = event.category AS category
FIELDALIAS-Category2 = event.category AS Category
FIELDALIAS-clientip = event.clientip AS clientip
FIELDALIAS-ClientIP2 = event.clientip AS “Client IP”
FIELDALIAS-cloudname = “AA in Cloud” AS cloudname
FIELDALIAS-interface = event.interface AS interface
FIELDALIAS-Interface2 = event.interface AS Interface
FIELDALIAS-preaction = event.pre_action AS preaction
FIELDALIAS-preaction2 = event.pre_action AS “Pre Action”
FIELDALIAS-postaction = event.post_action AS postaction
FIELDALIAS-post_action2 = event.post_action AS “Post Action”
FIELDALIAS-Subcategory2 = event.subcategory AS Subcategory
FIELDALIAS-time = event.time AS time
FIELDALIAS-Time2 = event.time AS Time
FIELDALIAS-recordid = event.recordid AS recordid
FIELDALIAS-resource = event.resource AS resource
FIELDALIAS-result = event.result AS result
FIELDALIAS-src_ip = event.clientip AS src_ip
FIELDALIAS-stripEventPrefix = event.adminid AS adminid
FIELDALIAS-sub_category = event.subcategory AS sub_category
FIELDALIAS-user = event.adminid AS user
FIELDALIAS-user2 = event.adminid AS User
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%d-%H:%M:%S-GMT%z
TIME_PREFIX = time” =
TRUNCATE = 0
category = Splunk App Add-on Builder
description = Zscaler ZIA Admin Audit Logs
pulldown_type = true

Line-by-line Explanation:

  • FIELDALIAS-*: Create field aliases (alternative names for same data)
    • Original field: event.action
    • Aliases: action, Action
    • Why multiple? Support different search conventions (lowercase, Title Case)

Key Field Mappings:

Raw Field (from syslog) Normalized Field CIM Field Description
event.action action, Action action Admin action performed
event.adminid user, User, adminid user Administrator username
event.clientip src_ip, clientip, “Client IP” src_ip Source IP address
event.pre_action preaction, “Pre Action” – State before action
event.post_action postaction, “Post Action” – State after action
event.recordid recordid – Unique record ID
event.resource resource – Resource modified
event.result result – Action result
event.category category, Category category Action category
event.subcategory sub_category, Subcategory – Action subcategory

  • TIME_FORMAT = %Y-%m-%d-%H:%M:%S-GMT%z: Timestamp format for audit logs

    • Example: 2025-11-20-17:43:52-GMT+0000

  • TIME_PREFIX = time" =: Pattern before timestamp in raw data

    • Looks for: time" = 2025-11-20-17:43:52-GMT+0000

Why FIELDALIAS?

  • Audit logs have event. prefix in field names
  • We create aliases without the prefix for easier searching
  • Also create CIM-compliant field names like src_ip, user

Step 4: Configure zscalernss-web Sourcetype (After Renaming)

Purpose: Extract and normalize fields for web logs after sourcetype has been renamed

[zscalernss-web]
pulldown_type = true
EVAL-action = lower(action)
EVAL-dlp_type = “Inline Gateway”
EVAL-duration = clienttranstime + servertranstime
EVAL-dvc = “Zscaler Cloud Proxy”
EVAL-dvc_zone = “Cloud Proxy”
EVAL-vendor_product = “Zscaler_ZIA_Proxy”
FIELDALIAS-appname_as_app = appname AS app
FIELDALIAS-ClientIP_as_src = ClientIP AS src
FIELDALIAS-ClientIP_as_src_ip = ClientIP AS src_ip
FIELDALIAS-aob_gen_zscalernss_web_alias_1 = protocol AS transport
FIELDALIAS-aob_gen_zscalernss_web_alias_2 = user AS src_user
FIELDALIAS-aob_gen_zscalernss_web_alias_3 = dlpengine AS severity
FIELDALIAS-aob_gen_zscalernss_web_alias_4 = threatname AS signature
FIELDALIAS-aob_gen_zscalernss_web_alias_5 = contenttype AS http_content_type
FIELDALIAS-aob_gen_zscalernss_web_alias_6 = hostname AS dest
FIELDALIAS-aob_gen_zscalernss_web_alias_8 = responsesize AS bytes_in
FIELDALIAS-aob_gen_zscalernss_web_alias_9 = requestsize AS bytes_out
FIELDALIAS-clientpublicIP_as_src_translated_ip = clientpublicIP AS src_translated_ip
FIELDALIAS-clienttranstime_as_response_time = clienttranstime AS response_time
FIELDALIAS-department_as_src_user_bunit = department AS src_user_bunit
FIELDALIAS-dlpdictionaries_as_signature = dlpdictionaries AS signature
FIELDALIAS-filename_as_file_name = filename AS file_name
FIELDALIAS-md5_as_file_hash = md5 AS file_hash
FIELDALIAS-refererURL_as_http_referrer = refererURL AS http_referrer
FIELDALIAS-requestmethod_as_http_method = requestmethod AS http_method
FIELDALIAS-serverip_as_dest_ip = serverip AS dest_ip
FIELDALIAS-threatcategory_as_category = threatcategory AS category
FIELDALIAS-transactionsize_as_bytes = transactionsize AS bytes
FIELDALIAS-urlcategory_as_category = urlcategory AS category
FIELDALIAS-useragent_as_http_user_agent = useragent AS http_user_agent
REPORT-ta_builder_internal_use_kv_format_results_for_zscalernss_web = ta_builder_internal_use_kv_format_results_for_zscalernss_web
category = Network & Security
description = Zscaler Web/Proxy Logs

Line-by-line Explanation:

  • EVAL-*: Create calculated fields (compute new values)
    • EVAL-action = lower(action): Convert action to lowercase
    • EVAL-duration = clienttranstime + servertranstime: Calculate total duration
    • EVAL-vendor_product = "Zscaler_ZIA_Proxy": Set vendor product name

Key Field Mappings:

Raw Field (from syslog) Normalized Field CIM Field Description
ClientIP src, src_ip src_ip Client source IP
hostname dest dest Destination hostname
serverip dest_ip dest_ip Destination IP
requestmethod http_method http_method HTTP method (GET/POST)
useragent http_user_agent http_user_agent Browser user agent
contenttype http_content_type http_content_type Response content type
responsesize bytes_in bytes_in Bytes received
requestsize bytes_out bytes_out Bytes sent
transactionsize bytes bytes Total bytes
user src_user src_user Username
protocol transport transport Protocol (HTTP/HTTPS)
appname app app Application name
md5 file_hash file_hash File MD5 hash

Calculated Fields:

  • action: Normalized to lowercase for consistent searching
  • duration: Sum of client and server transaction times
  • vendor_product: Identifies source as Zscaler ZIA Proxy
  • dvc: Device name – “Zscaler Cloud Proxy”
  • dvc_zone: Device zone – “Cloud Proxy”
  • dlp_type: Set to “Inline Gateway”

Step 5: Apply Configuration

  1. Save transforms.conf
  2. Save props.conf
  3. Restart Splunk to apply changes:

    $SPLUNK_HOME/bin/splunk restart

Verification Queries

Check if sourcetype renaming is working:

index=proxy sourcetype=”zscaler*”
| stats count by sourcetype

Expected Output:

  • zscalernss-web (web logs) – Should show large count (e.g., 2,989,649)
  • zscalernss-audit (audit logs) – Should show smaller count (e.g., 9)

 

Complete Data Flow

Summary

What we achieved:

  1. Single syslog input (zscaler:zia:syslog) automatically splits into two sourcetypes
  2. Web traffic logs → zscalernss-web with HTTP-specific field extractions
  3. Admin audit logs → zscalernss-audit with audit-specific field extractions
  4. All fields normalized to CIM-compliant names for easy searching and reporting

Why this approach?

  • Efficiency: Automatic routing without manual intervention
  • Organization: Separate sourcetypes for different log types
  • Flexibility: Different field extractions for different data structures
  • Compliance: CIM-compliant field naming for standard dashboards and reports

If you are still facing an issue, feel free to Ask Doubts in the Comment Section Below and Don’t Forget to Follow us on 👍 Social Networks.

| Happy Splunking. 😉