🎯 Objective

To establish a Mutual TLS (mTLS) direct trust model by creating a single Master Root CA that signs certificates for both the Deployment Server (DS) and Universal Forwarders (UFs), while implementing a Hybrid Root CA approach to ensure zero-downtime migration for existing default-configured Universal Forwarders.

🏗️ Strategy Overview

All certificate generation (Root CA, DS, and UF) will be executed on the Linux Deployment Server to leverage the native Splunk OpenSSL toolset.
A Hybrid Trust file will be created on the DS to accept both legacy and new certificates simultaneously.
Once generated, the required files will be securely transferred to the Windows Universal Forwarders.

🚀 PHASE 1: Certificate Generation (Executed on Linux DS)

First, create a dedicated directory on the Deployment Server to generate and store the certificates:

mkdir -p /opt/splunk/etc/auth/mycerts
cd /opt/splunk/etc/auth/mycerts

🏛️ Step 1: Generate the Root CA (The Master Trust Anchor)

🎯 Purpose

To establish a master authority that will verify and sign the certificates for both the Deployment Server and the Universal Forwarders.

🔑 Generate the Root Private Key

/opt/splunk/bin/splunk cmd openssl genrsa -out root_private.pem 2048

Explanation: This creates a 2048-bit master private key (root_private.pem). This key must be kept secure, as it is used to sign all subsequent certificates.

📜 Generate the Root Public Certificate

/opt/splunk/bin/splunk cmd openssl req -new -x509 -days 3650 -key root_private.pem -out cacert.pem -subj "/C=US/ST=State/L=City/O=Client/CN=ClientRootCA"

Explanation: This command utilizes the master key to generate a public Root Certificate (cacert.pem) valid for 10 years (3650 days). This file will be distributed to both the DS and UFs to act as the ultimate trust anchor.

🌉 Step 1.5: Generate the Hybrid Root CA (The Zero-Downtime Bridge)

🎯 Purpose

To create a combined trust file on the DS that allows it to trust both the new Client certificates and the existing default Splunk certificates simultaneously.

🔄 Combine the Default Splunk CA with the New Client CA

cat /opt/splunk/etc/auth/cacert.pem cacert.pem > hybrid_root_ca.pem

Explanation: This merges the system’s default SplunkCommonCA text with the new Client Root CA text into a single file named hybrid_root_ca.pem. This allows the DS to authenticate clients from both “worlds”.

🖥️ Step 2: Generate Deployment Server (DS) Certificates

🎯 Purpose

To provision the Deployment Server with its own mTLS certificate to present to incoming UFs.

🔑 Generate the DS Private Key

/opt/splunk/bin/splunk cmd openssl genrsa -out ds_private.pem 2048

📄 Generate the DS Certificate Signing Request (CSR)

/opt/splunk/bin/splunk cmd openssl req -new -key ds_private.pem -out ds.csr -subj "/C=US/ST=State/L=City/O=Client/CN=BCCS-PDS"

Explanation: This creates a temporary request file (ds.csr) containing the DS identity details, which is submitted to the Root CA for signing.

✅ Sign the DS Certificate using the Root CA

/opt/splunk/bin/splunk cmd openssl x509 -req -in ds.csr -CA cacert.pem -CAkey root_private.pem -CAcreateserial -out ds_cert.pem -days 1095

Explanation: The Root CA signs the CSR, producing a valid DS public certificate (ds_cert.pem) valid for 3 years.

📦 Combine the DS Files

cat ds_cert.pem ds_private.pem > ds-combined.pem

Explanation: Splunk requires the public certificate and private key to be bundled into a single file.

💻 Step 3: Generate Universal Forwarder (UF) Certificates

🎯 Purpose

To provision the Universal Forwarder with its own mTLS certificate so the DS can authenticate it.

🔐 Generate the UF Private Key, CSR, and Sign

/opt/splunk/bin/splunk cmd openssl genrsa -out uf_private.pem 2048

📦 PHASE 2: File Distribution Strategy

🎯 Purpose

Isolate and distribute only the specific files required for each component.

⚙️ PHASE 3: Configuration & Enforcement

🖥️ Deployment Server Configuration (Linux)

📄 Target File

/opt/splunk/etc/system/local/server.conf

🔧 Configuration

[sslConfig]
enableSplunkdSSL = true
serverCert = /opt/splunk/etc/auth/mycerts/ds-combined.pem
sslRootCAPath = /opt/splunk/etc/auth/mycerts/hybrid_root_ca.pem
requireClientCert = true

🔄 Action

/opt/splunk/bin/splunk restart splunkd

💻 Universal Forwarder Configuration (Windows)

📄 Target File

C:\Program Files\SplunkUniversalForwarder\etc\system\local\server.conf

🔧 Configuration

[sslConfig]
serverCert = C:\Program Files\SplunkUniversalForwarder\etc\auth\mycerts\uf-combined.pem
sslRootCAPath = C:\Program Files\SplunkUniversalForwarder\etc\auth\mycerts\cacert.pem
sslVerifyServerCert = true

🔄 Action

splunk restart

🧪 Validation Result

📊 Image below shows 4 test clients, one containing the new certificate and the other three containing the default certificate. All are successfully being accepted by the Deployment Server even after client certificate enforcement due to the Hybrid Root CA trust model.

🔒 PHASE 4: Post-Migration Lock Down

🎯 Purpose

Once all UFs have been migrated to the new Client certificates, the DS should be locked down to only trust the new Root CA.

🛡️ Action

Update the DS server.conf to point directly to the new Root CA and remove the hybrid file reference.

# In /opt/splunk/etc/system/local/server.conf

🔄 Finalize Changes

Restart Splunk to finalize the secure state.

If you are still facing an issue, feel free to Ask Doubts in the Comment Section Below and Don’t Forget to Follow us on 👍 Social Networks.

| Happy Splunking 😉