In this blog, we will see how to demonstrate indexing the data from a centralized rsyslog server into Splunk.

In computing, Syslog is a standard for message logging. It allows separation of the software that generates messages, the system that stores them, and the software that reports and analyses them. Each message is labeled with a facility code, indicating the software type generating the message, and assigned a severity level.

Rsyslog is an open-source software utility used on UNIX and Unix-like computer systems for forwarding log messages in an IP network. It is also used in a scenario where one has many data in multiple hosts and one wants to gather the data into a central server which can be later accessed from it.
In our case, we will be gathering many sets of /var/log/secure data from multiple VM hosts running on Linux OS to a single VM Host and then forwarding that to Splunk using a UniversalForwarder.
Prerequisites before installing and configuring centralized rsyslog server:
1.Open the port for rsyslog traffic. By default UDP 514 port is used for rsyslog.
firewall-cmd –zone=public –add-port=514/udp –permanent
2.If we want to use any other port and if SELinux is enabled then the following steps need to be followed:
a.yum install policycoreutils-python
b.semanage port -a -t syslogd_port_t -p tcp 10514
where 10514 is desired port number.
 
Configuration of Rsyslog:-
 
1. Rsyslog is usually installed and already running in CentOS/RHEL. In order to check if the status is up and running, run the following command:-
           systemctl status rsyslog
If the service is not running by default, run the following command to start rsyslog daemon.
           systemctl start rsyslog
If rsyslog has been installed at all use the following command to run it and the start it as well.
            yum install rsyslog
2. Take a backup of/etc/rsyslog.conf file on client as well as a server by using the following command:-
cp myfile.txt myfilecopy.txt

The view of rsyslog.conf –

3. Login to the/etc/rsyslog.conffile and in the server uncomment the following lines:-
4. On client add the server ip and port as required by your standards as shown below:-
5. Restart rsyslog server after making changes in rsyslog.conf files on server and client.
         systemctl restart rsyslog
 
 
Indexing into splunk:-
 
Now, as we can see that rsyslog has been configured and that the syslogs from the different VM’s are being sent to the centralised server, we can now index the rsyslog data into splunk.
 
As usual it’s now time to make the inputs.conf file to monitor the rsyslog files. It’s usually best to keep the centralised rsyslog server as well as the splunk universal forwarder on the same instance. This is mainly due to ease of work.
 
1. In the inputs.conf, add the paths of which logs you want to monitor, the index and the sourcetype.
 
2. Now it’s time to create the props.conf to extract the host name from the logs that will be indexed into splunk as it will help us identify from which host the individual log has been originated.

 
3. Now create 2 different apps to place the inputs.conf and props.conf respectively. We use the app name Avo_FW_rsys for inputs.conf and Avo_SH_Rsyslog_parse for props.conf. After creating the app add both the .conf files into their individual default folders respectively. 
4. After the above step place the app in the /etc/deployment-apps folder in the deployment server and then push the app towards the splunk universal forwarder by using the following command. The class name is where you want to push the bundle to the deployer, cluster master or forwarder. The app named Avo_FW_rsys will be pushed towards the forwarder and Avo_SH_Rsyslog_parse will be pushed towards the deployer.
 
/opt/splunk/bin/splunk reload deploy-server –class
 
5.From the deployer the Avo_SH_Rsyslog_parse app needs to be sent towards the search head cluster so use the following command for it to happen.
/opt/splunk/bin/splunk apply shcluster-bundle –target :8089
Be sure to check which ip in the search head cluster is the captain as well. To check this go to the UI of any search head in the cluster > settings > Search Head Clustering and you will find who is the captain. After the bundle is pushed towards the search head cluster click on begin rolling restart which is present in the top left corner on the same page where you found the ip address of the captain of the search head cluster.
6. Now, after the above steps go to your search head cluster and check if you can see your rsyslog logs or not.
 
That’s all! Rsyslog is now configured as a centralised log server, can collect logs from remote clients and it has also been indexed into splunk as well.

If you are still facing issue regarding indexing the data from centralized rsyslog server Feel free to Ask Doubts in the Comment Box Below and Don’t Forget to Follow us on 👍 Social Networks, happy Splunking >😉