Community:Splunk for UNIX
From Splunk Wiki
This documentation applies to Splunk version 3.4.x and earlier ONLY
The Splunk for UNIX application supports a variety of different log records or data sources. Among them:
- Generic UNIX syslog events
- UNIX services, such as ssh, ftp, dhcp, cron, snmpd, named, and so on
- UNIX system monitoring (ps, top, vmstat, etc.)
- Linux iptables firewall
- OS X ipfw firewall
- OpenBSD PF firewall
The supported scripted inputs are:
- audit (rlog.sh)
Installation and configuration
See http://www.splunk.com/base/Documentation/latest/Admin/InstallSplunkApplications for generic instructions of how to install and configure this add-on.
For all the searches, you should make sure that they are owned by the correct user. The easiest way to do this is to use the savedsearches.conf_local file and place it in $SPLUNK_HOME/etc/apps/local/savedsearches.conf. Edit the file and assign the searches to the right user and role (based on the template). Note, if you are using LDAP, you need to use user names, and not user IDs in the configuration file.
Scripted input configuration
To enable the scripted inputs to monitor ps, vmstat, iostat, and so on, do the following:
- Modify any of the stanza's to tailor their enabled/disabled, interval ( sec ), source, sourcetype.
- If you have installed in non-default places you may need to fix the paths in some of the bin scripts
- By default all monitoring data is placed into the default index
- The iostat input requires iostat installed. It's in the sysstat RPM which can often be installed with "yum install sysstat"
- Enable the inputs you want in inputs.conf with "disabled = false"
- Don't forget to restart splunk. Once running, check splunkd.log for errors.
For event types to work your logging should be prefixed with an action value that reflects the action taken by the firewall. This can be achieve with the following parameter --log-prefix "action=[PASS|BLOCK] " including the trailing space.
It's advisable that you log both INPUT and OUTPUT traffic. The example snippet below shows all outbound allowed traffic expect loopback being logged as well as all the rejected inbound traffic and allowed traffic to port 22, allowed inbound traffic to ports 8000 and 8089 as well as ICMP traffic is not being logged.
SAMPLE IPTables Configuration
From /etc/sysconfig/iptables: (on RedHat)
-A INPUT -j RH-Firewall-1-INPUT -A FORWARD -j RH-Firewall-1-INPUT -A RH-Firewall-1-INPUT -i lo -j ACCEPT -A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT -A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8000 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8089 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j LOG --log-prefix "action=ALLOW " -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT -A RH-Firewall-1-INPUT -j LOG --log-prefix "action=REJECT " -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited -A OUTPUT -s 127.0.0.1 -d 127.0.0.1 -j ACCEPT -A OUTPUT -m state --state NEW -j LOG --log-prefix "action=ALLOW "
auditd is available on Linux. To get audit data into Splunk use the rlog.sh input script by enabling it in the inputs.conf. Set disabled=false to do so. Here's how to set auditd up on a UNIX system:
- Install if needed, but it should be there
- Make sure it starts on boot
- Start if necessary
- Add a file to be watched
- Make a change to the file
- see the change
yum install audit
chkconfig audit on
auditctl -w /etc/passwd -p wa -k splunk
-w is file, -p is actions (write, append, read, execute), -k is an optional key tag added to the audit.log entry.
ausearch -i -f /etc/passwd
-i means translate so you see user's name instead of uid. -f is file to look for. Ausearch has lots of args for searching
SplunkLightForwarder and Transplant Parsing
If the Splunk for Unix application is installed on a forwarder where parsing is not done, such as the SplunkLightForwarder in Splunk versions 3.4 and later, or if an earlier version is manually configured for Transparant Forwarding, the Splunk for Unix application must be installed on the receiver as well.
This is because some of the configuration is used at data acquisition time, on the forwarder, and some at parse time, which occurs on the receiver in this type of configuration.
Operating system support
This application supports most UNIX flavors, such as FreeBSD, Solaris, AIX, Linux, and OSX
Tips and tricks
If you install this application on a lightweight forwarder, you will need to re-enable the exec processor. See instructions: http://www.splunk.com/doc/latest/admin/ForwardingandReceiving#Lightweightforwardingandrouting