From Splunk Wiki
Deployment scenario: high security requirements
Splunk is often deployed in environments with high security requirements.
These high security requirements may be due to PCI, SOX, FISMA, and/or HIPAA mandates as well as other factors such as high risk of industrial espionage or operation on a classified or top secret network.
Splunk is able to meet most high security requirements without code modification. Splunk relies whenever possible on open standards, which provides transparency and facilitates accreditation.
Encrypt all data transmission to and from all Splunk instances in a manner which conforms to data encryption standards
All data transmission between Splunk instances (e.g. forwarder to indexer, indexer to client browser) can be strongly encrypted (SSLv2/3).
All data transmission between third-party devices such as firewalls or intrusion detection systems (IDS) can be strongly encrypted as long as the third-party device supports strong ciphering and hashing and can perform a standard handshake with other devices.
Harden all Splunk instances in a manner which conforms to existing hardening standards
Splunk can be hardened in a manner that meets or exceeds most hardening standards.
Aspects of Splunk that facilitate conformation with hardening standards includes but is not limited to:
- Lowering the number of network ports that Splunk is listening to in order to reduce overall exposure
- Enabling local and remote authentication to deter tampering
- Running as a non-privileged user to reduce risk in the case of Splunk account compromise
The most efficient method for Splunk to comply with existing policies and standards is to authenticate and authorize users against an organization's existing directory which conforms to these policies already.
Thus, an LDAP strategy will be implemented which securely authenticates users and performs authorization by mapping user's group memberships to internal Splunk roles.
To ensure that the aforementioned requirement for secure communication is maintained, LDAP over SSL (also known as LDAPS) will be configured to encrypt all traffic between the directory server and Splunk.
Manage and secure data in a manner which conforms to existing data security and data asset lifecycle policies.
This organization has the following information standards for applications:
- All applications must support the creation and maintenance of audit logs;
- All applications must provide integrity monitoring of application code and application data;
- All applications must provide notification in the event of critical audit and integrity events;
Splunk provides for all of these requirements through its audit.log, file system change monitor, block signing, and alerting capabilities.
The following hardware was used in this deployment:
- (2x) 4-core 2.4 Ghz Intel Xeon CPU
- 16 GB RAM
- (4x) 300 GB 7200 RPM SATA HDD
- PCI-x SATA RAID Controller (0+1)
- Intel 10/1000 4-port NIC
The hardware specifications may not seem important to this scenario given the requirements stated above. However, it is important to keep in mind that features such as traffic encryption/decryption and block signing will use additional hardware resources.
Data sources, input methods, and indexing volumes
The data sources and their respective input methods for this deployment were as follows:
- @@+++Windows Domain Controller event logs@@
- Tailed via a Splunk lightweight forwarder and output to TCP port 9997 over SSL.
- Total number of Domain Controllers 5
- Estimated volume per day: 5 GB
- @@+++Firewall syslog@@
- The enterprise ingress/egress and DMZ firewalls output syslog to Splunk instance on TCP port 5514. Since most network devices including firewalls do not support sending syslog over SSL/TLS, an VPN was configured between the devices and Splunk to ensure the encryption of data over the wire.
- Total number of firewalls: 25
- Estimated volume per day: 5 GB
To get these estimates, we analyzed the available archive data and calculated daily averages. In some cases, there was approximately one week of archive data available, while in other cases more thorough archive data was available.
Using the estimates above, the Splunk instance is projected to index approximately 10 GB per day.
The licensing and input assumptions should be validated during the first 30 days in production. It is a low risk endeavor, as Splunk's licensing model will never terminate forwarding or indexing, and furthermore will only disable search if the license is violated 7 times within a 30 day rolling window.
Server installation was performed using the standard linux tarball downloaded from http://www.splunk.com/download. The Splunk binaries and logs will go under /opt, while the data store will be located at /data.
- create the local user account "splunk" and the local group "splunk"
useradd -G splunk -m -u 666 splunk
- Change directory to /opt
- Unzip the compressed tarball
- Extract the uncompressed tarball
- Manually edit /opt/splunk/etc/splunk-launch.conf file and make the following addition:
- Open /opt/splunk_/etc/splunk.license and add the 10 GB enterprise license key.
- Set Splunk to start at system boot as the user "splunk"
/opt/splunk/bin/splunk enable boot-start -user splunk
Authentication & Authorization Configuration
To ensure that authentication and authentication to Splunk complies with enterprise security policies and password standards, we configured Splunk to authenticate to Active Directory(AD) and mapped security groups from AD to roles within Splunk. Rather than creating a separate user and group repository meeting policies and standards, it makes far more sense to leverage an existing directory where these policies and standards are already effective.
- For each instance, we created an authentication configuration that bound to the directory server corpdc01 with the bind account "svc_splunk". The default LDAPS port 636 was used since secure LDAP was required by the corporate security policy.
- "ou=splunk,ou=security_groups,dc=corp,dc=local" off of the directory root was configured as the group base, from which Splunk will enumerate security groups to be mapped to internal Splunk roles.
- "ou=users,dc=corp,dc=local" was configured as the user base, from which Splunk will enumerate users that belong to the aforementioned security groups mapped to Splunk roles.
- The security group "splunk_admin" was created for corporate administrators and mapped to the "admin" role within Splunk. In addition, the security groups "security_ops", "network_ops", and "hardware_ops" were created. Corresponding roles were created in /opt/splunk/etc/system/local/authorize.conf as follows:
[role_security_ops] importRoles = Power;Everybody srchFilter = [role_network_ops] importRoles = Power;Everybody srchFilter = [role_hardware_ops] importRoles = Power;Everybody srchFilter =
The below authentication configuration was created in /opt/splunk/etc/system/local/authentication.conf:
[corpdc01] Admin = splunk_admin; security_ops = security_ops; network_ops = network_ops; hardware_ops = hardware_ops; SSLEnabled = 1 bindDN = cn=svc_splunk,ou=service_accounts,ou=users,dc=corp,dc=local bindDNpassword = failsafeLogin = admin failsafePassword = groupBaseDN = ou=splunk,ou=security_groups,dc=corp,dc=local; groupBaseFilter = (objectclass=*) groupMappingAttribute = dn groupMemberAttribute = member groupNameAttribute = cn host = corpdc01 pageSize = 800 port = 636 realNameAttribute = name userBaseDN = ou=users,dc=corp,dc=local userBaseFilter = (objectclass=*) userNameAttribute = sAMAccountName [auth] authSettings = corpdc01 authType = LDAP
To allow Splunk to connect to the LDAPS port on the domain controller corpdc01, the following steps were taken:
- The enterprise root certificate authority was exported to plain-text as cert1.pem;
- The directory /opt/splunk/etc/openldap/certs was created;
- The exported ca certificate was placed in /opt/splunk/etc/openldap/certs;
- The file /opt/splunk/etc/openldap/ldap.conf was edited and the following two lines were added:
TLS_CACERT /opt/splunk/etc/openldap/certs/cert1.pem TLS_CACERTDIR /opt/splunk/etc/openldap/certs
- When Splunk was started, the secure bind to corpdc01 on port 636 was tested and verified.
Since Splunk was running as a non-root user, we could not bind to privileged ports (<1024). Thus, the following ports were decided upon:
|SplunkWeb port||Splunkd port||Splunk data port||Syslog Port|
By default, the SplunkWeb port is 8000 and the Splunkd port is 8089. Communication to and from the Splunkd port is encrypted by default. To enable SplunkWeb encryption, /opt/splunk/etc/system/local/web.conf was created as such:
[settings] enableSplunkWebSSL = true
To force each instance of Splunk to listen on the selected ports as well as to force encryption on the Splunk data port, /opt/splunk/etc/system/local/inputs.conf was created as such:
[SSL] serverCert=/opt/splunk/etc/auth/server.pem password=password rootCA=/opt/splunk/etc/auth/cacert.pem requireClientCert=false [splunktcp-ssl://9997] disabled = false queue = parsingQueue sourcetype = tcp-9997 [udp://1514] disabled = false sourcetype = syslog
The above configurations utilize the certificate and self-signed root certificate authority that are created when Splunk starts for the first time. If the customer had a requirement to leverage their own certificates generated from their enterprise root certificate authority, we would have replaced the serverCert, password, and rootCA values with the path and passwords to their own server certificate and rootCA certificate.
After configuration was completed, the service command was used to ensure that Splunk starts as the "splunk" user:
service splunk start
Note that if you still want to receive syslog on default port (514), you can use iptables to redirect incoming data to port 1514.
Forwarder Installation & Configuration
After changing the network devices to direct its syslog output to the correct UDP port, we installed Splunk in a lightweight forwarder configuration on the Windows servers as follows:
- Download the Windows binary from http://www.splunk.com/download
- Install interactively using the MSI, accepting all defaults (i.e. run as local system and index all event logs)
- Start splunk if it does not start automatically, and ensure that the service is set to start automatically at boot
- Open c:\program files\splunk\etc\splunk.license with a text editor and paste in your forwarder license
- Issue the following commands from a command prompt
c:\program files\splunk\bin> splunk.exe disable webserver c:\program files\splunk\bin> splunk.exe set server-type forwarder
- Modify c:\program files\splunk\etc\system\local\inputs.conf by placing the following at the top of the file:
- Create c:\program files\splunk\etc\system\local\outputs.conf as follows:
[tcpout:splunk] server = Splunk.corp.local:9997 [tcpout-server://Splunk.corp.local:9997] sslCertPath=c:\program files\splunk\etc\auth\server.pem sslRootCAPath=c:\program files\splunk\etc\auth\cacert.pem sslPassword=password sslVerifyServerCert=false
- Restart the forwarder
c:\program files\splunk\bin> splunk.exe restart
- Change the default admin password
c:\program files\splunk\bin> splunk.exe edit user admin -password r0tsky!~!1
After configuring all Windows forwarders as detailed above, we verified that the Splunk index server received new events from the domain controller event logs and network devices.
Block Signing & File System Monitoring Configuration
By default, auditing is enabled on the Splunk indexing server and audit events flow to the "_audit" index. Also by default, file system monitoring is enabled for the $SPLUNK_HOME/etc directory.
To meet the requirements for integrity monitoring and notification, we enabled block signing for Splunk's indexed data and file system change monitoring for the entire Splunk directory.
We first created an input which overrides the default file system monitoring, then created an input which will monitor the entire $SPLUNK_HOME directory.
- Modify /opt/splunk/etc/system/local/inputs.conf as follows:
[fschange:$SPLUNK_HOME/etc] disabled=true [fschange:$SPLUNK_HOME] disabled=false pollPeriod = 60 recurse=true followLinks=false hashMaxSize=-1 fullEvent=false sendEventMaxSize=-1 filesPerDelay = 10 delayInMills = 100
- Modify /opt/splunk/etc/system/local/indexes.conf as follows:
- Source setSplunkEnv and run a python script to generate audit keys:
>source /opt/splunk/bin/setSplunkEnv >python /opt/splunk/bin/genAuditKeys.py
- Create /opt/splunk/etc/system/local/audit.conf as follows:
[auditTrail] privateKey = /opt/splunk/etc/auth/audit/private.pem publicKey = /opt/splunk/etc/auth/audit/public.pem
- Restart Splunk.
- Create the following saved search to alert Splunk admins whenever a critical audit event occurs. In this case, an critical event consisted of a modification of a Splunk binary. The search was set to run every five minutes and email firstname.lastname@example.org if there was at least one event returned.
[audit_event alert] action_email = email@example.com action_rss = 0 counttype = number of events enableSched = 1 quantity = 0 relation = greater than role = Admin schedule = */12 * * * * search = index=_audit path=/opt/splunk/bin* sendresults = 0 userid = 1
Using the above configuration, all the given project requirements were met without issue or significant and complicated configuration.
All data transmission to and from all Splunk instances has been encrypted in a manner which conforms to data encryption standards using Splunk's internal SSL libraries on the forwarder and receiver as well as IPSec between Splunk and enterprise network devices.
All Splunk instances were hardened to comply with hardening standards by reducing network port usage on the forwarders and indexer, running as non-privileged users on the forwarders (Local System) and the indexer (splunk), and requiring authentication on all Splunk instances.
Authentication and authorization within Splunk conforms to existing security policies and password standards, as Active Directory is used to authenticate users and map security groups to internal Splunk roles.
Splunk was configured conforms to existing data security and data asset lifecycle policies by enabling block signing as well as file system monitoring, all of which feed in to Splunk's existing audit facilities.