Community:Splunk2Splunk SSL DefaultCerts

From Splunk Wiki

Jump to: navigation, search

Configuring Splunk forwarding to use the default SSL server certificate

This procedure will explain how to configure Splunk to send data from your forwarders to your indexer(s) using the default SSL server certificate.

This configuration will ensure that your data will be encrypted when in transit on your network, but it will not be securely encrypted since we will be using the default SSL server certificate that ships with every Splunk package : $SPLUNK_HOME/etc/auth/server.pem

If you want to ensure that no one can easily snoop on your Splunk-to-Splunk traffic or wrongfully send data to your indexers, we recommend that you use new SSL certificates signed by your own certificate authority.

Instructions to create your own root certificate and use it to sign new server certificates for Splunk-to-Splunk traffic can be found [[1]].

Instructions to use your own certificate authority to sign new server certificates for Splunk-to-Splunk traffic can be found [[2]].


1 - Set up the indexer(s) to use the default server certificate that ships with Splunk and listen for Splunk to Splunk traffic on a given port :

We will be using port 9997 to receive data from forwarders.

  • In $SPLUNK_HOME/etc/system/local/inputs.conf (or in the appropriate directory of any app you are using to distribute your forwarding configuration), set up the following stanzas :

[SSL]
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem
password = password

[splunktcp-ssl:9997]
compressed = true

We will not be using "requireClientCert = true" since it would be pointless to check the validity of the default server certificate that ships with Splunk and that the forwarders will present to the indexer.

Despite that fact, it is still necessary to indicate the path to the certificate authority public key with "rootCA = $SPLUNK_HOME/etc/auth/cacert.pem".

  • Restart splunkd after making these changes.

# $SPLUNK_HOME/bin/splunk restart splunkd

Note that the server certificate pass phrase will be hashed and stored in $SPLUNK_HOME/etc/system/local/inputs.conf, overwriting the clear-text value of "password" if it was defined there. If "password" was defined in clear-text in an inputs.conf located in an app, it *will not* be hashed there and will still be present in clear text in that location. This doesn't matter too much in this case since the pass phrase for the default server certificate is well known.


2 - Set up the forwarder(s) to use the default server certificate that ships with Splunk and to send Splunk to Splunk traffic to the indexer(s) receiving port :

In our example, the indexer's IP address is 10.1.12.112.

  • Define the following stanzas in $SPLUNK_HOME/etc/system/local/outputs.conf (or in the appropriate directory of any app you are using to distribute your forwarding configuration) :

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = 10.1.12.112:9997
compressed = true
sslVerifyServerCert = false
sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = password

Just as we did for the indexer, we will not be using "sslVerifyServerCert = true" since it would also be pointless to ask the forwarders to check the validity of the default server certificate that ships with Splunk and that the indexer(s) will be presenting.

Here too, it is still necessary to indicate the path to the certificate authority public key in with "sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem".

If you are distributing data to several indexers, you can simply add their HOST:PORT address as a comma-separated list in the "server" configuration parameter of the target group definition stanza.

  • Restart splunkd after making these changes.

# $SPLUNK_HOME/bin/splunk restart splunkd

Note that the server certificate pass phrase will be hashed and stored in $SPLUNK_HOME/etc/system/local/outputs.conf, overwriting the clear-text value of "sslPassword" if it was defined there. If "sslPassword" was defined in clear-text in an outputs.conf located in an app, it *will not* be hashed there and will still be present in clear text in that location. This doesn't matter too much in this case since the pass phrase for the default server certificate is well known.

3 - Check for a successful connection in splunkd.log :

  • This is what you should see during the indexer start-up sequence in $SPLUNK_HOME/var/log/splunkd.log :

02-06-2011 19:19:01.552 INFO TcpInputProc - using queueSize 1000
02-06-2011 19:19:01.552 INFO TcpInputProc - SSL cipherSuite=ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
02-06-2011 19:19:01.552 INFO TcpInputProc - supporting SSL v2/v3
02-06-2011 19:19:01.555 INFO TcpInputProc - port 9997 is reserved for splunk 2 splunk (SSL)
02-06-2011 19:19:01.555 INFO TcpInputProc - Port 9997 is compressed
02-06-2011 19:19:01.556 INFO TcpInputProc - Registering metrics callback for: tcpin_connections

  • This is what you should see during the forwarder start-up sequence in $SPLUNK_HOME/var/log/splunkd.log :

02-06-2011 19:06:10.844 INFO TcpOutputProc - Retrieving configuration from properties
02-06-2011 19:06:10.848 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
02-06-2011 19:06:10.848 INFO TcpOutputProc - found Whitelist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
02-06-2011 19:06:10.848 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
02-06-2011 19:06:10.850 INFO TcpOutputProc - Will retry at max backoff sleep forever
02-06-2011 19:06:10.850 INFO TcpOutputProc - Using SSL for server 10.1.12.112:9997, sslCertPath=/opt/splunk/etc/aut/server.pem
02-06-2011 19:06:10.854 INFO TcpOutputProc - ALL Connections will use SSL with sslCipher=
02-06-2011 19:06:10.859 INFO TcpOutputProc - initializing single connection with retry strategy for 10.1.12.112:9997


  • And this is what a successful connection attempt will look like indexer-side :

02-06-2011 19:19:09.848 INFO TcpInputProc - Connection in cooked mode from 10.1.12.111
02-06-2011 19:19:09.854 INFO TcpInputProc - Valid signature found
02-06-2011 19:19:09.854 INFO TcpInputProc - Connection accepted from 10.1.12.111

  • ...and forwarder-side :

02-06-2011 19:19:09.927 INFO TcpOutputProc - attempting to connect to 10.1.12.112:9997...
02-06-2011 19:19:09.936 INFO TcpOutputProc - Connected to 10.1.12.112:9997


4 - Troubleshooting :

  • First, check in $SPLUNK_HOME/var/log/splunk/splunkd.log on both ends for errors. On the indexer, check for the messages from the TCP input processor "TcpInputProc", and on the forwarder, check the messages from the TCP output processor "TcpOutputProc".
  • In general, it is a good idea to increase the logging level of the appropriate processors on the indexer and the forwarder in $SPLUNK_HOME/etc/log.cfg.
    On the forwarder, set "category.TcpOutputProc=DEBUG", on the indexer set "category.TcpInputProc=DEBUG". Restart Splunk for these to take effect and observe the start-up sequence for the pertinent component. Most configuration issues are explicitly revealed by this method.
  • Check the SSL configuration as it is seen by Splunk using btool.
    • On the indexer :
      $SPLUNK_HOME/bin/splunk cmd btool inputs list --debug
    • On the forwarder :
      $SPLUNK_HOME/bin/splunk cmd btool outputs list --debug
  • Make sure that the certificates are readable by the user that Splunk runs as. Indexer-side, two common problems are :
    • The path to the server certificate file set as the value of "serverCert" in inputs.conf is wrong, or the file cannot be read. This will generate the following error :
      12-16-2010 16:07:30.965 ERROR SSLCommon - Can't read certificate file /opt/splunk/etc/auth/server.pem errno=33558530 error:02001002:system library:fopen:No such file or directory
    • The password to the RSA private key contained in the server certificate file is wrong. This password is set as the value of "password" in inputs.conf. This will generate the following error :
      12-07-2010 07:56:45.663 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem

On *nix, you can manually test the password of the RSA key contained in the file by running the following openssl command :
# openssl rsa -in /opt/splunk/etc/auth/server.pem -text

The same can be done on Windows with the openssl binary that ships with Splunk :
C:\Program Files\Splunk\bin>openssl.exe rsa -in "c:\Program Files\Splunk\etc\auth\server.pem" -text

  • More information regarding the configuration of splunk2splunk SSL connections can be found here in the online documentation :

http://www.splunk.com/base/Documentation/latest/Admin/EncryptandauthenticatedatawithSSL
http://www.splunk.com/base/Documentation/latest/Admin/SecureaccesstoyourSplunkserverwithSSL

The appropriate sections of the spec files for inputs.conf and outputs.conf are also a very good resource :

http://www.splunk.com/base/Documentation/latest/Admin/Inputsconf
http://www.splunk.com/base/Documentation/latest/Admin/Outputsconf

These files can be found in $SPLUNK_HOME/etc/system/README.

Hot Wiki Topics


About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk