Community:Splunk2Splunk SSL SelfSignedCert NewRootCA

From Splunk Wiki

Jump to: navigation, search

Configuring Splunk forwarding to use SSL certificates self-signed by a newly generated root certificate

This procedure will explain how to configure Splunk to send data from your forwarders to your indexer(s) using your own SSL server certificates signed by a newly-generated root certificate acting as a certificate authority.

This configuration will ensure that your data will be securely encrypted when in transit on your network since we will be generating and using brand new server certificates and signing them with your own certificate authority.

1 - Create your own SSL root certificate :

This step can be done on one of your Splunk instances or on any other server where a recent version of openSSL is installed. Splunk 4.1.7 ships with openSSL 0.9.8p (16 Nov 2010).

For your Splunk-to-Splunk traffic to remain secure and confidential, the root certificate and the private keys that will be generated should be kept in a safe location.

  • Create a directory for your certificates and place yourself into it:

# mkdir $SPLUNK_HOME/etc/certs
# cd $SPLUNK_HOME/etc/certs

NOTE : For clarity's sake, it is best to put the new certificates in different directory than $SPLUNK_HOME/etc/auth in order not to overwrite those that exist there. In our example, we will create and use $SPLUNK_HOME/etc/certs. This will also ensure that you can keep using the certificates that ship with Splunk in $SPLUNK_HOME/etc/auth for other Splunk components if you wish to do so.


  • Generate a new, encrypted RSA private key for your root certificate :

We’ll use DES3 encryption and a 1024 bit key length.

# openssl genrsa -des3 -out myCAPrivateKey.key 1024

IMPORTANT : Keep this private key and it's pass phrase safe and secure! This key will be used to sign the server certificates we will install on indexer(s) and forwarders.


  • Generate a new certificate signing request (CSR) from your CA private key :

# openssl req -new -key myCAPrivateKey.key -out myCACertificate.csr

You will be prompted for the pass phrase to the private key we created in the previous step ($SPLUNK_HOME/etc/certs/myCAPrivateKey.key).


  • Use the CA private key to self-sign the CSR and generate the certificate authority public certificate :

# openssl x509 -req -in myCACertificate.csr -sha1 -signkey myCAPrivateKey.key -CAcreateserial -out myCACertificate.pem -days 1095

You will be prompted again for the pass phrase to the private key we created previously ($SPLUNK_HOME/etc/certs/myCAPrivateKey.key).

The file $SPLUNK_HOME/etc/certs/myCACertificate.pem is the public certificate (or key) of your certificate authority. It will be used by clients to verify the server certificate presented to them by the server.


NOTE : Alternatively, the script $SPLUNK_HOME/bin/genRootCA.(sh|bat) can be used to achieve the same result :

# $SPLUNK_HOME/bin/genRootCA.sh -d $SPLUNK_HOME/etc/certs

This will create a new certificate authority public certificate in $SPLUNK_HOME/etc/certs/cacert.pem This public CA certificate is to be distributed to all Splunk instances (indexers and forwarders) who will be checking server certificates signed with the root certificate that the script will also generate (ca.pem).


2 - Generate a new server certificate signed by our own certificate authority :

  • Generate a new RSA private key for your server certificate :

We’ll use DES3 encryption and a 1024 bit key length.

# openssl genrsa -des3 -out myServerPrivateKey.key 1024

IMPORTANT : Keep this private key and it's pass phrase safe and secure! This key will be used to encrypt the outgoing data on any Splunk instance where you install it as part of the server certificate.

  • Generate a new certificate signing request (CSR) from your server private key :

# openssl req -new -key myServerPrivateKey.key -out myServerCertificate.csr

You will be prompted for the pass phrase to the private key we created in the previous step ($SPLUNK_HOME/etc/certs/myServerPrivateKey.key).

IMPORTANT : If you plan to use common-name checking to ask your forwarders to verify the common-name present in the certificate presented by your indexers, make sure to answer the query for "Common Name (eg, your name or your server's hostname) []:" with the string that you will set as the value of "sslCommonNameToCheck" in outputs.conf on your forwarders. More on this later.

  • Use the CA private key to self-sign the CSR and generate a public server certificate signed by your certificate authority :

# openssl x509 -req -in myServerCertificate.csr -sha1 -CA myCACertificate.pem -CAkey myCAPrivateKey.key -CAcreateserial -out myServerPublicCertificate.pem -days 1095

This time, you will be prompted for the pass phrase to the certificate authority private key we created previously ($SPLUNK_HOME/etc/certs/myCAPrivateKey.key).

  • Consolidate the signed server certificate, the server private key and the CA’s public key in a single PEM file :

# cat myServerPublicCertificate.pem myServerPrivateKey.key myCACertificate.pem > myServerCertificate.pem

Once created, the contents of this file should contain the indexer’s certificate (x509 format), the indexer private key (RSA format) and the certificate authority public key (x509 format), in that order. Here's an example :

        -----BEGIN CERTIFICATE-----
        MIICUTCCAboCCQCscBkn/xey1TANBgkqhkiG9w0BAQUFADBtMQswCQYDVQQGEwJV
        ...
        <Server Certificate>
        ...
        8/PZr3EuXYk1c+N5hgIQys5a/HIn
        -----END CERTIFICATE-----
        -----BEGIN RSA PRIVATE KEY-----
        Proc-Type: 4,ENCRYPTED
        DEK-Info: DES-EDE3-CBC,CFCECC7976725DE5
        
        S+DPcQ0l2Z1bk71N3cBqr/nwEXPNDQ4uqtecCd3iGMV3B/WSOWAQxcWzhe9JnIsl
        ...
        <Server Private Key – Passphrase protected>
        ...
        -----END RSA PRIVATE KEY-----
        -----BEGIN CERTIFICATE-----
        MIICUTCCAboCCQCscBkn/xey1TANBgkqhkiG9w0BAQUFADBtMQswCQYDVQQGEwJV
        ...
        <Certificate Authority Public Key>
        ...
        8/PZr3EuXYk1c+N5hgIQys5a/HIn
        -----END CERTIFICATE-----


IMPORTANT : You are free to use a different server certificate for the forwarders than for the indexers. In that case, repeat these steps to create a new certificate (forwarder.pem for example), distribute it to the forwarders and define as the value of "sslCertPath" in outputs.conf.
You can also create and configure one server certificate with a unique common name (CN) per indexer. In that case, the forwarder-side configuration gets a bit more complicated, requiring one server-specific stanza per indexer. More on this in step 4.

NOTE : Here too you can use a script that ships with Splunk to create and sign a new server certificate from the root certificate we created. Example to create server certificate "phobos.pem" with common name "phobos" :

# $SPLUNK_HOME/bin/genSignedServerCert.sh -d $SPLUNK_HOME/etc/certs -n phobos.pem -c phobos -p


3 - Set up the indexer(s) to use the new self-signed server certificate we just created to listen for Splunk to Splunk traffic on a given port :

  • If they aren't present there, copy the server certificate myServerCertificate.pem and the CA public certificate myCACertificate.pem into $SPLUNK_HOME/etc/certs/ on the indexer(s) you are intending to configure.


  • In $SPLUNK_HOME/etc/system/local/inputs.conf (or in the appropriate directory of any app you are using to distribute your forwarding configuration), set up the following stanzas :

[SSL]
rootCA = $SPLUNK_HOME/etc/certs/myCACertificate.pem
serverCert = $SPLUNK_HOME/etc/certs/myServerCertificate.pem
password = server_privkey_password
requireClientCert = false

[splunktcp-ssl:9997]
compressed = true

As you can see, we will be using port 9997 to receive data from the forwarders.

  • IMPORTANT NOTE ABOUT "requireClientCert" :

As of Splunk 4.2.4, setting "requireClientCert = true" in the indexer's inputs.conf will cause forwarding to fail! A bug (SPL-37637) is currently open to address this issue. In the meantime, keep requireClientCert set to "false".

We have set "requireClientCert = true". This requires the following conditions to be met :

a) "rootCA" must point to a file containing the CA's public key. In our example, it's the myCACertificate.pem file we generated in step 1.
b) The forwarder's server certificate defined by "sslCertPath" in outputs.conf (see step 4) is signed by that CA.
c) The forwarder has the password to read his own certificate ("sslPassword" in outputs.conf, as defined in step 4). This password is "server_privkey_password" in our example.

The purpose of this setup is to ensure that only forwarders that you have distributed a signed certificate to can connect to this indexer.

  • Restart splunkd after making these changes.

# $SPLUNK_HOME/bin/splunk restart splunkd

Note that the server certificate pass phrase will be hashed and stored in $SPLUNK_HOME/etc/system/local/inputs.conf, overwriting the clear-text value of "password" if it was defined there. If "password" was defined in clear-text in an inputs.conf located in an app, it *will not* be hashed there and will still be present in clear text in that location. This doesn't matter too much in this case since the pass phrase for the default server certificate is well known.

4 - Set up the forwarder(s) to use the new self-signed server certificate we just created and to send Splunk to Splunk traffic to the indexer(s) receiving port :

  • If they aren't present there, copy the server certificate myServerCertificate.pem and the CA public certificate myCACertificate.pem into $SPLUNK_HOME/etc/certs/ on the forwarders you are intending to configure.


  • Define the following stanzas in $SPLUNK_HOME/etc/system/local/outputs.conf (or in the appropriate directory of any app you are using to distribute your forwarding configuration) :

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = 10.1.12.112:9997
compressed = true
sslRootCAPath = $SPLUNK_HOME/etc/certs/myCACertificate.pem
sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate.pem
sslPassword = default
sslVerifyServerCert = true

In our example, we have one indexer with IP address 10.1.12.112. If you are distributing data to several indexers, you can simply add their HOST:PORT address as a comma-separated list in the "server" configuration parameter of the target group definition stanza.

  • IMPORTANT NOTE ABOUT COMMON-NAME CHECKING:

If you have created one server certificate per indexer and you have set a unique common name (CN) in each indexer certificate to be checked by the forwarders, you will need to configure one [tcpout-server://HOST:PORT] configuration stanza per indexer in outputs.conf. This is so that you can specify which common name to check for which indexer.
Here's an example where our forwarders will check that the indexer @10.1.12.112 displays a server certificate with CN="phobos" and that the indexer @10.1.12.113 displays a server certificate with CN="deimos". If one of those conditions is not matched, the forwarder will refuse to send data to that indexer :

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = 10.1.12.112:9997
compressed = true

[tcpout-server://10.1.12.112:9997]
sslRootCAPath = $SPLUNK_HOME/etc/certs/myCACertificate.pem
sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate.pem
sslPassword = server_privkey_password
sslVerifyServerCert = true
sslCommonNameToCheck = phobos

[tcpout-server://10.1.12.113:9997]
sslRootCAPath = $SPLUNK_HOME/etc/certs/myCACertificate.pem
sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate.pem
sslPassword = server_privkey_password
sslVerifyServerCert = true
sslCommonNameToCheck = deimos

  • IMPORTANT NOTE ABOUT "sslVerifyServerCert" :

In both cases, we have set "sslVerifyServerCert = true". This requires the following conditions to be met :

a)"sslRootCAPath" must point to a file containing the CA's public key.
b) The indexer's "serverCert" (as defined in inputs.conf on step 3) is signed by that CA.
c) The indexer has the password to read his own certificate ("password" in inputs.conf, as defined in step 3).
d) The common name on the indexer's server certificate ("serverCert" in inputs.conf, as defined in step 3) matches the "sslCommonNameToCheck" setting.

This setup ensures that the forwarder will only send data to an indexer presenting a certificate signed by the known CA and that this certificate was issued to someone with the expected name (CN), i.e., so that the client only sends to someone to whom that certificate is specific. The name doesn't have to match the DNS, hostname, servername, or IP or anything outside of these files. (Thus this is not subject to DNS SSL attacks). All that needs to match is the name in the Splunk configuration file ("sslCommonNameToCheck") and the common name signed into the indexer's certificate.

  • Restart splunkd after making these changes.

# $SPLUNK_HOME/bin/splunk restart splunkd

Note that the server certificate pass phrase will be hashed and stored in $SPLUNK_HOME/etc/system/local/outputs.conf, overwriting the clear-text value of "sslPassword" if it was defined there. If "sslPassword" was defined in clear-text in an outputs.conf located in an app, it *will not* be hashed there and will still be present in clear text in that location. This doesn't matter too much in this case since the pass phrase for the default server certificate is well known.


5 - Check for a successful connection in splunkd.log :

  • This is what you should see during the indexer start-up sequence in $SPLUNK_HOME/var/log/splunkd.log :

02-06-2011 19:19:01.552 INFO TcpInputProc - using queueSize 1000
02-06-2011 19:19:01.552 INFO TcpInputProc - SSL cipherSuite=ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
02-06-2011 19:19:01.552 INFO TcpInputProc - supporting SSL v2/v3
02-06-2011 19:19:01.555 INFO TcpInputProc - port 9997 is reserved for splunk 2 splunk (SSL)
02-06-2011 19:19:01.555 INFO TcpInputProc - Port 9997 is compressed
02-06-2011 19:19:01.556 INFO TcpInputProc - Registering metrics callback for: tcpin_connections

  • This is what you should see during the forwarder start-up sequence in $SPLUNK_HOME/var/log/splunkd.log :

02-06-2011 19:06:10.844 INFO TcpOutputProc - Retrieving configuration from properties
02-06-2011 19:06:10.848 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
02-06-2011 19:06:10.848 INFO TcpOutputProc - found Whitelist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
02-06-2011 19:06:10.848 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
02-06-2011 19:06:10.850 INFO TcpOutputProc - Will retry at max backoff sleep forever
02-06-2011 19:06:10.850 INFO TcpOutputProc - Using SSL for server 10.1.12.112:9997, sslCertPath=/opt/splunk/etc/aut/server.pem
02-06-2011 19:06:10.854 INFO TcpOutputProc - ALL Connections will use SSL with sslCipher=
02-06-2011 19:06:10.859 INFO TcpOutputProc - initializing single connection with retry strategy for 10.1.12.112:9997


  • And this is what a successful connection attempt will look like indexer-side :

02-06-2011 19:19:09.848 INFO TcpInputProc - Connection in cooked mode from 10.1.12.111
02-06-2011 19:19:09.854 INFO TcpInputProc - Valid signature found
02-06-2011 19:19:09.854 INFO TcpInputProc - Connection accepted from 10.1.12.111

  • ...and forwarder-side :

02-06-2011 19:19:09.927 INFO TcpOutputProc - attempting to connect to 10.1.12.112:9997...
02-06-2011 19:19:09.936 INFO TcpOutputProc - Connected to 10.1.12.112:9997


6 - Troubleshooting :

  • First, check in $SPLUNK_HOME/var/log/splunk/splunkd.log on both ends for errors. On the indexer, check for the messages from the TCP input processor "TcpInputProc", and on the forwarder, check the messages from the TCP output processor "TcpOutputProc".
  • In general, it is a good idea to increase the logging level of the appropriate processors on the indexer and the forwarder in $SPLUNK_HOME/etc/log.cfg.
    On the forwarder, set "category.TcpOutputProc=DEBUG", on the indexer set "category.TcpInputProc=DEBUG". Restart Splunk for these to take effect and observe the start-up sequence for the pertinent component. Most configuration issues are explicitly revealed by this method.
  • Check the SSL configuration as it is seen by Splunk using btool.
    • On the indexer :
      $SPLUNK_HOME/bin/splunk cmd btool inputs list --debug
    • On the forwarder :
      $SPLUNK_HOME/bin/splunk cmd btool outputs list --debug
  • Make sure that the certificates are readable by the user that Splunk runs as. Indexer-side, two common problems are :
    • The path to the server certificate file set as the value of "serverCert" in inputs.conf is wrong, or the file cannot be read. This will generate the following error :
      12-16-2010 16:07:30.965 ERROR SSLCommon - Can't read certificate file /opt/splunk/etc/auth/server.pem errno=33558530 error:02001002:system library:fopen:No such file or directory
    • The password to the RSA private key contained in the server certificate file is wrong. This password is set as the value of "password" in inputs.conf. This will generate the following error :
      12-07-2010 07:56:45.663 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem

On *nix, you can manually test the password of the RSA key contained in the file by running the following openssl command :
# openssl rsa -in /opt/splunk/etc/auth/server.pem -text

The same can be done on Windows with the openssl binary that ships with Splunk :
C:\Program Files\Splunk\bin>openssl.exe rsa -in "c:\Program Files\Splunk\etc\auth\server.pem" -text

  • More information regarding the configuration of splunk2splunk SSL connections can be found here in the online documentation :

http://docs.splunk.com/Documentation/Splunk/latest/Security/AboutsecuringyourSplunkconfigurationwithSSL

The appropriate sections of the spec files for inputs.conf and outputs.conf are also a very good resource :

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf

These files can be found in $SPLUNK_HOME/etc/system/README.

Personal tools
Hot Wiki Topics


About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk