Community:Splunk2Splunk SSL 3rdPartyCA

From Splunk Wiki

Jump to: navigation, search

Configuring Splunk forwarding to use SSL certificates signed by a third party Certificate Authority

Note: Please note that the information in this topic is not valid for version 6.4 or later. Instead, see http://docs.splunk.com/Documentation/Splunk/6.4.0/Security/ConfigureSplunkforwardingtousesignedcertificates for the most up to date information.


This procedure will explain how to configure Splunk to send data from your forwarders to your indexer(s) using new SSL server certificates signed by your own certificate authority.

This configuration will ensure that your data will be securely encrypted when in transit on your network since we will be generating and using brand new server certificates and signing them with your own certificate authority.


1 - Generate a new server certificate signed by our own certificate authority :

  • Create a directory for your certificates and place yourself into it:

# mkdir $SPLUNK_HOME/etc/certs
# cd $SPLUNK_HOME/etc/certs

NOTE : For clarity's sake, it is best to put the new certificates in different directory than $SPLUNK_HOME/etc/auth in order not to overwrite those that exist there. In our example, we will create and use $SPLUNK_HOME/etc/certs. This will also ensure that you can keep using the certificates that ship with Splunk in $SPLUNK_HOME/etc/auth for other Splunk components if you wish to do so.


  • Generate a new RSA private key for your server certificate :

We’ll use DES3 encryption and a 1024 bit key length.

# openssl genrsa -des3 -out myServerPrivateKey.key 1024

IMPORTANT : Keep this private key and it's pass phrase safe and secure! This key will be used to encrypt the outgoing data on any Splunk instance where you install it as part of the server certificate.

  • Generate a new certificate signing request (CSR) from your server private key :

# openssl req -new -key myServerPrivateKey.key -out myServerCertificate.csr

You will be prompted for the pass phrase to the private key we created in the previous step ($SPLUNK_HOME/etc/certs/myServerPrivateKey.key).

IMPORTANT : If you plan to use common-name checking to ask your forwarders to verify the common-name present in the certificate presented by your indexers, make sure to answer the query for "Common Name (eg, your name or your server's hostname) []:" with the string that you will set as the value of "sslCommonNameToCheck" in outputs.conf on your forwarders. More on this later.

  • Send the CSR to your certificate authority and request in return a new server certificate and the CA's public key :

How you input the CSR to your CA completely depends on the setup of your CA. As an example, here is how Splunk’s own self-signing script located at $SPLUNK_HOME/bin/genSignedServerCert.py does it :
# openssl x509 -req -in mycert.csr -sha1 -passin pass:password -CA ca.pem -CAkey ca.pem -CAcreateserial -out signedcert.pem -days 1095

The “ca.pem” file referenced here is the certificate authority pem file that contains both the CA’s certificate and private key. There is a self-generated one by default in $SPLUNK_HOME/etc/auth/ca.pem. The “password” in the “-passin pass:password” argument is the password to the CA’s private key. You can remove the “-pass” argument if needed and type your CA’s private key password when prompted.

IMPORTANT : Splunk uses SSL certificates in PEM format. If your certificate authority provides you with certificates in DER or PKCS7 certificates, they will need to be converted using openssl.

  • Check that your new certificate contains the expected issuer and subject information :

# openssl x509 -in myServerPublicCertificate.pem -text

The “Issuer” entry should refer to your CA’s information. The “Subject” entry should show the information (country name, organization name, CN, etc) that you entered when creating the CSR earlier.

  • Consolidate the signed server certificate, the server private key and the CA’s public key in a single PEM file :

# cat myServerPublicCertificate.pem myServerPrivateKey.key myCAPublicCertificate.pem > myServerCertificate.pem

Once created, the contents of this file should contain the indexer’s certificate (x509 format), the indexer private key (RSA format) and the certificate authority public key (x509 format), in that order. Here's an example :

        -----BEGIN CERTIFICATE-----
        MIICUTCCAboCCQCscBkn/xey1TANBgkqhkiG9w0BAQUFADBtMQswCQYDVQQGEwJV
        ...
        <Server Certificate>
        ...
        8/PZr3EuXYk1c+N5hgIQys5a/HIn
        -----END CERTIFICATE-----
        -----BEGIN RSA PRIVATE KEY-----
        Proc-Type: 4,ENCRYPTED
        DEK-Info: DES-EDE3-CBC,CFCECC7976725DE5
        
        S+DPcQ0l2Z1bk71N3cBqr/nwEXPNDQ4uqtecCd3iGMV3B/WSOWAQxcWzhe9JnIsl
        ...
        <Server Private Key – Passphrase protected>
        ...
        -----END RSA PRIVATE KEY-----
        -----BEGIN CERTIFICATE-----
        MIICUTCCAboCCQCscBkn/xey1TANBgkqhkiG9w0BAQUFADBtMQswCQYDVQQGEwJV
        ...
        <Certificate Authority Public Key>
        ...
        8/PZr3EuXYk1c+N5hgIQys5a/HIn
        -----END CERTIFICATE-----


IMPORTANT : If there is a certificate chain involved, the public certificates that constitute that chain should be listed in the PEM file in decreasing order of hierarchy right after the server private key. The chain should end with the certificate authority public key, sometimes referred to as the root certificate.

IMPORTANT : You are free to use a different server certificate for the forwarders than for the indexers. In that case, repeat these steps to create a new certificate (forwarder.pem for example), distribute it to the forwarders and define as the value of "sslCertPath" in outputs.conf.
You can also create and configure one server certificate with a unique common name (CN) per indexer. In that case, the forwarder-side configuration gets a bit more complicated, requiring one server-specific stanza per indexer. More on this in step 3.


2 - Set up the indexer(s) to use the new server certificate we just created and signed to listen for Splunk to Splunk traffic on a given port :

  • If they aren't present there, copy the server certificate myServerCertificate.pem and the CA public certificate myCAPublicCertificate.pem into $SPLUNK_HOME/etc/certs/ on the indexer(s) you are intending to configure.


  • In $SPLUNK_HOME/etc/system/local/inputs.conf (or in the appropriate directory of any app you are using to distribute your forwarding configuration), set up the following stanzas :

[SSL]
rootCA = $SPLUNK_HOME/etc/certs/myCAPublicCertificate.pem
serverCert = $SPLUNK_HOME/etc/certs/myServerCertificate.pem
password = server_privkey_password
requireClientCert = false

[splunktcp-ssl:9997]
compressed = true

As you can see, we will be using port 9997 to receive data from the forwarders.

  • IMPORTANT NOTE ABOUT "requireClientCert" :

As of Splunk 4.2.4, setting "requireClientCert = true" in the indexer's inputs.conf will cause forwarding to fail! A bug (SPL-37637) is currently open to address this issue. In the meantime, keep requireClientCert set to "false".

Setting "requireClientCert = true" would requires the following conditions to be met :

a) "rootCA" must point to a file containing the CA's public key. In our example, it's the myCAPublicCertificate.pem file that must be provided by your CA..
b) The forwarder's server certificate defined by "sslCertPath" in outputs.conf (see step 3) is signed by that CA.
c) The forwarder has the password to read his own certificate ("sslPassword" in outputs.conf, as defined in step 4). This password is "server_privkey_password" in our example.

The purpose of this setup is to ensure that only forwarders that you have distributed a signed certificate to can connect to this indexer.

  • Restart splunkd after making these changes.

# $SPLUNK_HOME/bin/splunk restart splunkd

Note that the server certificate pass phrase will be hashed and stored in $SPLUNK_HOME/etc/system/local/inputs.conf, overwriting the clear-text value of "password" if it was defined there. If "password" was defined in clear-text in an inputs.conf located in an app, it *will not* be hashed there and will still be present in clear text in that location. This doesn't matter too much in this case since the pass phrase for the default server certificate is well known.

3 - Set up the forwarder(s) to use the new self-signed server certificate we just created and to send Splunk to Splunk traffic to the indexer(s) receiving port :

  • If they aren't present there, copy the server certificate myServerCertificate.pem and the CA public certificate myCAPublicCertificate.pem into $SPLUNK_HOME/etc/certs/ on the forwarders you are intending to configure.


  • Define the following stanzas in $SPLUNK_HOME/etc/system/local/outputs.conf (or in the appropriate directory of any app you are using to distribute your forwarding configuration) :

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = 10.1.12.112:9997
compressed = true
sslRootCAPath = $SPLUNK_HOME/etc/certs/myCAPublicCertificate.pem
sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate.pem
sslPassword = default
sslVerifyServerCert = true

In our example, we have one indexer with IP address 10.1.12.112. If you are distributing data to several indexers, you can simply add their HOST:PORT address as a comma-separated list in the "server" configuration parameter of the target group definition stanza.

  • IMPORTANT NOTE ABOUT COMMON-NAME CHECKING:

If you have created one server certificate per indexer and you have set a unique common name (CN) in each indexer certificate to be checked by the forwarders, you will need to configure one [tcpout-server://HOST:PORT] configuration stanza per indexer in outputs.conf. This is so that you can specify which common name to check for which indexer.
Here's an example where our forwarders will check that the indexer @10.1.12.112 displays a server certificate with CN="phobos" and that the indexer @10.1.12.113 displays a server certificate with CN="deimos". If one of those conditions is not matched, the forwarder will refuse to send data to that indexer :

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = 10.1.12.112:9997
compressed = true

[tcpout-server://10.1.12.112:9997]
sslRootCAPath = $SPLUNK_HOME/etc/certs/myCAPublicCertificate.pem
sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate.pem
sslPassword = server_privkey_password
sslVerifyServerCert = true
sslCommonNameToCheck = phobos

[tcpout-server://10.1.12.113:9997]
sslRootCAPath = $SPLUNK_HOME/etc/certs/myCAPublicCertificate.pem
sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate.pem
sslPassword = server_privkey_password
sslVerifyServerCert = true
sslCommonNameToCheck = deimos

  • IMPORTANT NOTE ABOUT "sslVerifyServerCert" :

In both cases, we have set "sslVerifyServerCert = true". This requires the following conditions to be met :

a)"sslRootCAPath" must point to a file containing the CA's public key.
b) The indexer's "serverCert" (as defined in inputs.conf on step 2) is signed by that CA.
c) The indexer has the password to read his own certificate ("password" in inputs.conf, as defined in step 2).
d) The common name on the indexer's server certificate ("serverCert" in inputs.conf, as defined in step 2) matches the "sslCommonNameToCheck" setting.

This setup ensures that the forwarder will only send data to an indexer presenting a certificate signed by the known CA and that this certificate was issued to someone with the expected name (CN), i.e., so that the client only sends to someone to whom that certificate is specific. The name doesn't have to match the DNS, hostname, servername, or IP or anything outside of these files. (Thus this is not subject to DNS SSL attacks). All that needs to match is the name in the Splunk configuration file ("sslCommonNameToCheck") and the common name signed into the indexer's certificate.

  • Restart splunkd after making these changes.

# $SPLUNK_HOME/bin/splunk restart splunkd

Note that the server certificate pass phrase will be hashed and stored in $SPLUNK_HOME/etc/system/local/outputs.conf, overwriting the clear-text value of "sslPassword" if it was defined there. If "sslPassword" was defined in clear-text in an outputs.conf located in an app, it *will not* be hashed there and will still be present in clear text in that location. This doesn't matter too much in this case since the pass phrase for the default server certificate is well known.


4 - Check for a successful connection in splunkd.log :

  • This is what you should see during the indexer start-up sequence in $SPLUNK_HOME/var/log/splunkd.log :

02-06-2011 19:19:01.552 INFO TcpInputProc - using queueSize 1000
02-06-2011 19:19:01.552 INFO TcpInputProc - SSL cipherSuite=ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
02-06-2011 19:19:01.552 INFO TcpInputProc - supporting SSL v2/v3
02-06-2011 19:19:01.555 INFO TcpInputProc - port 9997 is reserved for splunk 2 splunk (SSL)
02-06-2011 19:19:01.555 INFO TcpInputProc - Port 9997 is compressed
02-06-2011 19:19:01.556 INFO TcpInputProc - Registering metrics callback for: tcpin_connections

  • This is what you should see during the forwarder start-up sequence in $SPLUNK_HOME/var/log/splunkd.log :

02-06-2011 19:06:10.844 INFO TcpOutputProc - Retrieving configuration from properties
02-06-2011 19:06:10.848 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
02-06-2011 19:06:10.848 INFO TcpOutputProc - found Whitelist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
02-06-2011 19:06:10.848 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
02-06-2011 19:06:10.850 INFO TcpOutputProc - Will retry at max backoff sleep forever
02-06-2011 19:06:10.850 INFO TcpOutputProc - Using SSL for server 10.1.12.112:9997, sslCertPath=/opt/splunk/etc/aut/server.pem
02-06-2011 19:06:10.854 INFO TcpOutputProc - ALL Connections will use SSL with sslCipher=
02-06-2011 19:06:10.859 INFO TcpOutputProc - initializing single connection with retry strategy for 10.1.12.112:9997


  • And this is what a successful connection attempt will look like indexer-side :

02-06-2011 19:19:09.848 INFO TcpInputProc - Connection in cooked mode from 10.1.12.111
02-06-2011 19:19:09.854 INFO TcpInputProc - Valid signature found
02-06-2011 19:19:09.854 INFO TcpInputProc - Connection accepted from 10.1.12.111

  • ...and forwarder-side :

02-06-2011 19:19:09.927 INFO TcpOutputProc - attempting to connect to 10.1.12.112:9997...
02-06-2011 19:19:09.936 INFO TcpOutputProc - Connected to 10.1.12.112:9997


5 - Troubleshooting :

  • First, check in $SPLUNK_HOME/var/log/splunk/splunkd.log on both ends for errors. On the indexer, check for the messages from the TCP input processor "TcpInputProc", and on the forwarder, check the messages from the TCP output processor "TcpOutputProc".
  • In general, it is a good idea to increase the logging level of the appropriate processors on the indexer and the forwarder in $SPLUNK_HOME/etc/log.cfg.
    On the forwarder, set "category.TcpOutputProc=DEBUG", on the indexer set "category.TcpInputProc=DEBUG". Restart Splunk for these to take effect and observe the start-up sequence for the pertinent component. Most configuration issues are explicitly revealed by this method.
  • Check the SSL configuration as it is seen by Splunk using btool.
    • On the indexer :
      $SPLUNK_HOME/bin/splunk cmd btool inputs list --debug
    • On the forwarder :
      $SPLUNK_HOME/bin/splunk cmd btool outputs list --debug
  • Make sure that the certificates are readable by the user that Splunk runs as. Indexer-side, two common problems are :
    • The path to the server certificate file set as the value of "serverCert" in inputs.conf is wrong, or the file cannot be read. This will generate the following error :
      12-16-2010 16:07:30.965 ERROR SSLCommon - Can't read certificate file /opt/splunk/etc/auth/server.pem errno=33558530 error:02001002:system library:fopen:No such file or directory
    • The password to the RSA private key contained in the server certificate file is wrong. This password is set as the value of "password" in inputs.conf. This will generate the following error :
      12-07-2010 07:56:45.663 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem

On *nix, you can manually test the password of the RSA key contained in the file by running the following openssl command :
# openssl rsa -in /opt/splunk/etc/auth/server.pem -text

The same can be done on Windows with the openssl binary that ships with Splunk :
C:\Program Files\Splunk\bin>openssl.exe rsa -in "c:\Program Files\Splunk\etc\auth\server.pem" -text

  • More information regarding the configuration of splunk2splunk SSL connections can be found here in the online documentation :

http://docs.splunk.com/Documentation/Splunk/latest/Security/AboutsecuringyourSplunkconfigurationwithSSL

The appropriate sections of the spec files for inputs.conf and outputs.conf are also a very good resource :

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf

These files can be found in $SPLUNK_HOME/etc/system/README.

Personal tools
Hot Wiki Topics


About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk