From Splunk Wiki

< Back to Best Practices

How to deploy multiple instances of Splunk to a single machine

This topic gives a step-by-step procedure for installing a second instance of Splunk on a machine. To install additional instances, repeat these instructions.

Install your second instance

• Choose an installation location that's different from the location of your existing Splunk instance(s).

• Install Splunk into that directory using the appropriate Installation Manual instructions.
  • A duplicate RPM install may not work on linux. Consider using that tar installer.

What to do with configuration files

Make configuration changes using the following information:

Do you want the second/subsequent Splunk instance to behave and treat data the same way as the first instance?

• If yes – copy all .conf files EXCEPT inputs.conf and web.conf from $SPLUNK_HOME1/etc/system/local to $SPLUNK_HOME2/etc/system/local, and any other applications/bundles you are currently running.
• If no – do not copy any .conf files between your indexes.

Q: Why not inputs.conf?

A: For network inputs, you must specify different ports. For file inputs, you will index the same data twice.

Q: Why not web.conf?

A: You must specify different ‘httpport’ and ‘mgmtHostPort’ settings.

Changes to splunk_server name

In versions 3.4.9 and prior:

Change ‘servername’ for the second instance in $SPLUNK_HOME/etc/myinstall/splunkd.xml. By default, the installation populates this with your server ‘hostname' on first time run. To differentiate between your installations and to allow distributed search to work, you should change the name of the new Splunk instance.

To accomplish this before first time run, copy $SPLUNK_HOME/etc/myinstall/splunkd.xml.default to $SPLUNK_HOME/etc/myinstall/splunkd.xml and edit the 'servername' before starting the instance.

In versions 3.4.10 and later, and 4.0 and later:

Change the serverName parameter in $SPLUNK_HOME/etc/system/local/server.conf. By default, the installation populates this with your server ‘hostname' on first time run. To differentiate between your installations and to allow distributed search to work, you should change the name of the new Splunk instance.

To accomplish this before first time run, edit $SPLUNK_HOME/etc/system/local/server.conf and create or edit the 'serverName' before starting the instance.

What else do I need to consider?

Review these sections to make sure you've covered everything.

Splunk Web

You can disable Splunk Web in the new instance if you won't need it to operate and maintain that instance.

Startup script (e.g. /etc/init.d/splunk)

You must modify this script to include the new Splunk instance if you require automatic startup of both instances.

Deployment server/client

Your new instance may need to be included in your deployment server/client configuration. Refer to the deployment server documentation for more information.

Distributed Search

Your new instance may need to be included in your distributed search configuration on the search node(s). Refer to the distributed search documentation for more information.

Forwarding Data

You may need to update forwarding configurations on Splunk forwarders or other devices to include the new Splunk instance. Refer to the forwarding documentation for more details.

Saved Search Alerts

If you copied savedsearches.conf from another instance, ensure any scripted/email alerts are configured properly using the information in the alerting documentation for more details.

Resource Usage

Memory - By default, bucket size on a 64-bit instance is 10000MB. If you have more than one instance on a server, you don't want them all running with default settings. You need tune your bucket size so that your hot-DB's don't consume all available memory.

CPU - By default, there will be one index-thread created for each index. You may want to ensure this is not increased and limit the number of custom indexes created. Splunk will also create up to 6 concurrent splunk-optimize processes for an index, you should also tune this number lower to avoid a performance impact

Tune all of these settings in indexes.conf