Community:Run multiple Splunks on one machine

From Splunk Wiki

Jump to: navigation, search

< Back to Best Practices

How to deploy multiple instances of Splunk to a single *nix machine

This topic gives a step-by-step procedure for installing a second instance of Splunk on a machine. To install additional instances, repeat these instructions. Disclaimer: Recommendations such as separate RAID volumes and index replication impact when running multiple instances on same box are not addressed here.

Install your second instance

  • Choose an installation location that's different from the location of your existing Splunk instance(s).

What to do with configuration files

Make configuration changes using the following information:

Do you want the second/subsequent Splunk instance to behave and treat data the same way as the first instance?

  • If yes – copy all .conf files EXCEPT inputs.conf and web.conf from $SPLUNK_HOME1/etc/system/local to $SPLUNK_HOME2/etc/system/local, and any other applications/bundles you are currently running.
  • If no – do not copy any .conf files between your indexes.

Q: Why not inputs.conf?

A: For network inputs, you must specify different ports. For file inputs, you will index the same data twice.

Q: Why not web.conf?

A: You must specify different ‘httpport’ and ‘mgmtHostPort’ settings.

Changes to splunk_server name

In versions 3.4.9 and prior:

Change ‘servername’ for the second instance in $SPLUNK_HOME/etc/myinstall/splunkd.xml. By default, the installation populates this with your server ‘hostname' on first time run. To differentiate between your installations and to allow distributed search to work, you should change the name of the new Splunk instance.

To accomplish this before first time run, copy $SPLUNK_HOME/etc/myinstall/splunkd.xml.default to $SPLUNK_HOME/etc/myinstall/splunkd.xml and edit the 'servername' before starting the instance.

In versions 3.4.10 and later, and 4.0 and later:

Change the serverName parameter in $SPLUNK_HOME/etc/system/local/server.conf. By default, the installation populates this with your server ‘hostname' on first time run. To differentiate between your installations and to allow distributed search to work, you should change the name of the new Splunk instance.

To accomplish this before first time run, edit $SPLUNK_HOME/etc/system/local/server.conf and create or edit the 'serverName' before starting the instance.

What else do I need to consider?

Review these sections to make sure you've covered everything.

Splunk Web

You can disable Splunk Web in the new instance if you won't need it to operate and maintain that instance.

Startup script (e.g. /etc/init.d/splunk)

You must modify this script to include the new Splunk instance if you require automatic startup of both instances.

OR you can create a different init script for your 2nd install; to do this start by backing up the current init script:

mv /etc/init.d/splunk /etc/init.d/splunk.bak

Then set the 2nd Splunk install to auto-start

/opt/splunk_2/bin/splunk enable boot-start -user [user]

Then change the init script just a bit while also creating a different name for it:

sed s/splunkd/splunkd_2/g /etc/init.d/splunk >/etc/init.d/splunk2

Then use chkconfig to add the new init script:

chkconfig --add splunk2

Now copy the backed up init script from your 1st splunk install back to where it belongs:

mv /etc/init.d/splunk.bak /etc/init.d/splunk

Now reboot and both splunk installs should start automagically

  • Make sure they run on different ports too

Deployment server/client

Your new instance may need to be included in your deployment server/client configuration. Refer to the deployment server documentation for more information.

Distributed Search

Your new instance may need to be included in your distributed search configuration on the search node(s). Refer to the distributed search documentation for more information.

Forwarding Data

You may need to update forwarding configurations on Splunk forwarders or other devices to include the new Splunk instance. Refer to the forwarding documentation for more details.

Saved Search Alerts

If you copied savedsearches.conf from another instance, ensure any scripted/email alerts are configured properly using the information in the alerting documentation for more details.

Resource Usage

Memory - By default, bucket size on a 64-bit instance is 10000MB. If you have more than one instance on a server, you don't want them all running with default settings. You need tune your bucket size so that your hot-DB's don't consume all available memory.

CPU - By default, there will be one index-thread created for each index. You may want to ensure this is not increased and limit the number of custom indexes created. Splunk will also create up to 6 concurrent splunk-optimize processes for an index, you should also tune this number lower to avoid a performance impact

Tune all of these settings in indexes.conf



How to deploy multiple instances of Splunk to a single Windows machine

This is NOT a supported procedure. Only do this if you are very comfortable around a Windows environment, including services and permissions.

If you are running a machine with more than 16 cores and more than 16 GB memory, running multiple copies of Splunk may help better utilize the capacity of your machine.

Remember, there are differing limits of CPU, Memory, and IO usage, and depending on your hardware and configuration, one will likely get hit before another. Test to find the optimal number of instances, but 2-3 is a good place to start if you have many cores and GB Memory. Remember that it is all using the same IO, unless you can partition different SAN mounts etc.

For this example, Splunk1 and Splunk2 are used. You can use any directories, just be consistent.


Install the first instance

> msiexec /i C:\Full\Path\to\splunk-ve.rs.ion-build-xbit-release.msi INSTALLDIR="C:\Program Files\Splunk1" LAUNCHSPLUNK=0

This uses the MSI, so any further upgrades (below), repair, or uninstallation will affect this instance ONLY.

If it doesn’t work (due to permissions issues) place that line in a text file, rename it with a .bat or .cmd extension, and run it as Administrator.

Clean and Prep the first instance

Annoyingly, even when LAUNCHSPLUNK=0 is set, some instance-specific files get set by the installer. Time to remove them.

Go to C:\Program Files\Splunk1 and delete the following:

  • \etc\auth\ all but *.default files
  • \etc\system\local\ all but README
  • \etc\system\metadata\local.meta
  • \etc\licenses
  • \var

Create a 0-byte file as \ftr - this ensures Splunk runs its First Time Run check, generates certificates, etc, upon next start. DO NOT start yet.

Create the Second Instance

Copy C:\Program Files\Splunk1 to C:\Program Files\Splunk2

Configure and Differentiate Instances

Copy in to each instance any configs necessary for deployment (such as a yourorg_deploymentclient app) and any specific apps that may differentiate the two instances from one another, such as deployed apps that give different "Deployment Client Names" to each instance, and change port numbers.

As with *nix above, Splunk's default port number on the second instance needs to be changed. Do this in a deployed app that is pre-seeded to the instance now, or if it will never need to be changed, in etc\system\local\web.conf:

  • [settings]
  • mgmtHostPort = 127.0.0.1:8088

Additionally, differentiate the instances' internal names. There are two, one for data (mainly internal logs, but this will become the host field of any data generated by this instance unless otherwise specified) in inputs.conf and the other for Splunk internal functions (like distributed search and license management) in server.conf:

  • \etc\system\local\inputs.conf: [default] host= … add -instance* for appropriate instance name
  • \etc\system\local\server.conf: [general] serverName= … add -instance* for appropriate instance name

Also important, on Windows, is to tell Splunk which services belong to it. This is in etc\splunk-launch.conf, which is not editable through Deployment Server:

  • SPLUNK_SERVER_NAME=Splunkd2 (for Splunk2 instance only, makes “Splunk2\bin\splunk start” start the correct service.)

Disable the Splunkweb services on Splunk2, as they are likely not desired. This can be via an pre-seeded deployment app or in an etc\system\local\web.conf.

If using deployment server you must set the new instance to new use a different client name in deploymentclient.conf.

Be sure to consider the other differentiation items in the *nix section above.

Set up additional Services

Splunk on Windows uses services to run. \bin\splunk.exe start actually contacts the service to start and stop, so it is important that the service names are defined properly (see previous section).

The second instance now needs a service name of its own. To do this, run these commands (yes, there are spaces after the equals signs):

sc create Splunkd2 binPath= "\"C:\Program Files\Splunk2\bin\splunkd.exe\" service" DisplayName= "Splunkd2" start= auto

sc description Splunkd2 "Splunkd is the indexing and searching engine for Splunk, a data platform for operational intelligence. It is required for Splunk instances acting as an indexer. If it is stopped, Splunk will not process data and will be unavailable for search. Splunkweb depends on Splunkd. Please see www.splunk.com for more information. Questions can be submitted to www.splunk.com/answers or for supported customers www.splunk.com/page/submit_issue"

If need be, do the same for splunkweb. Its executable and description can be found in the Services control panel. It appears that post-6.2, the splunkweb service is no longer required.

Start the Instances

C:\Program Files\Splunk1\bin\splunk.exe start C:\Program Files\Splunk2\bin\splunk.exe start

It will ask about a migration, select yes.

You should see both start their own service (Splunkd vs Splunkd2) and Splunk2 may complain about a missing SplunkWeb, if you configured it in etc\splunk-launch.conf but then did not create it. Not a big deal.


They can also be started/stopped by net {start|stop} {splunkd|splunkweb|splunkd2} – but note that this only starts/stops one service at a time, not splunkd and splunkweb at once.

Upgrades

Instance 1

Instance 1 (in this example, C:\Program Files\Splunk1) is easy to upgrade, since it is in the registry and in add/remove programs. Simply use the Splunk MSI installer to upgrade as per the normal instructions.

Instance 2+

Additional instances become more tricky to upgrade, since the MSI installer will only deal with the one in the registry (the original instance).

This is a COMPLETELY unsupported procedure

  • Install, but do not run, the new version of Splunk on ANOTHER windows machine - one that does not have Splunk already installed. (A UniversalForwarder is okay, it is installed as a different service.) Make sure it is the same bit version as your multi-instance machine.
  • > msiexec /i C:\Full\Path\to\splunk-ve.rs.ion-build-xbit-release.msi INSTALLDIR="C:\Program Files\SplunkTemp" LAUNCHSPLUNK=0 /quiet
  • If it doesn’t work (due to permissions issues) place that line in a text file, rename it with a .bat extension, and run it as Administrator.

Then, clean the install by deleting:

  • etc\system\local
  • etc\users
  • etc\licenses
  • etc\auth
  • var\
  • etc\splunk-launch.conf
  • etc\log.cfg
  • etc\openldap\ldap.conf
  • etc\system\metadata\local.meta
  • etc\myinstall\splunkd.xml

And add one zero byte file as C:\Program Files\SplunkTemp\ftr

Then, zip up the whole directory. Copy it to your multi-instance machine and unzip it to a temporary location.

Stop your secondary Splunk instance, copy all the files/folders from the temporary location to the secondary instance location, allowing overwrite and giving permission as needed.

Start Splunk, go through the migration, and enjoy your new version!


This procedure was tested with 4.3.2 x64 as a base multi-install on a Windows 7 64bit laptop. The instances were upgraded to 4.3.3 with the help of a Win2008 R2 64 bit server.

Personal tools
Hot Wiki Topics


About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk