Deploy:Migrating a Splunk Install
From Splunk Wiki
Migrating a Splunk installation
Migrating of Splunk installations usually involves some combination of three related concepts
- Upgrading Splunk across releases of the Splunk product, from one version to another.
- Moving a Splunk installation to a new operating system, or new architecture.
- Moving a Splunk installation to a new computer (or VM, or container).
While these are different concepts, often several of them come into play at once. For example, when changing operating systems and architectures, this usually involves changing computers, and when migrating an install, it often seems like a good time to upgrade Splunk to a new version.
Note: Splunk Inc. does not test upgrades from every Splunk version released to every other Splunk version released. To avoid upgrading problems, we recommend you upgrade to each invidual major release, for example 3.0, 3.1, 3.2, in turn.
The most common situation is an upgrade.
The contract of Splunk installs and local modifications is:
- We do not ship any files that we expect users to directly modify
- We will not overwrite any file that we expect users to modify or create
This enables a very simple upgrade process, you can always just overwrite your install with a new version and keep your changes. (If you are not sure if your installation has followed the contract, it's not hard to check. Unpack a splunk zip in a new place and diff the files.) If we ever violate this contract, please file a support ticket to get the behavior fixed.
The process to upgrade is a little different on Windows versus Unix. On Windows, the upgrade is handled by an installer program. On Unix the process is, at its simplest:
- Shut down splunk:
- unpack the new splunk installation onto the splunk directory:
cd $SPLUNK_HOME/..; tar xvzf ~/downloads/splunk-3.4.5-47883-linux-intel.tar.gz
- Start up the new splunk version:
On startup, the first-time-run scripts attempt to ensure that your configuration will work fine on the new version and will prompt you for situations which may require manual consideration.
Operating system moves, or architecture changes
Similar to upgrades, the idea of an install of updating a Splunk install for a new architecture or new operating system is to just unpack the appropriate splunk version on top of the existing install.
Migrating a splunk install to a new operating system builds on this, with some caveats. The general idea is as follows.
- Shut down your old Splunk install:
- Copy (via whatever means, scp, rsync, tar, zip) the Splunk install directory to the desired install location on the new computer.
- Unpack or install the Splunk tar or package for new system on top of the copied files.
- Start it up.
You can even "upgrade" to the same version you were already using with this method, just to validate that the operating system transition was effective.
- If you have data that was indexed under Splunk versions prior to 4.2 (i.e., 4.1.x and lower), the splunk index files that were created under that version are endian sensitive. For this data, Solaris Sparc and MacOS X and AIX PowerPC have compatible index formats, and similarly, all supported platforms using Intel x86 and AMD 64 / Intel EMT64 have compatible indexes; but you cannot take an index from Sparc to x86 and vice versa. If all the data that you are migrating was indexed under version 4.2 or later, then it is compatible with all platform versions of Splunk.
- The windows uninstaller will erase the splunk directory entirely, so switching from 32bit windows to 64bit windows requires special steps.
- Splunk config files use the '/' for directory seperator on Unix, and '\' on Windows. Config settings which refer to directory names will need to be modified.
- If you transition from a 64bit arch to a 32bit arch (eg amd64/x64 to x86) you may experience search performance issues, because the buckets will be larger than desirable for the available memory space.
Moving to a new computer / environment
The difficult part of migrations is integrating with the new environment. Splunk quite actively receives data from the outside world, and frequently sends data to the outside world. This means you must arrange for data input, alerting, scripts, custom programs feeding and run by Splunk to all work on the new system.
Points to consider:
- Most importantly, if the new machine will be using a different hostname or IP address, ensure that other systems which you expect to communicate with Splunk are updated to reflect the change, including:
- Splunk forwarders
- Distributed search nodes which will request searches of this node
- Network based inputs such as direct syslog inputs
- Any health monitoring tools you use to verify Splunk operation
- If Splunk is installed in a different directory on the new environment, update SPLUNK_HOME in etc/splunk-launch.conf, and review all of your configuration for possible use of absolute paths which must be updated.
- If you do not store your Splunk indexes in the default location of ($SPLUNK_HOME/var/lib/splunk), then also update SPLUNK_DB in etc/splunk-launch.conf
- If you have any scripted inputs, review them. Verify the paths used to invoke them, whether they will run on the new arch or operating system at all, and whether their dependencies and data inputs are set up and available where they expect them. Test them outside splunk by seeing that something reasonable is sent to standard out.
- If you have scripted alerts, verify them in the same general manner as scripted inputs.
- If using scripted authentication, you must verify this in the same way.
- Again for script-based lookups.
- If you have email-based alerting, ensure your mailserver will accept mail from the new system, and ensure that the new system will have routing, DNS and firewall clearances to reach your mailserver.
- If using distributed search, be certain that you can reach and be reached by the other distributed nodes.
- If you wish splunk to start on boot, these settings are located outside the Splunk installation so will not be copied over. Be sure to run
splunk enable boot-startor manually enable for it to boot on startup.