From Splunk Wiki

< Back to Best Practices

under construction

This document is not complete - do not perform any of this without consulting Splunk Support.

Refer to "Move an Index" in the Admin Manual for the supported procedure.

How to move an index from one Splunk installation to another

This topic covers how to move an index from one Splunk installation to another. Before proceeding, you should be familiar with the following:

• General Splunk administration
• Creating and managing a Splunk index
• database buckets
• Unix commands

Overview

There are a few scenarios where you might want to move an index. The most basic is moving an existing index into a new installation. At the other end of the spectrum, you may be moving an old 3.x index to a working 4.x installation of Splunk.

   1. Roll the hot buckets to warm
   2. Physically move the index components to the new location
   3. Scrub the bucket IDs if necessary
   4. Point Splunk at the newly moved index

Step 1 - Roll the hot buckets to warm

Depending on your scenario, choose one of the following options.

In version 4.1.x, you can run the following command to roll the hot DBs to warm:

./splunk _internal call /data/indexes/main/roll-hot-buckets -auth admin:password

In version 4.0.x, you can run the following command to roll the hot dbs to warm:

./splunk search "| debug cmd=roll index=main"

You will see that the hot_v# directories are now pushed to warm buckets (db_#_#_id).

For version 3.x, you can run the following command to roll the hot dbs to warm:

./splunk search '| oldsearch !++cmd++::roll' -auth splunk

./splunk search ' | oldsearch index=<INDEX_NAME> !++cmd++::roll' -auth admin:<ADMIN_PASSWORD>

You will know the db has rolled when you no longer see contents in db-hot. Instead, you should find a very recently time stamped bucket.

Step 2 - Copy the dbs to the new location

There are many ways to get the data to the new installation. Perform the step which is applicable to your installation.

• Splunk instance is on the same machine: Rename/Move (atomically) the data to the new installation. For example:

mv /opt/oldsplunk/var/lib/splunk/defaultdb /opt/newsplunk/var/lib/splunk/defaultdb

If you are moving archived data or a portion of a database, then you should stop now and contact support.

Step 3 - Start Splunk Point Splunk at the index

Modify your indexes.conf file to point at this location There are two scenarios you should consider when pointing the installation at the migrated database:

• You are moving the complete index (hot/warm/cold)
• You are moving a portion of the index

Advanced Usage

For migrations which are not completely new installations or include archived data from a separate installation, read on...

Scrubbing the bucket IDs

This step is not required if you are moving to a brand new installation which has never been started. You should not perform this without consulting support

Once you have your buckets ready to be moved (Step 2), you must make sure that the bucket IDs do not conflict. Bucket IDs are identified by the final number in the bucket directory name. For example, the following bucket has an ID of 31:

db_1249455599_1249369200_31

Carefully note the number range of all of the buckets from each index to be moved. If any of the following are true:

• there are multiple buckets with the same ID
• the number range for the buckets is overlapping
• the number range for the existing installation is LOWER than the one you are trying to move

you cannot merge these buckets in the same location without modification. See below for more information.

If the buckets do not fulfill the above criteria, you can move them by performing the following:

1. Stop Splunk

2. Move the db (archived bucket) to the new location. For example:

mv /opt/oldsplunk/var/lib/splunk/defaultdb/db/colddb/db_1249455599_1249369200_31 /opt/newsplunk/var/lib/splunk/defaultdb/colddb/db_1249455599_1249369200_31

3. Start Splunk

If the bucket IDs require modification, you may wish to instead create a new index for the desired buckets. In 4.x you can search across multiple indexes, either explicitly or by changing your default index configuration.

To correctly merge buckets from multiple instances with conflicting IDs, you need to verify and/or renumber not only any buckets moved in from elsewhere but those already existing in the active index. Renumber buckets by changing the last number in the bucket directory name only:

db_1200135653_1194130890_6

becomes

db_1200135653_1194130890_14

The bucket ID must be unique and also different sets of buckets from different indexes cannot have overlapping ranges. So if you have one set with IDs 45-85, you cannot have another set with 60-100 even if any individual bucket in each does not have the same ID. Also, since new buckets are created with strictly increasing IDs, the buckets from the active instance must have the highest bucket IDs. Bucket IDs in each set need not be strictly sequential, but the number range must be contiguous.

Example

• Buckets 12-123 from indexer A
• Buckets 38-53 from indexer B
• Buckets 1-49 from active indexer C

To merge all these buckets in the warm directory on indexer C:

1. Manually roll from hot to warm on all indexers and shut down.

2. Leave the buckets from indexer A as is, because there are a lot of them but all in one range that can be worked around.

3. Rename the buckets from indexer B to 200-216

4. Rename the buckets already in the indexer C warm directory to 300-349

5. Remove the .metaManafest and .bucketManifest files in the indexer C warm directory to force rebuild of these files on next start.

6. Restart Splunk.

Now new buckets will be created beginning with the next highest number in sequence.

Please note: rechecking the buckets may take some time (over 30 seconds per bucket has been seen) and since checking indexes happens early in Splunk startup, many services such as data inputs and splunkweb will not start until it finishes! Watch splunkd.log to determine the progress.