Community:MoveIndexes

From Splunk Wiki

Jump to: navigation, search

< Back to Best Practices


WARNING - This process no longer applies to versions 5.x and 6.x of splunk. For the most up to date version of these docs please see the following link

http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Moveanindex

under construction

This document is not complete - do not perform any of this without consulting Splunk Support.

Refer to "Move an Index" in the Admin Manual for the supported procedure.

How to move an index from one Splunk installation to another

This topic covers how to move an index from one Splunk installation to another. Before proceeding, you should be familiar with the following:

  • General Splunk administration
  • Creating and managing a Splunk index
  • database buckets
  • Unix commands

Overview

There are a few scenarios where you might want to move an index. The most basic is moving an existing index into a new installation (for example, to replace an aging server). At the other end of the spectrum, you may be moving an old 3.x index to a working 4.x (or newer) installation of Splunk.

The basic steps are:

  1. Roll the hot buckets to warm
  2. Physically move or copy the index components to the new location
    1. Scrub the bucket IDs if necessary
  3. Point Splunk at the newly moved index and/or rebuild metadata and manifests

Step 1 - Roll the hot buckets to warm

On the source indexer, you will need to roll your hot buckets into warm buckets. To do so, execute the commands appropriate to your version of Splunk. (The commands given assume your working directory is %SPLUNK_HOME%/bin)

  • Version 4.1.x:

./splunk _internal call /data/indexes/main/roll-hot-buckets -auth admin:password

(Replace "main" with the name of the index.)

  • Version 4.0.x:

./splunk search "| debug cmd=roll index=main"

(Replace "main" with the name of the index.)

You will see that the hot_v# directories are now pushed to warm buckets (db_#_#_id).

  • Version 3.x:

./splunk search '| oldsearch !++cmd++::roll' -auth splunk

./splunk search '| oldsearch index=<INDEX_NAME> !++cmd++::roll' -auth admin:<ADMIN_PASSWORD>

You will know the db has rolled when you no longer see contents in db-hot. Instead, you should find a very recently time stamped bucket.

Step 2 - Copy the index buckets to the new location(s)

Move or copy the buckets to their new location(s) atomically. On *nix, you can use "mv".

For example, if the both instances are on the same *nix server in /opt/oldsplunk and /opt/newsplunk, you can use mv:

mv /opt/oldsplunk/var/lib/splunk/defaultdb /opt/newsplunk/var/lib/splunk/defaultdb

This is assuming that the index does not already exist under "newsplunk". If it does, and you aren't careful, you can break your index! Thus, a "Warning": If you are moving archived data or a portion of a database, then you should stop now and contact support. Care must be taken to ensure there are no duplicate bucket IDs in your destination or things will not work right.

Step 2.1 - Scrub the bucket IDs

Bucket scrubbing describes the process of ensuring that no two buckets have the same bucket ID at your destination. If you are moving an index to an indexer has never been started or has never known about the index you are moving, you do not need to worry about this step. Proceed to the next step.

If you are merging indexes or restoring archived data, "advanced usage" section. You should probably also contact Splunk Support for assistance.

Step 3 - Point Splunk at the newly moved index

Again, this is used in a scenario where you moving an index to a server that does not already contain an identically named index.

Modify your indexes.conf file to point at the location you moved the index to in the previous step.

There are two scenarios you should consider when pointing the installation at the migrated database:

  • You are moving the complete index (hot/warm/cold)

Refer to indexes.conf.spec or the admin manual for instructions on creating an appropriate stanza for your newly moved index.

  • You are moving a portion of the index.

If you are adding buckets to an existing index, follow the instructions under "Advanced Usage".

Advanced Usage

For migrations which are not completely new installations or include archived data from a separate installation, read on...

You may also wish to refer to Restore Archived Data

Scrubbing the bucket IDs

This step is not required if you are moving to a brand new installation which has never been started. You should not perform this without consulting support

Once you have your buckets ready to be moved (Step 2), you must make sure that the bucket IDs do not conflict. Bucket IDs are identified by the final number in the bucket directory name. For example, the following bucket has an ID of 31:

db_1249455599_1249369200_31

From Splunkweb you can run "| dbinspect index=main" to see the current details of your running index. This will show you bucket numbers, paths, sizes, etc. This command is useful for verifying that you've completed the process correctly.

Carefully note the number range of all of the buckets from each index to be moved. If any of the following are true:

  • there are multiple buckets with the same ID
  • the number range for the buckets is overlapping
  • the number range for the existing installation is LOWER than the one you are trying to move

you cannot merge these buckets in the same location without modification. See below for more information.

If the buckets do not fulfill the above criteria, you can move them by performing the following:

1. Stop Splunk

2. Move the db (archived bucket) to the new location. For example:

mv /opt/oldsplunk/var/lib/splunk/defaultdb/db/colddb/db_1249455599_1249369200_31 /opt/newsplunk/var/lib/splunk/defaultdb/colddb/db_1249455599_1249369200_31

3. Start Splunk

If the bucket IDs require modification, you may wish to instead create a new index for the desired buckets. In 4.x you can search across multiple indexes, either explicitly or by changing your default index configuration.

To correctly merge buckets from multiple instances with conflicting IDs, you need to verify and/or renumber not only any buckets moved in from elsewhere but those already existing in the active index. Renumber buckets by changing the last number in the bucket directory name only:

db_1200135653_1194130890_6

becomes

db_1200135653_1194130890_14

The bucket ID must be unique and also different sets of buckets from different indexers cannot have overlapping ranges. So if you have one set with IDs 45-85, you cannot have another set with 60-100 even if any individual bucket in each does not have the same ID. Also, since new buckets are created with strictly increasing IDs, the buckets from the active instance must have the highest bucket IDs. Bucket IDs in each set need not be strictly sequential, but the number range must be contiguous.

Example

  • Buckets 12-123 from indexer A
  • Buckets 38-53 from indexer B
  • Buckets 1-49 from active indexer C

To merge all these buckets in the warm directory on indexer C:

1. Manually roll from hot to warm on all indexers and shut down.

2. Leave the buckets from indexer A as is, because there are a lot of them but all in one range that can be worked around.

3. Rename the buckets from indexer B to 200-216

4. Rename the buckets already in the indexer C warm directory to 300-349

5. Restart Splunk.

6. run: ./splunk _internal call /data/indexes/*/rebuild-metadata-and-manifests

(replace * with your index name). Alternatively, restart splunk and the rebuild will happen automatically.

Now new buckets will be created beginning with the next highest number in sequence.

Please note: rechecking the buckets may take some time (over 30 seconds per bucket has been seen) and since checking indexes happens early in Splunk startup, many services such as data inputs and splunkweb will not start until it finishes! Watch splunkd.log to determine the progress.

Personal tools
Hot Wiki Topics


About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk