Community:TroubleshootingIndexing

Troubleshoot Indexing

Splunk can encounter certain situations which may prevent it from indexing more data. Here are some tips for how to troubleshoot an indexing problem.

1 - Is Splunk capable of searches?

• If not, then there may potentially be another underlying problem

2 - Does Splunk have enough disk space?

• Run a df on any partitions which you have indexes, if space is lower than 2GB, indexing has probably been paused.
• You can also query splunkd.log for any warning/error messages indicating disk is full (the exact message may vary from release to release)

3 - Is this instance also forwarding?

• Run the following splunk command to see if forwarding (outputs) are configured:
• ./splunk cmd btool outputs list
• If outputs are configured, make sure the connection from this server to the indexer is not blocked. You can also just disable forwarding.

4 - Is Splunk having trouble "optimizing"?

• Examine your indexes. Check the tsidx count in each of the buckets (hot_xyz or db_xyz directories). If any individual buckets contains more than 100 tsidx files and the number is not shrinking, your index is being throttled until splunk-optimize can run to decrease this number down. This may point to a problem w/ splunk-optimize and/or locking issue with tsidx.lock files.

5 - Are your indexes disabled?

• Run the following command to see if any are disabled:
• ./splunk cmd btool indexes list | grep disabled