From Splunk Wiki

Jump to: navigation, search

Troubleshoot Indexing

Splunk can encounter certain situations which may prevent it from indexing more data. Here are some tips for how to troubleshoot an indexing problem.

1 - Is Splunk capable of searches?

  • If not, then there may potentially be another underlying problem

2 - Does Splunk have enough disk space?

  • Run a df on any partitions which you have indexes, if space is lower than 2GB, indexing has probably been paused.
  • You can also query splunkd.log for any warning/error messages indicating disk is full (the exact message may vary from release to release)

3 - Is this instance also forwarding?

  • Run the following splunk command to see if forwarding (outputs) are configured:
  • ./splunk cmd btool outputs list
  • If outputs are configured, make sure the connection from this server to the indexer is not blocked. You can also just disable forwarding.

4 - Is Splunk having trouble "optimizing"?

  • Examine your indexes. Check the tsidx count in each of the buckets (hot_xyz or db_xyz directories). If any individual buckets contains more than 100 tsidx files and the number is not shrinking, your index is being throttled until splunk-optimize can run to decrease this number down. This may point to a problem w/ splunk-optimize and/or locking issue with tsidx.lock files.

5 - Are your indexes disabled?

  • Run the following command to see if any are disabled:
  • ./splunk cmd btool indexes list | grep disabled
Personal tools
Hot Wiki Topics

About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk