From Splunk Wiki
Splunk can encounter certain situations which may prevent it from indexing more data. Here are some tips for how to troubleshoot an indexing problem.
1 - Is Splunk capable of searches?
- If not, then there may potentially be another underlying problem
2 - Does Splunk have enough disk space?
- Run a df on any partitions which you have indexes, if space is lower than 2GB, indexing has probably been paused.
- You can also query splunkd.log for any warning/error messages indicating disk is full (the exact message may vary from release to release)
3 - Is this instance also forwarding?
- Run the following splunk command to see if forwarding (outputs) are configured:
- ./splunk cmd btool outputs list
- If outputs are configured, make sure the connection from this server to the indexer is not blocked. You can also just disable forwarding.
4 - Is Splunk having trouble "optimizing"?
- Examine your indexes. Check the tsidx count in each of the buckets (hot_xyz or db_xyz directories). If any individual buckets contains more than 100 tsidx files and the number is not shrinking, your index is being throttled until splunk-optimize can run to decrease this number down. This may point to a problem w/ splunk-optimize and/or locking issue with tsidx.lock files.
5 - Are your indexes disabled?
- Run the following command to see if any are disabled:
- ./splunk cmd btool indexes list | grep disabled