From Splunk Wiki
Troubleshooting scheduled searches
Questions to answer when troubleshooting scheduled searches:
Does the user have permissions to schedule searches?
By default only the Power and Admin roles can schedule searches.
Does the job manager show the last times it ran? Can you view results for the latest search that was run?
Check the time it was dispatched, number of events returned and the status
Dispatched at Owner Application Events Run time Expires Status Actions 10/4/10 4:51:19 PM admin search 0 00:00:00 Oct 4, 2010 5:01:19 PM Done Save | Delete
Does manager show the savedsearch with the next time it is to run?
Maybe the config is garbled, or possibly the cron string is garbled.
Is the search being starved by other saved searches?
Check scheduler.log for SavedSplunker messages.
09-02-2009 10:50:01.034 WARN SavedSplunker - Maximum number (1) of concurrent scheduled searches reached. 6 ready-to-run scheduled searches pending. 09-02-2009 10:50:06.086 WARN SavedSplunker - Maximum number (1) of concurrent scheduled searches reached. 4 ready-to-run scheduled searches pending. 09-02-2009 10:50:11.143 WARN SavedSplunker - Maximum number (1) of concurrent scheduled searches reached. 2 ready-to-run scheduled searches pending.
Also check audit.log to see if it runs sometimes, but perhaps rarely.
Dig deeper into behavior, by enabling logging
[splunkd] category.SavedSplunker = DEBUG
Where to look for ERRORS
Where to look for errors:
- ERROR script - External search command 'runshellscript' returned error code 1.
There is something wrong with your script running. Refer to the following for more info:
More on troubleshooting alerts: Community:TroubleshootingAlertScripts
- ERROR SearchOperator:loadjob - Cannot find artifacts for savedsearch_ident
Usually shows up when one is trying to compare results from a current search to a previous one. The results from the earlier search cannot be found.