< Back to Troubleshooting

Troubleshooting scheduled searches

Questions to answer when troubleshooting scheduled searches:

Does the user have permissions to schedule searches?

By default only the Power and Admin roles can schedule searches.

Does the job manager show the last times it ran? Can you view results for the latest search that was run?

Check the time it was dispatched, number of events returned and the status

Dispatched at  	        Owner  	Application  	Events  	Run time  	Expires  	        Status  	Actions
10/4/10 4:51:19 PM 	admin 	search 	        0 	00:00:00 	Oct 4, 2010 5:01:19 PM 	Done 	Save | Delete

Does manager show the savedsearch with the next time it is to run?

Maybe the config is garbled, or possibly the cron string is garbled.

Is the search being starved by other saved searches?

Check scheduler.log for SavedSplunker messages.

09-02-2009 10:50:01.034 WARN  SavedSplunker - Maximum number (1) of concurrent scheduled searches reached. 6 ready-to-run scheduled searches pending.
09-02-2009 10:50:06.086 WARN  SavedSplunker - Maximum number (1) of concurrent scheduled searches reached. 4 ready-to-run scheduled searches pending.
09-02-2009 10:50:11.143 WARN  SavedSplunker - Maximum number (1) of concurrent scheduled searches reached. 2 ready-to-run scheduled searches pending.

Also check audit.log to see if it runs sometimes, but perhaps rarely.

Dig deeper into behavior, by enabling logging

in $SPLUNK_HOME/etc/log-local.cfg:

[splunkd]
category.SavedSplunker = DEBUG

Where to look for ERRORS

Where to look for errors:

• splunkd.log
• python.log
• scheduler.log

Common ERRORS

• ERROR script - External search command 'runshellscript' returned error code 1.

There is something wrong with your script running. Refer to the following for more info:
More on troubleshooting alerts: Community:TroubleshootingAlertScripts

• ERROR SearchOperator:loadjob - Cannot find artifacts for savedsearch_ident

Usually shows up when one is trying to compare results from a current search to a previous one. The results from the earlier search cannot be found.