From Splunk Wiki

Jump to: navigation, search

A few variations on the message

  ... PipelineComponent WARN 0x234deadbeef took too long (234234 millieseconds) in callbackRunnerThread.

have been in the splunk product. What's this about?

Essentially this message is smoke. It doesn't know what the problem is, but it's pretty sure something is wrong.

callbackRunnerThread is asked to perform work on the behalf of other requesters. It doesn't really know what the work is, it's just pretty sure that the work isn't supposed to take a really long time. We've resolved at least one problem in splunk as a result of these messages, so they're worth investigating, but they don't necessarily mean there's a problem with the splunk program. If the system is just running really slow (overburdened, disk contention, heavily into swap, etc), then the callback may take a really long time completely out of splunk's control. So definitely review the system picture as well as what is working OK in your splunk environment, and what is not.

More detail: callbackRunner is a single-threaded scheduling mechanism. Requesters register callbacks they would like invoked and scheduling for when these calls should occur, and this thread invokes those calls sequentially as best it can fit the schedule. Because of this design, it's essential that the callbacks be brief, so we can be pretty certain that a long time to complete a callback indicates an undesirable situation, and thus produce this message.

Personal tools
Hot Wiki Topics

About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk