From Splunk Wiki

Jump to: navigation, search

How to find a "lost" forwarder

If you have deployed a number of Splunk forwarders and they are all pushing data to Splunk, you might not notice if one of them goes out of service, because the other forwarders are still pushing data to Splunk. You can run the following search to detect forwarders that have been up in the last 24 hours but not in the last 2 minutes. It uses the forwarder heartbeat, which is a feature of Splunk versions 3.2 and later.

index=_internal sourcetype="fwd-hb" starthoursago=24 | dedup host | eval age = strftime("%s","now") - _time | search age > 120 age < 86000 <-This command yields "Error in 'eval' command: Typechecking failed. '-' only takes numbers."

You can set this search up as an alert every several minutes so that Splunk will let you know if any of your active forwarders have not responded in the last 2 minutes.

If you're running a version of Splunk that is later than 3.3', the heartbeat message is not longer sent. Use the following search instead:

index=_internal "group=tcpin_connections" | stats max(_time) as latest by sourceHost | eventstats max(latest) as latest_all | eval lag = latest_all - latest | where lag > 120 | fields sourceHost lag

The following search works in 3.4.5 and finds all hosts who haven't sent a message in the last 24 hours

| metadata type=hosts | eval age = strftime("%s","now") - lastTime | search age > 86400 | sort age d | convert ctime(lastTime) | fields age,host,lastTime

and in 4.0:

| metadata type=hosts | eval age = now() - lastTime | search age > 86400 | sort age d | convert ctime(lastTime) | fields age,host,lastTime

Another 4.0 variant

| metadata type=hosts | sort recentTime desc | convert ctime(recentTime) as Recent_Time


Many of these methods do not account for decommissioned hosts, which you are bound to have after a length of time. These hosts will also show up in the search results, as they also fit the criteria. Incorporating a host tag ('decommisioned', etc) into this search may help with this, but requires you to tag known hosts that are no longer valid.

Personal tools
Hot Wiki Topics

About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk