Useful reports on Splunk metrics

Splunk stores metrics on its own performance and what was indexed within its logs and these are discussed in various places in the docs. This page consolidates some of them in one place so that they can easily be found and used. Feel free to contribute more of these to share with others.

Indexing Performance

Plot the average instantaneous events per second and KB per second indexed over time:

index=_internal "group=thruput" | timechart avg(instantaneous_eps) avg(instantaneous_kbps)

Plot the average events per second and KB per second indexed over time:

index="_internal" metrics | fields eps, kbps|timechart avg(eps), avg(kbps)

Number of Events indexed over time:

index=_internal eps group=per_source_thruput NOT filetracker Metrics | eval events=eps*kb/kbps | timechart sum(events)

Plot index latency over time:

* | eval lag = _indextime - _time | timechart perc10(lag) median(lag) perc90(lag)

CPU Time Used by Splunk

See the CPU utilization of Splunk processes in a bar chart sorted in descending order:

index=_internal "group=pipeline" | stats sum(cpu_seconds) as totalCPUSeconds by processor | sort totalCPUSeconds desc

CPU Utilization for the indexer processor (or use any processor you wish):

index=_internal "group=pipeline" processor=indexer| stats sum(cpu_seconds) by _time

Forwarders

List of Forwarders that have connected in the last 3 minutes:

index=_internal group="tcpin_connections" startminutesago=3 | stats count(sourceHost) by sourceHost

Average TCP KB per second per connected forwarder

index=_internal group="tcpin_connections" | stats avg(tcp_KBps) by sourceHost

Average TCP Events per second per connected forwarder

index=_internal group="tcpin_connections" | stats avg(tcp_eps) by sourceHost

How much was indexed

These will limit themselves to 10 sources by default. To increase this number, change your local copy of limits.conf to have a higher value for maxseries under the metrics stanza.

maxseries = 10

KBs indexed per each source over time:

index=_internal group=per_source_thruput | stats sum(kb) by series

KBs indexed per sourcetype over time:

index=_internal per_sourcetype_thruput | timechart sum(kb) by series useother=false

KBs indexed per host over time:

index=_internal per_host_thruput | timechart sum(kb) by series useother=false

KBs indexed per index over time:

index=_internal per_index_thruput | timechart sum(kb) by series useother=false

Total KBs indexed per host:

index=_internal per_host_thruput | chart sum(kb) by series useother=false

Daily volume indexed per host in MB:

index=_internal todaysBytesIndexed LicenseManager -Audit NOT source=*web_service.log NOT source=*web_access.log | eval Daily_Indexing_Volume_in_MBs = todaysBytesIndexed/1024/1024 | timechart avg(Daily_Indexing_Volume_in_MBs) by host