Deploy:Splunk Metric Reports
From Splunk Wiki
Useful reports on Splunk metrics
Splunk stores metrics on its own performance and what was indexed within its logs and these are discussed in various places in the docs. This page consolidates some of them in one place so that they can easily be found and used. Feel free to contribute more of these to share with others.
Indexing Performance
Plot the average instantaneous events per second and KB per second indexed over time:
index=_internal "group=thruput" | timechart avg(instantaneous_eps) avg(instantaneous_kbps)
Plot the average events per second and KB per second indexed over time:
index="_internal" metrics | fields eps, kbps|timechart avg(eps), avg(kbps)
Number of Events indexed over time:
index=_internal eps group=per_source_thruput NOT filetracker Metrics | eval events=eps*kb/kbps | timechart sum(events)
Plot index latency over time:
* | eval lag = _indextime - _time | timechart perc10(lag) median(lag) perc90(lag)
CPU Time Used by Splunk
See the CPU utilization of Splunk processes in a bar chart sorted in descending order:
index=_internal "group=pipeline" | stats sum(cpu_seconds) as totalCPUSeconds by processor | sort totalCPUSeconds desc
CPU Utilization for the indexer processor (or use any processor you wish):
index=_internal "group=pipeline" processor=indexer| stats sum(cpu_seconds) by _time
Forwarders
List of Forwarders that have connected in the last 3 minutes:
index=_internal group="tcpin_connections" startminutesago=3 | stats count(sourceHost) by sourceHost
Average TCP KB per second per connected forwarder
index=_internal group="tcpin_connections" | stats avg(tcp_KBps) by sourceHost
Average TCP Events per second per connected forwarder
index=_internal group="tcpin_connections" | stats avg(tcp_eps) by sourceHost
How much was indexed
These will limit themselves to 10 sources by default. To increase this number, change your local copy of limits.conf to have a higher value for maxseries under the metrics stanza.
maxseries = 10
KBs indexed per each source over time:
index=_internal group=per_source_thruput | stats sum(kb) by series
KBs indexed per sourcetype over time:
index=_internal per_sourcetype_thruput | timechart sum(kb) by series useother=false
KBs indexed per host over time:
index=_internal per_host_thruput | timechart sum(kb) by series useother=false
KBs indexed per index over time:
index=_internal per_index_thruput | timechart sum(kb) by series useother=false
Total KBs indexed per host:
index=_internal per_host_thruput | chart sum(kb) by series useother=false
Daily volume indexed per host in MB:
index=_internal todaysBytesIndexed LicenseManager -Audit NOT source=*web_service.log NOT source=*web_access.log | eval Daily_Indexing_Volume_in_MBs = todaysBytesIndexed/1024/1024 | timechart avg(Daily_Indexing_Volume_in_MBs) by host