Community:Search Report: How To Analyze Difference between the timestamp Vs IndexedTime
From Splunk Wiki
How To Analyze Difference between the timestamp Vs IndexedTime
This is a useful search when you want to analyze if the timestamp is away from the index time.
When you notice that there is no new events since today. You might think the indexer is not indexing events. But, actually the indexer might be still indexing events with incorrect timestamp. So, you want to check the events' timestamp (_time), the index time (_indextime), and the difference between them.
# # Table Comparison TimeStamp Vs. IndexTime # index=_internal earliest=-3h@h latest=-2h@h | rename _indextime as IndexTime | eval diff=IndexTime-_time | convert ctime(IndexTime) as IndexTime | eval diff=if(diff < 0, "0", diff) | table _time IndexTime diff
This is another example. Events such as syslog is indexed in incorrect timestamp. And you might need to troubleshoot. At the same time, you might need to verify there are no missing events in the incorrectly indexed events.
The following search report helps us to identify such troubleshooting.
# # Search Report # => Search by indexed time # # # How to identify events the indextime and timestamp are off (more than 1 hour=3600 sec) # source="udp:514" | eval IndexTime=_indextime | eval TimeStamp=_time | eval delta=_indextime-_time | eval Raw=_raw | where delta > 3600 | convert ctime(IndexTime) | convert ctime(TimeStamp) | table TimeStamp IndexTime delta Raw
This following search is to specify time range based on index time. It is more expensive search than general search with a time range based on event's time stamp
# # How to specify the timerange based on indexed time # source="udp:514" | eval IndexTime=_indextime | eval CurrentTime=now() | eval D0=relative_time(now(), "@d") | eval D1=relative_time(now(), "-1d@d") | where IndexTime < D0 | where IndexTime > D1 | eval Raw=_raw | convert ctime(IndexTime) as IndexTime01 | table IndexTime IndexTime01 Raw