Community:Search Report: How To Analyze Difference between the timestamp Vs IndexedTime

From Splunk Wiki

Jump to: navigation, search

How To Analyze Difference between the timestamp Vs IndexedTime

This is a useful search when you want to analyze if the timestamp is away from the index time.

When you notice that there is no new events since today. You might think the indexer is not indexing events. But, actually the indexer might be still indexing events with incorrect timestamp. So, you want to check the events' timestamp (_time), the index time (_indextime), and the difference between them.

# Table Comparison TimeStamp Vs. IndexTime

index=_internal earliest=-3h@h latest=-2h@h
   | rename _indextime as IndexTime
   | eval diff=IndexTime-_time
   | convert ctime(IndexTime) as IndexTime
   | eval diff=if(diff < 0, "0", diff)
   |  table _time IndexTime diff 

Table IndexTimeVsTimestamp.png

This is another example. Events such as syslog is indexed in incorrect timestamp. And you might need to troubleshoot. At the same time, you might need to verify there are no missing events in the incorrectly indexed events.

The following search report helps us to identify such troubleshooting.

# Search Report
# => Search by indexed time

# How to identify events the indextime and timestamp are off (more than 1 hour=3600 sec)

    | eval IndexTime=_indextime
    | eval TimeStamp=_time
    | eval delta=_indextime-_time
    | eval Raw=_raw
    | where delta > 3600
    | convert ctime(IndexTime)
    | convert ctime(TimeStamp)
    | table  TimeStamp IndexTime delta Raw

This following search is to specify time range based on index time. It is more expensive search than general search with a time range based on event's time stamp

# How to specify the timerange based on indexed time
source="udp:514" | eval IndexTime=_indextime
   | eval CurrentTime=now()
   | eval D0=relative_time(now(), "@d")
   | eval D1=relative_time(now(), "-1d@d")
   | where IndexTime < D0 | where IndexTime > D1
   | eval Raw=_raw
   | convert ctime(IndexTime) as IndexTime01
   | table IndexTime IndexTime01 Raw

Personal tools
Hot Wiki Topics

About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk