Community:Daily Log Review
From Splunk Wiki
Use Splunk for daily log review
A broad number of regulations require that organizations store, retain, and review logs on a regular basis in order to ensure that problems are caught quickly. One of the best practices (or required mandates) toward this goal is reviewing your logs daily. This document addresses methods for accomplishing daily log review efficiently by letting Splunk do a lot of your heavy lifting.
Indexing the right logs
Before you can use Splunk on an IT data source, you need to make sure that Splunk indexes that data source so that you can search on it. If you want to make the most of Splunk and the solutions Splunk offers, then you will want to set up your server so that you're using [howto:Understanding_Splunk's_interface_standards Splunk-compliant] [howto:Install_Splunk_Compliant_Data_Source_Add-ons fields, event types, and more]. You also may want to check SplunkBase for applications written specifically for the areas of compliance that you're interested in.
Be sure that you're capturing data from every server that handles data covered in compliance regulations.
Building your log review search
The first time you run through this process, it will be fully manual. In the process of this manual endeavor, you develop saved searches and other processes that make daily log review simpler and simpler. To begin, access the Splunk UI (it's up to you whether you use the command line or GUI version) and do a search for all activity in the last 24 hours by leaving the search bar blank, making sure
last 24 hours is selected in the time dropdown list box, and pressing Enter. The search box will become populated with an asterisk (*) and the search will show you everything that happened in the last 24 hours.
If you have a significant number of logs that you don't want to include in your daily log review, then you can approach this issue in one of two ways:
- You can develop a search that excludes the logs you don't want to deal with.
- You can develop a search that includes only the logs you want to deal with.
Which is the better approach depends on the organization of your data. Typically the shorter search, the one with the least ANDs, ORs, and NOTs, is the more efficient--and therefore the preferred search. Here are some hints on how to make your searches shorter:
- You may not want to see any logs from entire hosts when you do your daily log review, so with a clever use of host tags you can either include or eliminate. For example, you might tag all desired hosts with
daily_review_yesand tag all non-desired hosts
daily_review_no. Then, you could just do a search on
hosttag="daily_review_yes"to see the logs from the hosts that contain data you're interested in. Or, you could use
NOT hosttag="daily_review_no"to eliminate the hosts that contain nothing of interest.
- You can build event types that either match the data you want to look at, or the data you don't, and then tag them in a similar fashion to the hosts. You can even use the same tag name in both cases.
Note: Event types are not visible by default. To utilize them, click Fields to open the field list, click Core Only to uncheck its check box, under Search click eventtype, and then click the X in the upper left of the dialog box to close it and see the event types associated with each event.
When it comes to tags, it's important to follow the [howto:Understanding_Splunk's_interface_standards Splunk interface standard]. The closer you adhere to this standard, the more likely that any add-ons you download from Splunk will work seamlessly with your setup.
Using a combination of host tags, event type tags, and other search components--or just a * for everything--you should be able to build a search that encompasses your entire daily log review. If not, you may need to build a collection of searches that, together, help you do your daily log review.
Once you have your daily log review search(es), save them with useful name(s) such as
Daily Log Review. Be sure to set a particular time for this search to run each day, so it really is a daily review.
Making it easier to find problems
By itself, a search that pulls up the last day's logs is only so helpful. While in the beginning you'll need to look through many of the events manually, over time you can take advantage of event type tagging in order to weed out those events that are innocuous, and mark the ones that signal problems. Not that you can tag individual events--doing so wouldn't be particularly useful. Instead, you can make event types that match the uninteresting or bad events, and then apply useful event type tags that you can use in your search. For example, you might use
ok for those events you don't want to worry about, and
not_ok for the events that signal potential problems.
See [howto:Create_event_types_in_Splunk Create event types in Splunk] for best practices on creating event types. The same topic will also teach you how to use filters.
After you've started applying these tags, it can be useful to regularly (typically every hour) run a search that includes the term
NOT eventtypetag="ok". Using this tag instead of
not_ok means that untagged events will show up as well as those that you have marked to watch out for. You can use these moments to make yet more event types to tag as okay, so you can trim down the number of events you have to look at each time.
From here, you might choose to [howto:Make_your_data_source_work_with_Splunk-provided_reports build reports], dig more deeply into events that concerned you, or move on to some other task knowing that Splunk is helping to watch over your systems, and tracking your reviews so that auditors can tell you did them. One interesting and quick report you can run is to take your daily
NOT eventtypetag=ok search and add at the end
| stats first(_raw) count by punct | sort -count, such as:
NOT eventtypetag="ok" | stats first(_raw) count by punct | sort -count
This report will arrange your suspect events by their punct:: (punctuation) to make it easier to see everything at a glance. While you cannot assume that all identical punctuations mean mostly identical events, this report works well as a quick view into how many times particular things happened. You can then click through the punctuation you're in to investigate more deeply. Once you've clicked through a punctuation, you can then use filters to check and see if the events with the same punctuation really are all the same thing or if they come from wildly different sources, sourctypes, or whatever field is important in your specific situation.