Community:Extracting ISA Fields

From Splunk Wiki

Jump to: navigation, search

Here is an example for how to configure ISA server logs as a Splunk input. You will need to perform the following steps:

1. Add the log file as an input

In this situation I want Splunk to index both Firewall and Web Proxy logs. These files reside on a mounted filesystem in a single directory:

/mount/isalogs/FWSEXTD20010623.log
/mount/isalogs/WEBEXTED20010623.log

Since they are uniquely named, but with identifying attributes, we can use a wildcard to monitor them all. The inputs.conf file would have the following entry:

[monitor:///mount/isalogs/*]
host = demomachine

2. Classify/Name the data source

Since we are monitoring a single directory at the input level, we must be able to distinguish each input for the purpose of extraction data. To do this, name a unique source type so we can perform field extraction. Add the following entries to the props.conf file:

[source::.../mount/isalogs/FW*.log]
sourcetype=isa_fw

[source::.../mount/isalogs/WEB*.log]
sourcetype=isa_web

3. Set the time stamp extraction

This is done during indexing time and is set in the props.conf file. You must tell Splunk to recognize a certain prefix and time format.

For our first example, we are monitoring a web proxy log which looks as follows:

10.100.2.3, jdoe, Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; InfoPath.1), -, 9/17/2009, 0:00:00, -, hostname, -, dest, 10.200.3.4, 8888, 15, 616, 2305, http, -, GET, http://www.splunk.com/support, -, Cache, 0, -, Web Traffic, -, Internal, External, 0x880, Allowed, -

To extract the correct timestamp you would add the following parameters to the stanza of the above source or sourcetype:

TIME_PREFIX = ^(?:[^\,]+,){4}\s*
TIME_FORMAT = %m/%d/%Y, %H:%M:%S

These settings will pull the correct time stamp when the date is located in the 4th comma delimited field and the time is located in the fifth comma delimited field.

For our second example, we have the following Firewall Log:

firewall-servername , 9/17/2009, 0:00:00, TCP, 10.200.1.2:15241, 10.100.3.4:8888, 10.3.47.4, Local Host, External, Terminate, 0x80074e20, -, Appname, -, 8504, 8504, 49264, 49264, -, -, -, -, -, -, -, -, 3659416, 72408468, -, -, -, -

The above date and time exists in the 2nd and 3rd delimited field so we will use:

TIME_PREFIX = ^(?:[^\,]+,)\s*
TIME_FORMAT = %m/%d/%Y, %H:%M:%S

Including all of the settings from Step 2, our props.conf file for both examples now looks as follows:

[source::.../mount/isalogs/WEB*.log]
TIME_PREFIX = ^(?:[^\,]+,){4}\s*
TIME_FORMAT = %m/%d/%Y, %H:%M:%S
sourcetype=isa_web

[source::.../mount/isalogs/FW*.log]
TIME_PREFIX = ^(?:[^\,]+,)\s*
TIME_FORMAT = %m/%d/%Y, %H:%M:%S
sourcetype=isa_fw

4. Extract fields

Since these types of logs use comma separated values, you can leverage the DELIMS parameter to extract the fieldvalues. To retrieve the correct time stamp, you need to manually configure the prefix within Splunk. But first, you should create a REPORT stanza to tell Splunk to run a transform on this data. To do this, add the following stanza to the props.conf file:

[isa_web]
REPORT-isa_web = isa_web_csv

Let's revisit our earlier log file example:

10.1.2.3, jdoe, Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; InfoPath.1), -, 9/17/2009, 0:00:00, -, hostname, -, dest, 10.2.3.4, 8888, 15, 616, 2305, http, -, GET, http://www.splunk.com/support, -, Cache, 0, -, Web Traffic, -, Internal, External, 0x880, Allowed, -

For this exercise, we want to extract the first 9 fields. Here is how you would build the stanza in <ocde>transforms.conf</code>:

[isa_web_csv]
DELIMS = ","
FIELDS = "isa_cliip", "isa_user", "isa_agent", "isa_authstatus", "isa_date", "isa_time", "isa_svcname", "isa_serv", "isa_compname"

To map additional fields, simply add them to the FIELDS argument in the appropriate location. If you need to exclude a field, you must still specify a set of double quotes and comma. So, in the above example, excluding "isa_user" would look like:

FIELDS = "isa_cliip", "", "isa_agent", "isa_authstatus", "isa_date", "isa_time", "isa_svcname", "isa_serv", "isa_compname"

Although ISA server logs conform to a standard, you will need to double check the order of your fields as they differ between servers and versions.

5. Reload the transforms

This can be done by running a search command or restarting Splunk. A restart of Splunk may be necessary if you have manually added the inputs.

Run the following search command to reload the props.conf and transforms.conf settings.

In Splunk Web: <search>| extract reload=True</search>

In the CLI:

$SPLUNK_HOME/bin/splunk search "| extract reload=True"

Example Files

inputs.conf

[monitor:///mount/isalogs/*]
host = demomachine

props.conf

[source::.../mount/isalogs/WEB*.log]
SHOULD_LINEMERGE = FALSE
TIME_PREFIX = ^(?:[^\,]+,){4}\s*
TIME_FORMAT = %m/%d/%Y, %H:%M:%S
sourcetype=isa_web

[source::.../mount/isalogs/FW*.log]
SHOULD_LINEMERGE = FALSE
TIME_PREFIX = ^(?:[^\,]+,)\s*
TIME_FORMAT = %m/%d/%Y, %H:%M:%S
sourcetype=isa_fw

[isa_web]
REPORT-isa_web = isa_web_csv

[isa_fw]
REPORT-isa_fw = isa_fw_csv

transforms.conf

[isa_web_csv]
DELIMS = ","
FIELDS = "web_cliip", "web_user", "web_agent", "web_authstatus", "web_date", "web_time", "web_svcname", "web_serv", "web_compname"

[isa_fw_csv]
DELIMS = ","
FIELDS = "fw_serv", "fw_date", "fw_time", "fw_ipproto", "fw_src", "fw_dst", "fw_cliip", "fw_srcnet", "fw_dstnet", "fw_action", "fw_status", "fw_rule", 
Personal tools
Hot Wiki Topics


About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk