Community:Multi-line Events Multi-value Field Extraction

Field Extraction: Multi-line Events Multi-value

This configuration example is to extract fields which contain multi-values in an an multi-line event.
Let's take a look at the sample event below.

#
# A sample multi-line event 
#

Sep 7, 2011 6:41:57 PM
myhost01-Win7-Updates    | win-p1-filezilla	| 1909-0    | network     | required
myhost22-Win2k8-Updates  | win-pwdutils		| 2001-0    | util        | not required    
myhost13-CentOS5-Updates | lin-perl 		| 2023-0    | security    | required        
myhost11-Ubuntu-Updates  | lin-python		| 3312-0    | security    | not required

And, here is how we want to extract fields;
  - Each line as a eachline field
  - Each column separated by "|" as each field; UpdateType, patchType, patchVersion, patchCategory, PatchRequirement

This event is a kind of table format. So, you might think our magic "multikv" can help us to extract multi-value field.
Unfortunately you cannot use "multikv" command because there is no field name in the first row.

For this specific sample event, we also need to take line break into consideration to index the event correctly. It's because Splunk would be confused when trying to find timestamp, and might parse 2001-0 or similar numbers as a timestamp. In order to avoid the potential problem, we added the LINE_BREAKER.

Let's try!

1. Create the sample log file as /var/log/patch-update.log

2. Configure props.conf for line-breaking (LINE_BREAKER & SHOULD_LINEMERGE) and search time field extraction (REPORT)

#  
# This config will extract fields; each line, and each fields 
#
- props.conf
[mvadd]
LINE_BREAKER = ([\r\n]+)\w{3}\s+\d{1,2},\s+\d{4}\s+[\d\:]+\s+(?:AM|PM)
SHOULD_LINEMERGE = false
REPORT-patchlog = extract_patchlog

3. Configure transforms.conf to define the regex to find the field values. For multi-value field extraction, we use MV_ADD.

- transforms.conf
[extract_patchlog]
REGEX = ^(([\w\-]+)\s+\|\s+([\w\-]+)\s+\|\s+([\w\-]+)\s+\|\s+([\w\-]+)\s+\|\s+(.*?))(?:[\r\n]|$)
FORMAT = eachline::$1 UpdateType::$2 patchType::$3 patchVersion::$4 patchCategory::$5 PatchRequirement::$6
MV_ADD = true

4. Index the events by oneshot command

  # $SPLUNK_HOME/bin/splunk add oneshot /var/log/patch-update.log -sourcetype mvadd -auth admin:changeme

5. Login to Splunkweb, and search with sourcetype="mvadd", and check how each fields are extracted.