Deploy:Extend Splunk search language with legacy scripts

This is an example how to extend Splunk search language with legacy scripts you have used for analysis and reporting of logs that are now stored in Splunk. In this example I'm using dummy script called linelength.awk that counts frequencies of different line lengths in it's input, but similar method can be used to extend Splunk search language with more complex real life scripts as well.

Structure of Splunk application looks like this

${SPLUNK_HOME}/etc/apps/linelength/
 +-- bin
 |     linelength.awk
 |     linelength.py
 +-- default
 |     app.conf
 |     commands.conf
 +-- metadata
 |     default.meta
 +-- local

Get the latest source from https://bitbucket.org/kallu/splunkextension

Scripts

• linelength.awk

Original script doing the magic. Or actually in this case there is no magic in here. There are 2 minor changes I made to make it work better with Splunk; excluding the 1st field that is Splunk timestamp added by python wrapper, and formating output in CSV to make it easier to parse.

#!/usr/bin/awk -f
#
# linelength - count line length frequencies
# (excluding the 1st field which is timestamp added by Splunk)
#
BEGIN { print "timestamp,len,freq"; }

# Read throught the input and keep track of
# the lastest timestamp for each word/field
{ l = length($0) - length($1) - 1; ll[l]++; ts = $1 }

END { for (len in ll) printf "%s,%s,%s\n", ts, len, ll[len] }

• linelength.py

Python wrapper to run linelength.awk from Splunk

#!/usr/bin/python
import sys, os
import csv
import subprocess
import splunk.Intersplunk
import StringIO

Assume that your legacy script is found in the same directory as your python wrapper

my_prog = os.path.join(sys.path[0],'linelength.awk')

Start reading CSV formated data from STDIN right away. First row must contain column headers so the wrapper script knows what part of it should be forwarded to your script. Remember to set enableheader=false in commands.conf to prevent Splunk from adding it's own header.

reader = csv.DictReader(sys.stdin)

Then spawn your script. For Python subprocess tutorial see http://www.doughellmann.com/PyMOTW/subprocess/

p = subprocess.Popen(my_prog, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

Parse _time and _raw -fields from Splunk and pass them to your script STDIN

for row in reader:
   p.stdin.write(row['_time'] + ' ' + row['_raw'] + '\n')

Wait for the script to terminate and parse it's output

results = []
output = p.communicate()[0]

f = StringIO.StringIO(output)
reader = csv.DictReader(f)

Loop through the results and generate output to Splunk in key=value format that Splunk can parse automatically. You only need to have _raw and _time -fields for Splunk. Everything else is optional. Below code depends on certain fieldnames in CSV output from your script. Use it only as an example how remove and add fields.

for row in reader:
   splunk_time = row['timestamp']
   # delete fields that are not wanted in key=value output for Splunk
   del row['timestamp']
   # generate _raw field of key=value -pairs for Splunk to parse
   # Splunk can automatically recognize and parse key=value -format
   splunk_raw = []
   for key in row:
      splunk_raw.append(key + '=' + row[key])   
   row['_raw'] = ','.join(splunk_raw)
   row['_time'] = splunk_time
  
   results.append(row)
  
splunk.Intersplunk.outputResults(results)

Splunk config

• app.conf

Standard application metadata, nothing special here to see :-)

• commands.conf

Here you define a name of new search command and how Splunk should execute your script. Most of these are what they look like, or Splunk docs explain what they do, but streaming attribute is bit different than you would first think. If you set streaming=false script will be run only once and it can receive at maximum 50k events. Setting streaming=true will tell Splunk to use script as "mapper" and spawn multiple instances of it, each receiving a slice of total events. There is no upper limit how many event can be processed by streaming script, but single instance will still receive max. 50k events. Difference between streaming and non-streaming scripts is explained here

[linelength]
type = python
filename = linelength.py
local = false
perf_warn_limit = 0
streaming = true
maxinputs = 0
passauth = false
retainsevents = false
generating = false
overrides_timeorder = true
enableheader = false

• default.meta

Access control and visibility of your application is defined here. The key thing is to export your command. Without this, your new command isn't visible outside of your application, ie. you can not use it in normal Splunk searches.

[commands]
export = system

Using linelength -command in Splunk searches

Now everything is ready and you can use your new search command to count stats from logs. There is one more catch however, if you just do

... | linelength

you won't get the results you expected :-( Remember that note about streaming=true above. As Splunk did spaw multiple instances of script, you have now your results split into pieces. In results you have multiple entries for the same line length. You still need your "reducer" to get overall results. Luckily in this case you can simply sum frequencies and get total results with

... | linelength | stats sum(freq) by len

If your metrics would have included items like "response time of 98th percentile", you can not simply sum, reducer would have been much more difficult (or impossible) to do. In those cases you have 2 options; modify your original script or set streaming=false to make sure only a single instance of script is run (-> don't need reducer) But keeping in mind that will limit you to max. 50k source events.

References

• Custom search command in Python
• Extend Splunk - Custom search commands
• Debugging a custom search command
• Sample application sourcecode

--Kallu 11:57, 10 July 2012 (PDT)