From Splunk Wiki
StdoutOutputProcessor is an internal debugging tool that generates information about incoming events as they are indexed. You may want to do this if you are having problems with a complex source, like a testing a new scripted input.
Caution: This must be used with extreme care, the output is verbose and will impact both performance and disk usage. The recommended method is to reproduce your issue in a test environment for debugging purposes, with as few data inputs as possible.
This works by adding another processor to a pipeline, which will execute in sequence as data is handled. For some background on working with processors and pipelines, see this ancient documentation: http://www.splunk.com/base/Documentation/3.1.4/Developer/SelectingEventsToProcessWithABundle
The output goes to splunkd_stdout.log and looks like this:
[disabled] = false [MetaData:Source] = source::/var/log/system.log [MetaData:Host] = host::andrea-118500.splunk.com [MetaData:Sourcetype] = sourcetype::syslog [_path] = /var/log/system.log [_charSet] = UTF-8 [_origPath] = /var/log/system.log [_MetaData:Index] = main [_stanzaKey] = monitor:///var/log/system.log [_raw] = Feb 5 10:42:49 andrea-118500 /System/Library/CoreServices/backupd: Post-back up thinning complete: 1 expired backups removed [_utf8] = _utf8 [_linebreaker] = _linebreaker [_time] = 1265395369 [_conf] = source::/var/log/system.log|host::andrea-118500.splunk.com|syslog|
You will also see events that come through with empty _raw but a _done key:
[_raw] = [_done] = _done
That means we are finished indexing from that source, at least for the moment.
Add StdoutOutputProcessor to the main parsing pipeline
Before starting this example, be sure you understand how to work with Splunk conf files and understand configuration precedence.
The main parsing pipeline is defined in $SPLUNK_HOME/etc/modules/parsing/config.xml. Everything being indexed passes through this pipeline, including internal events. There are several pipelines defined here, but the one we are interested in starts with
<pipeline name="parsing" type="startup">
Each processor in this pipeline is defined by <processor ... > </processor> in the xml. For our purposes, the important ones are at either end: queueinputprocessor gets data into the pipeline and queueoutputprocessor passes it along to the next stage of indexing. So to add a new processor, it must be in the middle. Since StdoutOutputProcessor doesn't modify data it otherwise does not need to be in a specific location to work correctly.
Start in the $SPLUNK_HOME/etc/modules/parsing/ directory. Save a copy of config.xml so you can replace it when you are finished (or restore if you mangle the xml.) Insert the following xml before queueoutputprocessor:
<processor name="stdout" plugin="stdoutoutputprocessor"> <config> </config> </processor>
Whitespace is not critical here, but correct xml is. Save this file.
Go to the $SPLUNK_HOME/etc directory. Locate all the inputs.conf files, for Unix systems this is the command:
$ find . -name inputs.conf ./apps/sample_app/default/inputs.conf ./apps/search/local/inputs.conf ./apps/SplunkLightForwarder/default/inputs.conf ./apps/unix/default/inputs.conf ./modules/distributedDeployment/classes/deployable/inputs.conf ./system/local/inputs.conf
You almost certainly want to disable all inputs execpt the one you are investigating. To do this, move the inputs.conf files to inputs.conf.disabled for everything else currently enabled. To check which ones are enabled, look in the app.conf files, which may be in either appname/default or appname/local. A disabled app will contain this in app.conf:
[install] state = disabled
You may see one in default and one in local, which means the one in local is the active file.
If needed, edit the inputs.conf containing the data input of interest to enable it and disable others. Restart your Splunk instance.
Check splunkd_stdout.log for your events. Each set of pipeline data keys begins with "[disabled]" and contains one line for each key. _raw is the raw text of your event, MetaData:Sourcetype is the sourcetype and so on.