Community:SplunkLightForwarderInternalsRevisions
From Splunk Wiki
Some notes on changes that have occurred on the light forwarder across versions. This is meant as info for troubleshooting, investigation, rather than a nice friendly document that helps to accomplish anything in particular.
3.4.10
The SplunkLightForwarder app now no longer takes special actions when it is enabled/disabled. This means you should be able to enable/disable it by editing MANIFEST
, removing the directory, etc without trouble.
There is no more setup.conf
file (the delivered file is empty), there are no .xml file twiddles, and all of the necessary tweaks are declared in default-mode.conf
.
Implementation
When splunkd is reading through the composite-splunkd.xml
, instantiating pipelines and processors, it makes use of the values in default-mode.conf
to influence whether it should or should not actually create them. Investigating the behavior will likely require use of splunk start --debug
.
The intended run-time behavior is unchanged from 3.4.9.
3.4.8 - 3.4.9
The "SplunkLightForwarder" app is changed from 3.4.6 to use the parsing pipeline. All inputs pass their data to the parsing pipeline as in traditional Splunk, but the parsing pipeline is truncated, forwarding the data with minimal processing (only utf8 processor).
Configuration
The configuration is essentially the same method as the 3.4 - 3.4.6 configuration, using setup.conf, default-mode.conf, and the 'splunk enable app SplunkLightForwarder
' command to trigger xml file twiddling.
Implementation
The number of files twiddled is reduced to one, in setup.conf:
[files] parsing = config/parsing/config.xml
The number of processors disabled is increased, in default-mode.conf:
#Turn off a processor [pipeline:indexerPipe] disabled_processors=diskusage, signing, tcp-output-generic-processor
3.4 - 3.4.6
The "SplunkLightForwarder" application moved the policy of what should be enabled/disabled out of the python scripts (3.3.x) and into conf files in an application directory.
In 3.4 -> 3.4.6, there were no processors loaded for the parsing or indexing queue, and these queues were not intended to be used. The input-method specific queues simply contained a sendout processor. (This seemed to cause the lightforwarder to instantiate the sendout processor multiple times, and make multiple connections to the receiver, at least in some cases.)
Configuration
The configuration of this is two part:
- The app specifies in
setup.conf
that it wants certain modules disabled in[modules]
, and also that it wants certainconfig.xml
files replaced in[files]
, and disables certain processors usingdefault-mode.conf
. - A command is run to apply these settings , eg
splunk enable app SplunkLightForwarder
Implementation
The modules are not enabled/disabled unless the command is explicitly run. The config.xml files for modules are renamed to config.xml.disabled, and the files in the 'files' section are copied from etc/apps/SplunkLightForwarder/config.... to etc/modules/... replacing the default files.
setup.conf from 3.4.6 prior to 3.4.8:
[apps] samples = disabled [modules] distributedDeployment = disabled distributedSearch = disabled input/FIFO = disabled input/UDP = disabled input/tcp = disabled input/syslogFIFO = disabled input/syslogUDP = disabled [files] input/exec = config/input/exec/config.xml input/tailfile = config/input/tailfile/config.xml input/wineventlog = config/input/wineventlog/config.xml input/fschangemonitor = config/input/fschangemonitor/config.xml
default-mode.conf from 3.4.6:
#Turn off a processor [pipeline:indexerPipe] disabled_processors=diskusage, signing
3.3.x
The "Lightweight Forwarder" simply disabled a variety of unnecessary input modules so they were not loaded.
Configuration
The means to set this up was "splunk set server-type forwarder
".
http://www.splunk.com/base/Documentation/3.3.3/Admin/EnableForwardingAndReceiving#Lightweight_forwarding
Implementation
The implementation was to rename various config.xml files in etc/modules/input/<type>/ to config.xml.disabled, this was what the set server-type command did.
There was an additional step that was documented to avoid parsing by routing inputs directly to the index queue, which we called Transplant Parsing.
intputs.conf: [inputstanza] queue=indexQueue