Community:MinimizingForwarderFootprint

From Splunk Wiki

Jump to: navigation, search

Splunk Forwarders can be configured in a SplunkLightForwarder mode to limit the amount of system resources required to run them, however there are additional tuning steps that may be desirable to limit the resources further.

Limiting Ram usage

The following steps can be employed to limit a SplunkLightForwarder's memory usage to a greater degree than the default configuration

  • Disable input modules
  • Shrink the outbound event data queue size

Disable input modules

Looking in etc/apps/SplunkLightForwarder/default/setup.conf, we see:

input/FIFO = disabled
input/UDP = disabled
input/tcp = disabled
input/syslogFIFO = disabled
input/syslogUDP = disabled

But there are other inputs that could be disabled, if they are not needed.

  • scripted inputs
  • File system Change monitor

And on Windows,

  • Windows Event log

They can be disabled using the command line invocation: splunk disable module input/<module_name>, for example:

jrodman@joshbook:~> splunk disable module input/exec
input/exec disabled.
You need to restart the Splunk Server for your changes to take effect.

jrodman@joshbook:~> splunk disable module input/fschangemanager
input/fschangemanager disabled.
You need to restart the Splunk Server for your changes to take effect.

jrodman@joshbook:~> splunk disable module input/wineventlog
input/wineventlog disabled.
You need to restart the Splunk Server for your changes to take effect.


Incidentally, note that the windows WMI feature is dependent upon the scripted input, also called input/exec.

Reduce the outbound event queue size

Splunk queues outgoing messages in a memory-based datastructure. If the rate that the forwarder acquires data exceeds the rate at which the data can be transmitted, or the rate at which the receiving side can accept the data, this queue will grow. The default configuration for the SplunkLightForwarder will allow this data to grow to a size of approximately 10MB. This queue can be reduced to limit the amount of data that the forwarder will buffer prior to blocking, if that is desired.

The default value can be found in etc/apps/SplunkLightForwarder/default/outputs.conf:

[tcpout]
maxQueueSize = 1000

This can be reduced. For example a value of 100 would cause the forwarder to only buffer approximately 1MB of data before blocking.

For example, you could add to etc/apps/SplunkLightForwarder/local/outputs.conf or of course any other outputs.conf (such as local or deployment server bundles)

[tcpout]
maxQueueSize = 100

Note: There is a tradeoff between resource consumption and data fidelity here. For inputs such as network inputs or polling inputs like registry monitoring or WMI, full buffers mean a halt in polling for information, or accepting network data, which result in some data not reaching splunk. For file-based inputs, blocking is a much more safe operation.

Limiting Disk usage

Log files

While the SplunkLightForwarder disables any local indexing of customer data as well as Splunk logs, it doesn't change the maximum size of the Splunk logs themselves.

The default splunk logging configuration keeps 5 backup files of 25 MB each in the categories splunkd.log, splunklogger.log, searchhistory.log, metrics.log, and audit.log. In practice on a forwarder the only files likely to be of concern are splunkd.log and metrics.log. The ceiling usage of both of these files and their backups together will be approximately 300MB.

If you require a smaller disk footprint of your forwarders, you may wish to edit etc/log.cfg to change the maxFileSize or maxBackupIndex of these files, for example:

appender.A1=RollingFileAppender
appender.A1.fileName=${SPLUNK_HOME}/var/log/splunk/splunkd.log
appender.A1.maxFileSize=25000000 # default: 25MB (specified in bytes).
appender.A1.maxBackupIndex=2
appender.A1.layout=PatternLayout
appender.A1.layout.ConversionPattern=%d{%m-%d-%Y %H:%M:%S.%l} %-5p %c - %m%n

[...]

# metrics spews a lot of logs, let's not pollute the other files.
appender.metrics=RollingFileAppender
appender.metrics.fileName=${SPLUNK_HOME}/var/log/splunk/metrics.log
appender.metrics.maxFileSize=25000000 # default: 25MB (specified in bytes).
appender.metrics.maxBackupIndex=2
appender.metrics.layout=PatternLayout
appender.metrics.layout.ConversionPattern=%d{%m-%d-%Y %H:%M:%S.%l} %-5p %c - %m%n
category.Metrics=INFO,metrics
category.StatusMgr=INFO,metrics

This would cut the log usage ceiling from around 300MB down to 150MB.

There is again a tradeoff here, where you may want to have log history to troubleshoot forwarder behavior, although in the usual case, the log data is forwarded to a central splunk indexer and available in the "_internal" index.

Personal tools
Hot Wiki Topics


About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk