Community:Deployment Considerations

From Splunk Wiki

Jump to: navigation, search

Deployment considerations for data inputs

Rachel 16:51, 18 November 2009 (PST)

Splunk supports five primary data input types - file and directory inputs, FIFO queues, network ports, scripted inputs, and Windows event logs.

File and directory inputs

Splunk can accept data input from local or mounted systems, and can read data through the use of the Splunk file input processor. The file input processor can operate in a variety of modes and is capable of reading entire files, updates to files, and real-time changes to files as well as performing those tasks on entire directory trees. Splunk supports whitelisting and blacklisting inside directory inputs for additional flexibility of configuration. Refer to the documentation about file and directory inputs for more information.

FIFO queues

Caution: Due to their vulnerability, FIFOs are not recommended. Monitor is a more reliable, stable method. Support FIFO inputs is deprecated and will be removed in a future release of Splunk.

A FIFO (AKA named pipe) is a queue of data maintained in memory. File systems can write log messages directly to a FIFO. Splunk then accesses the FIFO as though it were a file. FIFO access is very fast, but FIFOs are vulnerable when there are processing disruptions because the in-memory data may be lost.

To configure FIFO queues, see this page.

Network ports

Splunk can accept data from both UDP and TCP ports. While you can use this to mimic a local system syslogd, it is equally useful for capturing any other IT data via normal network mechanisms. Like FIFO queues, network ports can offer higher indexing performance, but with similar vulnerability to data loss. Although TCP-based network communication can mitigate most data loss issues, If your deployment can tolerate absolutely no data loss, Splunk recommends that you choose files as the data input type. Refer to the documentation about network port inputs for more information.

Scripted inputs

You can configure Splunk to run an arbitrary command on any schedule, with the output being indexed by Splunk. The primary advantage of scripted inputs is that they make it possible to index almost any type of data. Examples of data inputs that can be scripted include performance data, system and network status commands, Web requests, and SNMP traps, as well as other types of IT data. Scripted inputs can represent varied performance impact, primarily due to the number of possibilities for integration, but low-overhead scripts usually have similar performance to file data inputs. Refer to the documentation about scripted inputs for more information.

Windows event logs and WMI

Splunk can index Windows event logs, and by default indexes the Application, System, and Security event logs. You can configure Splunk to index other Windows event logs sources if they are present on the system, use WMI to pull data from other Windows machines, and monitor changes to your Windows Registry. Refer to the documentation about inputs for Windows, the documentation about WMI configuration, and the documentation about Windows Registry monitoring for more information.

Personal tools
Hot Wiki Topics

About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk