From Splunk Wiki
UDP inputs with Splunk
By design, UDP is a lossy protocol. In cases where all data MUST be logged in complete form, UDP should not be used. Instead, a tcp or file/dir monitor should be implemented as the data input.
UDP data can be lost at any point in the communication process. This includes the originating server, network switch, network router, receiving server, the OS, and splunk. There is the possibility for the OS to drop UDP data at it's discretion. There is also the possibility for Splunk to lose UDP data due to the buffer being filled.
- Set the OS UDP buffer size to a larger amount (default *nix values are in the 64k+ range)
- Each OS has a specific kernel parameter. See table BELOW
- Set the UDP buffer size in Splunk (inputs.conf) http://www.splunk.com/base/Documentation/latest/Admin/Inputsconf
- set the following under your "[udp://]" stanza.
- _rcvbuf = 5000000
- This will set it to 5 megabytes.
- The default setting of 1 megabyte, can typically handle ~2000 UDP messages per second.
- we have seen this value set to 10 MB
* Run "netstat -s" before and after your test, or over a short period of time. Examine the UDP section for packet loss.
Settings to look for at the OS level:
|Operating System||command to change the value|
|Linux||sysctl -w net.core.rmem_max=8388608|
|Solaris||ndd -set /dev/udp udp_max_buf 8388608|
|FreeBSD, Darwin||sysctl -w kern.ipc.maxsockbuf=8388608|
|AIX||no -o sb_max=8388608 (note: AIX only permits sizes of 1048576, 4194304 or 8388608)|
Working example from Linux:
inputs.conf - setting the value of _rcvbuf to 20000000
kernel tuning (via sysctl) -
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216