Community:Splunk Socket Behavior

From Splunk Wiki

Jump to: navigation, search

As for socket behavior, there are a few situations.

Forwarding protocol

  • General expectation is that the socket behavior is very vanilla. We open a tcp socket, and send data one way. There is some special inferences around socket state (below). If there is anything that looks different here from eg a boring telnet setup, I'd like to hear about it.
  • AutoLB client/forwarder makes a DNS query to discover the identies of the AutoLB servers. It supports DNS returning a list of records and using that list. Of course there are other DNS situations, but this seems the one that is interesting to forwarding.
  • Every 30 seconds, an AutoLB forwarder selects a new (random) indexer from the set of indexers, and opens a connection there. If new streams (eg newly found files) come to exist, it will send that data to this indexer.
  • When both: all streams attached to an indexer are done, AND 30 seconds have gone by, it shuts down the connection to an indexer.
  • Reviewing the above, you see that multiple connections can be (will often be) open at once.
  • A forwarder that is sending to a given indexer periodically will open a new socket to that indexer, just to see that the listening port is still available. This is a state test, no information is sent.
  • A reciever which is going to shutdown, will close the listening port, so that forwarders can find out that shutdown is coming, and finish sending what they have.
  • A receiver which has blocked queues (full disk, etc) will similarly close the listening port.
  • If a connection comes in which does not appear to be the splunk forwarding protocol, the connection is closed without any response.
  • When an indexer socket is open, it will send at heartbeat intervals, fake events, that say "i'm just a heartbeat", to force the socket to stay open through networking equipment.
  • In 4.1, these connetions are one way, the only backchannel is TCP layer window control, etc.
  • In 4.2+, the receiver will do a form of windowing, if enabled, manually acknowledging data chunks at the application layer, which informs forwarder behavior. This is the useACK feature.

Management Port

  • Mangement port is used for all non-forwarding communications.
  • HTTP, and should mostly look just like vanilla HTTP socket behavior.
  • There are three client implementations for this that we ship: splunk.exe command line in C; mrsparkle/splunkweb in python, splunkd.exe code in C++.
  • This is nearly always HTTP/SSL. You can turn off the SSL, but you lose too much for anything but debugging. Compression/performance, cross-splunk instance trust network, etc.
  • These are always short-lived reqest-response links. (Well when pulling gigabytes out, short-lived is.. a stretch) We should someday maybe allow more streaming-style connections (very thorny in the real world of http proxies etc), but right now such things are implemented via polling.
  • Thus, depending how thorough you wish to be, you could try:
    • Forwarder establishment, and behavior, during normal operation of periodic new file creation. May want to look at actual forwarding socket, as well as accessability-test socket.
    • mangement port access, such as from cli (easiest to control) or splunkd-to-splunkd, eg distributed search (possibly more sensitive for security concerns)
Personal tools
Hot Wiki Topics

About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk