Community:Splunk Socket Behavior

As for socket behavior, there are a few situations.

Forwarding protocol

• General expectation is that the socket behavior is very vanilla. We open a tcp socket, and send data one way. There is some special inferences around socket state (below). If there is anything that looks different here from eg a boring telnet setup, I'd like to hear about it.

• AutoLB client/forwarder makes a DNS query to discover the identies of the AutoLB servers. It supports DNS returning a list of records and using that list. Of course there are other DNS situations, but this seems the one that is interesting to forwarding.

• Every 30 seconds, an AutoLB forwarder selects a new (random) indexer from the set of indexers, and opens a connection there. If new streams (eg newly found files) come to exist, it will send that data to this indexer.

• When both: all streams attached to an indexer are done, AND 30 seconds have gone by, it shuts down the connection to an indexer.

• Reviewing the above, you see that multiple connections can be (will often be) open at once.

• A forwarder that is sending to a given indexer periodically will open a new socket to that indexer, just to see that the listening port is still available. This is a state test, no information is sent.

• A reciever which is going to shutdown, will close the listening port, so that forwarders can find out that shutdown is coming, and finish sending what they have.

• A receiver which has blocked queues (full disk, etc) will similarly close the listening port.

• If a connection comes in which does not appear to be the splunk forwarding protocol, the connection is closed without any response.

• When an indexer socket is open, it will send at heartbeat intervals, fake events, that say "i'm just a heartbeat", to force the socket to stay open through networking equipment.

• In 4.1, these connetions are one way, the only backchannel is TCP layer window control, etc.

• In 4.2+, the receiver will do a form of windowing, if enabled, manually acknowledging data chunks at the application layer, which informs forwarder behavior. This is the useACK feature.

Management Port

• Mangement port is used for all non-forwarding communications.

• HTTP, and should mostly look just like vanilla HTTP socket behavior.

• There are three client implementations for this that we ship: splunk.exe command line in C; mrsparkle/splunkweb in python, splunkd.exe code in C++.

• This is nearly always HTTP/SSL. You can turn off the SSL, but you lose too much for anything but debugging. Compression/performance, cross-splunk instance trust network, etc.

• These are always short-lived reqest-response links. (Well when pulling gigabytes out, short-lived is.. a stretch) We should someday maybe allow more streaming-style connections (very thorny in the real world of http proxies etc), but right now such things are implemented via polling.

• Thus, depending how thorough you wish to be, you could try:
  • Forwarder establishment, and behavior, during normal operation of periodic new file creation. May want to look at actual forwarding socket, as well as accessability-test socket.
  • mangement port access, such as from cli (easiest to control) or splunkd-to-splunkd, eg distributed search (possibly more sensitive for security concerns)