Community:Splunk Socket Behavior
From Splunk Wiki
As for socket behavior, there are a few situations.
- General expectation is that the socket behavior is very vanilla. We open a tcp socket, and send data one way. There is some special inferences around socket state (below). If there is anything that looks different here from eg a boring telnet setup, I'd like to hear about it.
- AutoLB client/forwarder makes a DNS query to discover the identies of the AutoLB servers. It supports DNS returning a list of records and using that list. Of course there are other DNS situations, but this seems the one that is interesting to forwarding.
- Every 30 seconds, an AutoLB forwarder selects a new (random) indexer from the set of indexers, and opens a connection there. If new streams (eg newly found files) come to exist, it will send that data to this indexer.
- When both: all streams attached to an indexer are done, AND 30 seconds have gone by, it shuts down the connection to an indexer.
- Reviewing the above, you see that multiple connections can be (will often be) open at once.
- A forwarder that is sending to a given indexer periodically will open a new socket to that indexer, just to see that the listening port is still available. This is a state test, no information is sent.
- A reciever which is going to shutdown, will close the listening port, so that forwarders can find out that shutdown is coming, and finish sending what they have.
- A receiver which has blocked queues (full disk, etc) will similarly close the listening port.
- If a connection comes in which does not appear to be the splunk forwarding protocol, the connection is closed without any response.
- When an indexer socket is open, it will send at heartbeat intervals, fake events, that say "i'm just a heartbeat", to force the socket to stay open through networking equipment.
- In 4.1, these connetions are one way, the only backchannel is TCP layer window control, etc.
- In 4.2+, the receiver will do a form of windowing, if enabled, manually acknowledging data chunks at the application layer, which informs forwarder behavior. This is the useACK feature.
- Mangement port is used for all non-forwarding communications.
- HTTP, and should mostly look just like vanilla HTTP socket behavior.
- There are three client implementations for this that we ship: splunk.exe command line in C; mrsparkle/splunkweb in python, splunkd.exe code in C++.
- This is nearly always HTTP/SSL. You can turn off the SSL, but you lose too much for anything but debugging. Compression/performance, cross-splunk instance trust network, etc.
- These are always short-lived reqest-response links. (Well when pulling gigabytes out, short-lived is.. a stretch) We should someday maybe allow more streaming-style connections (very thorny in the real world of http proxies etc), but right now such things are implemented via polling.
- Thus, depending how thorough you wish to be, you could try:
- Forwarder establishment, and behavior, during normal operation of periodic new file creation. May want to look at actual forwarding socket, as well as accessability-test socket.
- mangement port access, such as from cli (easiest to control) or splunkd-to-splunkd, eg distributed search (possibly more sensitive for security concerns)