Deploy:Combine bi-directional network logs
From Splunk Wiki
Combine one-directional messages to recreate a conversation
A network flow is traditionally defined (following Cisco) as a uni-directional sequence of packets sharing all of the following 7 values:
- Source IP address
- Destination IP address
- Source port for UDP or TCP, 0 for other protocols
- Destination port for UDP or TCP, type and code for ICMP, or 0 for other protocols
- IP protocol
- Ingress interface (SNMP ifIndex)
- IP Type of Service
An example of netflow messages might be:
2007-09-14 10:54:58.130 0.896 TCP 22.214.171.124:2691 -> 126.96.36.199:80 3 144 1
2007-09-14 10:54:55.378 5.184 TCP 188.8.131.52:25 -> 184.108.40.206:26490 26 1453 1
It is often useful to combine these uni-directional flow messages to re-construct the entire conversation. Common use cases include traffic shaping analysis and detection of data exfiltration.
The following query will find conversations (flows to and from the same ip/port combination) where the bytes transferred out of your network exceed the bytes transferred into your network:
index=netflow | head 100000 | eval src=(replace(srcaddr,"\.","").srcport) | eval dest=(replace(dstaddr,"\.","").dstport) | eval commonid=if(dest>src, prot." ".dstaddr." ".dstport."<-->".srcaddr." ".srcport, prot." ".srcaddr." ".srcport."<-->".dstaddr." ".dstport) | eval ingressbytes=if(dest>src, bytes,0) | eval egressbytes=if(dest>src, 0,bytes)|stats sum(ingressbytes) as totalingressbytes sum(egressbytes) as totalegressbytes list(_raw) by commonid | eval byteratio=totalegressbytes/totalingressbytes | search byteratio>1
Basically, what I first do is concatenate the ip and port numbers into two fields: “src” and “dest”, removing all periods so that they are integers. I then create a “commonid” that matches across both ingress and egress events (by normalizing the order of dest and src), and also create fields that copy the “bytes” field for the ingress events and egress events separately. I then sum those ingress/egressbytes values for each commonid and then use stats to group it by the commonid, and then calculate the ratio.
Note: though my netflow format may be different than yours, the query will work as long as you have the srcaddr, srcport, dstaddr, dstport, prot, and bytes fields extracted in your instance.