Deploy:Combine bi-directional network logs

From Splunk Wiki

Jump to: navigation, search

< Back to Best Practices

Combine one-directional messages to recreate a conversation

The background

A network flow is traditionally defined (following Cisco) as a uni-directional sequence of packets sharing all of the following 7 values:

  • Source IP address
  • Destination IP address
  • Source port for UDP or TCP, 0 for other protocols
  • Destination port for UDP or TCP, type and code for ICMP, or 0 for other protocols
  • IP protocol
  • Ingress interface (SNMP ifIndex)
  • IP Type of Service

An example of netflow messages might be:

2007-09-14 10:54:58.130 0.896 TCP 216.129.82.250:2691 -> 209.104.58.141:80 3 144 1

2007-09-14 10:54:55.378 5.184 TCP 209.191.118.103:25 -> 209.104.37.200:26490 26 1453 1

It is often useful to combine these uni-directional flow messages to re-construct the entire conversation. Common use cases include traffic shaping analysis and detection of data exfiltration.

The search

The following query will find conversations (flows to and from the same ip/port combination) where the bytes transferred out of your network exceed the bytes transferred into your network:

index=netflow | head 100000 | eval src=(replace(srcaddr,"\.","").srcport) | eval dest=(replace(dstaddr,"\.","").dstport) | eval commonid=if(dest>src, prot." ".dstaddr." ".dstport."<-->".srcaddr." ".srcport, prot." ".srcaddr." ".srcport."<-->".dstaddr." ".dstport) | eval ingressbytes=if(dest>src, bytes,0) | eval egressbytes=if(dest>src, 0,bytes)|stats sum(ingressbytes) as totalingressbytes sum(egressbytes) as totalegressbytes list(_raw) by commonid | eval byteratio=totalegressbytes/totalingressbytes | search byteratio>1

Basically, what I first do is concatenate the ip and port numbers into two fields: “src” and “dest”, removing all periods so that they are integers. I then create a “commonid” that matches across both ingress and egress events (by normalizing the order of dest and src), and also create fields that copy the “bytes” field for the ingress events and egress events separately. I then sum those ingress/egressbytes values for each commonid and then use stats to group it by the commonid, and then calculate the ratio.

Note: though my netflow format may be different than yours, the query will work as long as you have the srcaddr, srcport, dstaddr, dstport, prot, and bytes fields extracted in your instance.

The result

Netflow bi-directional.png

Personal tools
Hot Wiki Topics


About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk