Author: Vivek Rajagopalan

I got to read this comprehensive blog post by Charles Smutz today titled “Flushing out leaky taps 2” ; Like others working on products that do reconstruction we too have wrangled with this issue. Here is my take on this along with a small plug towards the end.

How good can an analyzer get in detecting network vs probe packet loss when deployed in a single observation point ?

• Estimate packet loss on network : It is well known that it is hard to pin point which TCP packets were lost in the network. In the forward path you can look at retransmissions and make a guess. But you cant say all retransmitted packets were originally lost due to a late ACK or a retransmission timeout. The reverse path is even more hairy due to the use of delayed ACKs. All this means is that you can only make a rough guess of network packet loss.

• Measure holes caused by the capture toolchain : Say a TCP session completed successfully with both sides ACKing the total payload size sent by the other side. Now irrespective of network packet loss your analyzer ought to have the complete payloads from both ends.  If you reconstructed the TCP session and find holes, it is squarely the analyzers fault irrespective of the lossiness of the network. You can measure this accurately.

Analyzers can scramble the order

Frankly I have never found the “Lost Segment” flag in Wireshark to be very useful because it appears to flag all out of order segments which could be solely due to the components in capture stack shuffling the order around.  This is all too common especially if you are hanging off a SPAN port, bonding two interfaces,  or use an aggregating TAP sufficiently close to an endpoint.

Say you had a PCAP in which the remote side ACK-ed upto 1000 followed by the near side sending 1000 for the first time – this is clearly fishy. It is obviously inconceivable that the remote side would ACK 1000 which it never received up until that timestamp in your trace.  Some questions pop to mind.

• Was the packet with sequence 1000 dropped by the network  before it reached your analyzers observation point ?
• Did it make it to your observation point, but your capture stack could not reliably deliver it from the observation point to your application ?

In several captures, I have seen it is neither of these – if you just waited a little longer you would see the packet. The SPAN port, the adapter, the driver, the library, or even your code are simply not presenting the two uni-directional streams in perfect synchronized order.

Applications that reassemble need not be anything like a typical TCP endpoint. For starters  it can afford to have an really enormous window and focus purely on “hole filling” completely ignoring the ACKs.  If after the termination of a TCP flow, either by virtue of seeing a FIN/RST or a timeout,  holes still remain it is purely the fault of the capture stack. Of course baking the hole detection inside your application is debatable because it entails  more stateful tracking of each TCP connection..

If you are looking for another tool to add to your armory – read on for a small plug.

TCP Analysis with Unsniff

Unsniff has a TCP Congestion Analysis feature that can help here. This could be a useful addition to tshark. You can right click on any TCP Flow and select “TCP Congestion Analyzer”  which would then run the all the normal analysis tasks like sequence number, window, inflight data, bandwidth, etc. It also tries to estimate the cwnd (the best it can given the constraints of a single observation point mentioned earlier).  The two other things it can do are relevant to this blog post.

1. A packet by packet analysis (like Wireshark) which points out out of order, dup acks,  and retransmissions.
2. A hole analysis which points out which segments definitely did not make it to the analyzer even though they went across the network..

How to use

1. First download and install Unsniff Network Analyzer from here. It is a free download and works on all Windows platforms.

2. Import your capture file via File > Import > TCPDUMP  – You can use the capture files in this wireshark bug report

3. Once loaded you want to focus on these three tabs

1. Packets – all link layer packets
2. Sessions – a list of TCP flows, right click any of them to access the TCP Analyzer
3. User Objects – a list of user objects (files/images/etc) that are a result of reassembling content. If payload holes are present reassembly will fail, sometimes spectacularly when the content is streamed as in “gzip/chunked”

4. Switch the Sessions tab > right click on any flow > select TCP Congestion Analyzer

5. Within the congestion analyzer the Overview tab contains an item called Definitely Lost Segments for both directions. This contains a count of the number of holes  found. The analyzer is responsible for this.

6. Switch over to the Segment Wise  tab and scroll all the way to the bottom for a list of actual holes found.

7. If no holes were found, the User Objects tab would contain the reconstructed content. Here is Alice in Wonderland – an awesome read !

Good news for those on the bleeding edge of Ubuntu.

Trisul is now available for the Ubuntu 11.10 64-bit platform. You may download the packages from here http://trisul.org

Trisul is also available on these other platforms.

64-bit (recommended)

• CentOS 6.1
• CentOS 5.6+
• Ubuntu 10.04 LTS
• Ubuntu 11.10 Oneiric Ocelot

32-bit

• Ubuntu 10.04 LTS

Note : If you want to run Trisul on Security Onion get the Ubuntu 10.04 32-bit  packages.

A key capability required in network forensics is to be able to search for a string in a collection of pcap files or live traffic. You may want to search for a name with the intention of finding any relevant emails, tweets, followers, facebook, etc.  Naive packet level string matching wont work for the following reasons.

1. The transport is usually TCP so reassembly is needed before matching
2. The protocol is usually HTTP, so must process HTTP headers
3. Chunked encoding
4. GZIPped content
5. Encrypted content ( pretty much a dead end unless you are using a non EDH and have access to the private key)

Unsniff Network Analyzer allows you to search inside user objects after taking into account all of the above factors. To use the UI, switch to the User objects sheet, then press Ctrl+F and enter your patterns.  The UI works great in many cases, but many times you want to do more automatically.

These days javascript heavy websites like twitter use JSON as the dominant interchange format. Instead of sending back full HTML webpages – chunks of JSON are used to build the final user interface. So in order to present results (such as a twitter followers list) you need to wrap the results in your own application. This post tells you how to use the Scripting Interface to accomplish that.

1. Download and install the latest Unsniff from http://www.unleashnetworks.com/downloads.html

Assume you want to search for a pattern “mickey” in all content.

Step 1 : Have your program dump your packets into a libpcap file – say mypackets.pcap

Step 2 : Copy the following script into a directory in a file named searchuo.vbs

Step 3: Run the script like this

Step 4: All user objects – mails, attachments, contacts and any HTTP exchanges gzipped or chunked will be reassembled and put as separate files into that directory

Step 5 : Your code should read these files and integrate them into your UI. For example, Yahoo mails come as JSON, so maybe you can parse the JSON or display raw text with the pattern highlighted etc. Its your call.

In this mode, Unsniff will be the engine for content extraction.  You can then integrate the files into your own application.