Analyze web traffic traces

One of our favorite tech bloggers Richard Bejtlich has put up a new post about web traffic analysis using a new open source tool. This prompted me to write this post about using Unsniff Network Analyzer to perform similar analysis. I hope our users will find this article useful. Also see this article  for a detailed overview of HTTP analysis.

Unsniff allows you to :

  • Extract content from HTTP flows. We call this feature “User Objects”. Examples of user objects are images, css, flash, video, etc.
  • Put together separate flows into a web page. This is a powerful feature that allows you to reconstruct web pages completely. This way CSS styles are applied, and inline images and flash content appear in the reconstructed web pages. See picture below.
  • Script for automating analysis. For example: If you want to automatically extract all images from a lengthy capture into individual JPEG files, you can write a VBScript or Ruby program to do that. The Unsniff Scripting API allows such macro operations. See here for samples.
  • Support for HTTPS. If you are a system admin with access to the server certificate, you can perform the same type of analysis for HTTPS. This is quite valuable for debugging website problems at the server. This powerful feature is enabled by Unsniff’s top notch support for decrypting SSLv3 and TLS traffic.

Advantages of correlating flows into a single page.

A single webpage is rendered by the browser by putting together several individual HTTP responses (flows). Sometimes these object come from different hosts. For example, a CSS stylesheet, a flash SWF file, a set of JPEG files, and the master HTML document must all be combined to produce a webpage.

The sample webpages below have been reconstructed from a tcpdump.

Sample 1: Technorati website contains embedded images, complex stylesheet. http://technorati.com

trati.jpg

Sample 2: Richards website with CSS and embedded images. http://www.bejtlich.net/

httpcap.jpg

Notes about this feature.

  1. Switch to the “User Objects” sheet to see the individual flows.  
  2. Click on the HTML object to view reconstructed web page.
  3. Click on the image to see it rendered.
  4. Right click an item and select “View Data” to see raw contents (eg, HTML text, CSS text, Image bytes, Flash etc)
  5. Capture live traffic or select “File -> Import -> From TCPDUMP” to select a pre-captured file in libpcap (Ethereal or Wireshark native) format.

– – – – –

Here are solutions to some common problems emailed to us by users of this feature.

Step 1 : Enable HTTP reconstruction

  1. Go to menu “Plugins -> Customize” and enable the “Reconstruct HTTP Session” option.  This is not the default.
  2. Also check if the “Extract User Objects” option is enabled. This is the default.

Step 2 : Windows Firewall Warning

If you are running a personal firewall you may get a warning about a port. You can ignore that warning, that port is used internally by the HTTP reconstruction software.

Any questions ? Please leave a comment here or use our forum.

[tags] tcpdump, sniffer, website analysis, http analysis, unsniff [/tags]

Author: Vivek Rajagopalan

Vivek Rajagopalan is the a lead developer for Trisul Network Analytics. Prior products were Unsniff Network Analyzer and Unbrowse SNMP. Loves working with packets , very high speed networks, and helping track down the bad guys on the internet.

5 thoughts on “Analyze web traffic traces”

  1. Hello,

    Thanks for mentioning my post.

    I tried Unsniff and used it to watch a transfer of a file using SMB. I see the Unsniff product page says “Extract files transferred across network shares” for SMB. How do I do this? Under the “Customize Plugins” menu I changed the SMB “Extract User Objects” from the default “False” to “True”, and the file I transferred was 401k.

    Sincerely,

    Richard

  2. Hi Richard,

    Thanks for trying out Unsniff.

    To decode and extract files from SMB shares, you have to do one extra step to make it work. You must associate TCP port 445 with NetBIOS Session Service. We missed it in that build of Unsniff !

    To make it work,

    1. Select “Plugins” -> “Manage Access Points” from the main menu.

    2. Click on the TCP item and press the “New TCP Access Point” button above the list.

    3. Enter 445 for the TCP Port value, and select NB-SSN from the drop down combo. (You can type NB-SSN to select)

    Now you can import or capture live SMB traffic. If the “Extract User Objects” option is set, as you have correctly done, the transferred files are shown on the user object sheet.

    You can right-click on any user object to save the transferred file.

    Known problem
    ————-
    In some cases the transferred file name is incorrect or mangled. The contents of the file are however correct.

  3. Hi Richard,

    While you are trying out SMB check out the ‘PDU sheet’.

    After importing or capturing SMB traffic, switch to the “PDU Sheet”. This gives you a list of SMB commands independent of link layer packets. A PDU can span multiple link layer packets or a single link layer packet can contain multiple SMB PDUs. Unsniff enables the PDU view for many stream based protocols (LDAP, BGP, TLS, H.323 etc)

    Regards,
    Vivek Rajan (netscript)

  4. Hi Vivek,

    Thanks for your suggestions. I did as you said but didn’t get it to work. I actually got Unsniff to throw an exception and exit. Is there a newer version? For some reason accessing your Web site is really slow.

  5. Hi Richard,

    Sorry about the website downtime. Our webhost has been really flaky for the past one week or so. We will fix that soon.

    The SMB protocol was a difficult one for us because of the lack of documentation and inconsistency in behavior. We tested with shares on XP/XP SP2/W2K SP4. Are you using a Samba share ? If you can provide a sample that crashes or fails to reassemble we can provide a fix for it.

Comments are closed.