One of our favorite tech bloggers Richard Bejtlich has put up a new post about web traffic analysis using a new open source tool. This prompted me to write this post about using Unsniff Network Analyzer to perform similar analysis. I hope our users will find this article useful. Also see this article  for a detailed overview of HTTP analysis.
Unsniff allows you to :
- Extract content from HTTP flows. We call this feature “User Objects”. Examples of user objects are images, css, flash, video, etc.
- Put together separate flows into a web page. This is a powerful feature that allows you to reconstruct web pages completely. This way CSS styles are applied, and inline images and flash content appear in the reconstructed web pages. See picture below.
- Script for automating analysis. For example: If you want to automatically extract all images from a lengthy capture into individual JPEG files, you can write a VBScript or Ruby program to do that. The Unsniff Scripting API allows such macro operations. See here for samples.
- Support for HTTPS. If you are a system admin with access to the server certificate, you can perform the same type of analysis for HTTPS. This is quite valuable for debugging website problems at the server. This powerful feature is enabled by Unsniff’s top notch support for decrypting SSLv3 and TLS traffic.
Advantages of correlating flows into a single page.
A single webpage is rendered by the browser by putting together several individual HTTP responses (flows). Sometimes these object come from different hosts. For example, a CSS stylesheet, a flash SWF file, a set of JPEG files, and the master HTML document must all be combined to produce a webpage.
The sample webpages below have been reconstructed from a tcpdump.
Sample 1: Technorati website contains embedded images, complex stylesheet. http://technorati.com
Sample 2: Richards website with CSS and embedded images. http://www.bejtlich.net/
Notes about this feature.
- Switch to the “User Objects” sheet to see the individual flows. Â
- Click on the HTML object to view reconstructed web page.
- Click on the image to see it rendered.
- Right click an item and select “View Data” to see raw contents (eg, HTML text, CSS text, Image bytes, Flash etc)
- Capture live traffic or select “File -> Import -> From TCPDUMP” to select a pre-captured file in libpcap (Ethereal or Wireshark native) format.
– – – – –
Here are solutions to some common problems emailed to us by users of this feature.
Step 1 : Enable HTTP reconstruction
- Go to menu “Plugins -> Customize” and enable the “Reconstruct HTTP Session” option. This is not the default.
- Also check if the “Extract User Objects” option is enabled. This is the default.
Step 2 : Windows Firewall Warning
If you are running a personal firewall you may get a warning about a port. You can ignore that warning, that port is used internally by the HTTP reconstruction software.
Any questions ? Please leave a comment here or use our forum.
[tags] tcpdump, sniffer, website analysis, http analysis, unsniff [/tags]
Unbrowse SNMP is a MIB Browser and Trap Receiver that is intended to be used as a powerful troubleshooting and an all around utility. Unbrowse keeps track of agents (devices) and stores information such as IP addresses, community strings, timeouts, SNMP versions, V3 usernames, auth and priv passwords, and so forth. This allows the administrator to quickly manage devices without having to enter these details each time. This will work as long as the number of devices are few, when you are dealing with hundreds of devices it is not feasible to enter them manually into a tool like Unbrowse. Dont worry, help is at hand. You can write simple scripts in VBScript or Ruby to add devices automatically into Unbrowse.