Analyzing HTTP streams

This article will introduce you to various techniques for analyzing HTTP streams.

Unsniff has powerful analysis capabilities for HTTP analysis including.
    * Extract content (user objects) from HTTP streams
    * View entire HTML pages, images, flash, and media from within Unsniff
    * View all HTTP headers
    * View color-coded HTTP requests and responses
    * Full web pages including inline images, flash, stylesheets supported
    * Click through to other captured pages
    * Save pages for later analysis
    * Scripts to extract interesting data from HTTP headers
The reconstruction capabilities are so powerful that several of our customers are using Unsniff as an offline “recording” tool.

Viewing HTTP headers

From the “Packets” sheet click on any HTTP packet (except those labeled “Data continued..”).

HTTP analysis

View all HTTP headers as columns in a list

This is useful if you want to analyze all HTTP headers as a group.

Right click on any HTTP packet and select “Protocol View” from the popup menu.

The protocol details view shows all the HTTP header fields in a single list. You can select any item from the list to see the packet details in the pane below.

HTTP analysis

View entire HTTP stream

You can analyze entire HTTP sessions using the stream analysis capabilities of Unsniff. You can watch HTTP pipelining in action as well as TCP behavior including usage of RST and Keep-Alive.You can switch to the Streams sheet and watch all HTTP sessions in real-time, you can see requests and responses as they appear.

There are two ways to view a HTTP stream.

Bottom up – Like older network analyzers, you can select a HTTP packet – then right click and select “Locate Flow” from the popup menu. This takes you to the stream and the corresponding segment in that stream.

Top down – Unsniff features full stream analysis. You can simply choose the stream you are interested in from a list. You can then work your way down to link layer packets if you desire. You will find yourself using Top Down analysis more frequently as you become familiar with Unsniff

Either way you can see the entire stream as shown in the figure below, you can click on the ‘+’ icon next to each stream to show the individual segments that make up the stream.

HTTP analysis

View request response data

You can also view color-coded request/response data for the stream. Simply select the stream, right click and select “Show Data”. This shows the data in hex with blue for outgoing bytes and green for incoming data bytes. If you want to see an ASCII representation, simply right click on the data and select “UTF-8 with line breaks”
HTTP analysis

Save payload

You can save all incoming and outgoing data of the HTTP session. Simply right click on the stream and select “Reassemble and Save” from the popup menu.

View User objects (HTML pages, images, flash and other content)

User Objects is a cool new concept introduced by Unsniff. It represents any entity which is of interest to the user. When you are analysing HTTP you are probably interested in the actual HTML served, the stylesheet used, the quality of images served, even the google ads that were served up. All these entities are called “User Objects”. Unsniff will attempt to extract these user objects from the HTTP stream. The extracted user objects are then shown in the User Objects sheet.

Enable full HTML page reconstruction
To view entire HTML pages using only the content in the capture file, you must set the "Reconstruct HTTP streams" option from the Plugins->Customize menu.

Enable HTTP Reconstruction Option

To see HTML pages:

Switch to the user objects sheet
HTTP Analysis

You can then select any user object from the list. The selected user object will the rendered in the space below the list. You can also float the user object in a separate window by right click “Open in New Window”.

If you select a HTML page from the list, Unsniff will reconstruct the entire page. You can see how the page looked exactly – you can even click through to other viewed content such as other HTML pages, video, flash games, etc.

HTTP analysis

Conclusion

You have seen how Unsniffs top-down analysis combined with the powerful concept of user objects help you to analyze HTTP like never before. Experiment with these new tools. You can even try your hand at writing simple scripts to perform your own analysis.