Analyze Protocol Buffers

Protocol Buffers (protobuf) is a serialization scheme recently open sourced by Google. This article is a simple tutorial on how to analyze protobuf messages using Unsniff 1.8 Beta. You just have to drop the proto files in a specific folder. Unsniff then creates a decoder on the fly using the information in the file.

We built support for protocol buffers as a project to test the dynamic decoder framework for Unsniff 2.0. It is now in a shape that many might be of practical use to a lot of people.

What are Protocol Buffers ?

Protocol Buffers (protobuf) are a way of serializing structured data for storage or transmission. The basic idea is that you write a so-called 'proto' file, then run the 'proto compiler' to produce code in a variety of languages to read and write the messages you defined in the proto file. Google uses protocol buffers for most of its internal file formats and network protocols. It was open sourced under the Apache License in July 08.

For more details, visit the project site at http://code.google.com/p/protobuf/

Feature Overview

No coding or compiling needed
Parses proto files on the fly as and when required
Can read binary files as well as network captures
Handles IP defragmentation, TCP segmentation, tunneling, TLS decryption
Presents entire messages as complete PDUs instead of ethernet packets
Handles length prefixed messages
Points out missing fields and groups
Supports all protobuf standard types including groups

The rest of this article describes how to use the tool.

Step by step guide

We figured it would be easier to just show you how to use it by example.

Step 1 : Required downloads

You need the following :

Unsniff Network Analyzer 1.8 Beta (Get it from here)
Test files (unsniff_protobuf_test.zip) (Get it from here)
A PC running Windows Vista or XP.

Step 2 : Install Unsniff 1.8 Beta

If you are installing on Vista, you need to allow UAC to proceed with the install.

Note : You may also want to install Winpcap if you wish to capture packets from the network.

Run Unsniff by double clicking on the desktop icon. The first run creates the required folder structure in the current users data folders. You will put your proto files there in Step 4.

Step 3 : Unzip the test data

Unzip the unsniff_protobuf_test.zip file from Step 1 to a temporary folder.

You show see two sub folders.

proto_binary_data : Containing protobuf encoded files and a tcpdump capture
proto_plugins : Containing 4 sample protos (addressbook, northwind, unittest, unittest_extensions)

Step 4 : Install the proto files

Unsniff will automatically detect protocols when they are placed in the user's Application Data folder. This folder is typically at C:\Users\<Username>\AppData\Roaming, it is represented by the variable %APPDATA%. We need to use the Unsniff\XMLPlugs subfolder.

For Vista : Press the Start Button and type %APPDATA% in the search box. Locate the Unleash Networks > Unsniff > XMLPlugs directory. If the XMLPlugs directory does not exist create it.

For XP : Press Start > Run and type %APPDATA%. Locate the Unleash Networks > Unsniff > XMLPlugs directory. If the XMLPlugs directory does not exist create it.

Copy all the files in the proto_plugins folder into the XMLPlugs directory.

Thats it ! You are ready to test.

Step 5 : Start analyzing

This section walks you through analyzing a binary file and then a network capture over TCP.

Analyze from a protobuf binary file

This section shows you how to analyze a protobuf encoded binary file.

Start Unsniff Network Analyzer
Click on the "G" icon on the toolbar or select Tools > Google Protocol Buffers > Import File
Select the addressbook_in_a_file.bin file from the test data folder (from Step 3)
You will be shown a list of protocols, select the PB_ADDRESSBOOK

Thats it ! You will be shown a list of Person records.

You may view the tree on the left side and raw data on the right. (Click on Raw to see bytes instead of the picture)

The screenshot below shows how the nwind.proto.bin is displayed. Each "Order" message in the nwind.proto is shown separately as a PDU.

Sample data and their corresponding proto files are described in the table below.

Sample Data File	Proto File	Source
addressbook_in_a_file.bin	addressbook.proto	Generated
addressbook_over_tcp_stream	addressbook.proto (attached to port 20000)	Generated
golden_message	unittest.proto (TestAllTypes message)	From the Google Protocol Buffers distribution package
unittest_alltypes.dat	unittest.proto (TestAllTypes message)	Generated
unittest_some_extensions	unittest.proto (TestAllExtensions message)	Generated
nwind.proto.bin	nwind.proto	From the protobuf-net project page. See here

Analyze from a network capture

You can capture live packets or import files in tcpdump (libpcap) format. Note that you need to install Winpcap for this purpose.

We have a tcpdump file for you to try called addressbook_over_tcp_stream.tcpd

Start Unsniff Network Analyzer
Select "File > Import > From Tcpdump"
Select the addressbook_over_tcp_stream.tcpd file
Observe the results

Unsniff is a multi-layer packet analyzer. Observe the sheets tab at the bottom.

PDU

The "Packets" sheet will show you the individual ethernet packets. You can see the protobuf message in packet #4 for example.
The "PDU" sheet will show you individual PDUs. Each of these PDUs can span multiple packets or many could fit in a single packet.
The "Sessions" sheet will show you each TCP Session. You can expand each session or Right-Click on a session for more options.

Adding your own proto files

We said that you can put any proto file in the %APPDATA%\Unleash Networks\Unsniff\XMLPlugs folder and it automagically turns into a dynamic plugin. Well, almost ! There is an extra thing you have to do to hook up a proto to the Unsniff dynamic plugin framework.

You need to write a tiny XML describing the protocol. You can of course simply copy an existing XML file and change some fields.

Lets take an example. When we added the addressbook.proto file to the %APPDATA% folder, we also added an addressbook.proto.xml file which looks like this.

<?xml version="1.0" encoding="utf-8"?>

<USNFProtocol id="PB_ADDRESSBOOK" shortname="PBADDR" name="PBUF (addressbook.proto)" protid="{A11F24EC-599C-486f-9E7E-56F5026F3A7A}">

<vendor>Google Protocol Buffer Sample</vendor>
<version>1.0</version>
<color>#FF0000</color>
<rootfield>AddressBook</rootfield>
<SurrogateInfo>
    <CLSID>{FAB19DF6-581E-4F46-89B4-4574A3F2DCD2}</CLSID>
    <Description>Google Protocol Buffer Surrogate Object (do not change CLSID)</Description>
    <CustomURI>addressbook.proto</CustomURI>
</SurrogateInfo>

<AccessPoints>
<AccessPoint hostid="{14D7AB53-CC51-47e9-8814-9C06AAE60189}" apvalue="20000" hostdesc="UDP Protocol"/>
<AccessPoint hostid="{77E462AB-2E42-42ec-9A58-C1A6821D6B31}" apvalue="20000" hostdesc="TCP Protocol"/>
  </AccessPoints>

</USNFProtocol>

The key fields are shown in bold. The purpose of these fields is explained below.

The id, shortname, name can be any string to help you identify the protocol
The protid is a GUID. You can create this using Visual Studio, or just type in some random hex digits to uniquely identify your protocol.
The rootfield is the name of the container message in your proto file. If you have multiple container files create a separate XML for each.
The CustomURI is the name of your proto file.

Connecting your protobuf message to TCP port number(s)

You can either specify the connections in the XML file in the AccessPoint section (see above) or you can manually associate them via the "Manage > Access Points" menu. In the above example, the GUIDs represent the protocol ID of TCP and UDP protocols. If you wish to add more than one port, simply add an extra entry. To find out the protocol ID of any protocol use the "Plugins > Manage Protocols" menu. Then click on "Details" to find out the protocol ID.

Further

Click on "Plugins > Configure" to access other options for this feature.
Options

Conclusion

What started off as an exercise to develop our API to support custom dynamic dissectors evolved to a good support for protocol buffers. This is part of Unsniff 1.8 Beta, prior to Unsniff 2.0. We have not finalized our pricing model for Unsniff 2.0. We may make it free to process upto a thousand packets or stay with the current model. We can provide a free license to involved beta testers for this feature.

We can enhance this feature based on feedback.

Email us at info dottt unleashnetworks