We recently encountered an interesting problem with a leading cloud based financial services provider who uses Trisul for deep monitoring of application and bandwidth usage. Hope this helps other folks facing a similar problem.

The problem

Our customer is a cloud based financial services company which  buys bandwidth in bulk from an upstream provider. The upstream provider gives them rough SNMP based traffic reports for each day.  From their end,  the financial services company uses Trisul to break up the traffic and provide much more fine grained reports to their end users. The problem was that the SNMP based numbers did not match Trisul’s packet based numbers. Trisul always showed a 3%-4% smaller number daily. For their volumes this added up to $80 daily.

What happened to the missing packets

We banged our heads for a couple of days trying to figure out what was going on. We confirmed the following

• Trisul wasnt dropping any packets (even at a moderately high sustained rate of 30-40,000 pps) – neither was the kernel.
• Trisul was in inline mode w/ hardware bypass, not hanging off a tap or span.
• The cable directly connected the switch from upstream end to Trisul
• The cable was brand new in a high quality data center

It turns out the SNMP stats were produced by polling the ifHCInOctets/ifHCOutOctets counters in mib-2 interfaces group.  For Ethernet interfaces this is what RFC 3635 says.

The Interface MIB octet counters, ifInOctets, ifOutOctets,

ifHCInOctets and ifHCOutOctets, MUST include all octets in valid frames sent or received on the interface, including the MAC header and FCS, but not the preamble, start of frame delimiter, or extension octets.

 

Note that the 7 byte preamble + 1 byte SFD isnt included, just the 4 byte CRC.

Trisul of course did not see the CRC because we were relying on the PF_PACKET mechanism to supply packets.  Once we added the 4 bytes to every packet we found that the numbers tallied.

How to compensate for FCS

We find that only folks who use Trisul to check or implement billing care about matching the SNMP counters. So we introduced a new option on a per interface basis called AddEthernetFCS.  This is disabled by default. When enabled it will add 4 bytes to the reported packet length. To use this option Login as admin and select Manage Capture Profiles > Select an interface > Enable the AddEthernetFCS option.

————————

Trisul Network Analytics is an exciting new platform for gaining excellent visibility into all network activity. It has a unique engine that records all traffic, flows, and even packets for future analysis.

It is totally free for monitoring a recent 3-day window. Get it now.

In Part-1, we used the Unsniff Scripting API to read a PCAP file and print the Certificate Chain for all HTTPS connections in it.

What if you did not have a PCAP (Packet Capture) file ?

In the real world, getting a sufficiently narrow PCAP file is the problem. If you have an NSM system with a strong API – you can conjure up whatever analysis you want on previously captured data. Using the Trisul API  (a.k.a Trisul Remote Protocol or TRP) you can write Ruby scripts to :

• securely connect to a Trisul Probe
• search for various types of data (traffic stats, flows, alerts, URLs, DNS, and packets)
• pull out required PCAPs for further deep processing by Unsniff or Wireshark

Task for Part 2

We have a TRP Server running on demo2-dot-trisul-dot-org – your task is to connect to this server, search for all HTTPS activity from a suspicious host 192.168.1.105 over the past 1 month and print out the certificate chain of each connection. This will help you cut through several gigabytes of packets.

The setup for TRP

Try it out first

Before we explain the code, lets gratify ourselves by running the sample code and getting some output.

1. Install Ruby and the trisulrp gem (see the tutorial for help)
2. Install Unsniff Network Analyzer (free) from the downloads page. You need this to do the deep analysis. Sorry this is a Windows MSI. If you are running Linux just comment out the print_cert_stack function.
3. Download the csx.rb script from the samples page
4. Download the demo client cert and key from  and place them in the same directory

Note: You dont need to install Trisul or the Web Interface. We already have a probe running on demo2trisulorg. You are just setting up a script client environment.

Run as below (password for the private key file is ‘client’ )

The csx.rb code

The code is quite straightforward.

Step 1. We connect to TRP and retrieve 20 HTTPS flows for IP 192.168.1.105 for the entire time interval available. The message used here is KeySessionActivity (give me all flows by IP and/or Port)

Step 2 : For each flow in the response, pull the packets out of Trisul. The message used here is FiltereredDatagramsRequest for each flow. Note we have capped the :max_bytes at 20,000. We use a trick here, we only retrieve the first 20K bytes of each flow because the Server Certificate is usually exchanged at the very beginning of a SSL session. This dramatically reduces the data transferred.

The full code is available as csx.rb from the TRP Samples Page.

Have fun !

—

We just released Trisul 2.4. The major new feature in it is the API (Trisul Remote Protocol).  Download it and let it watch your network. You never know when you may need its data.

In this two part post, we are going to see how we can utilize the scripting capabilities of Unsniff and Trisul to build our own automated analysis tools. The task here is to scan all HTTPS traffic and print the certificate chain for each session seen.

In Part 1 (this post) : We will use a standalone Unsniff script in Ruby to extract this information from a packet capture.

In Part 2 (next post) – We will see how we can use a Ruby script to connect to a Trisul sensor, pull out all HTTPS certificate chains accessed by a particular IP over 24 hours.

In the following example, the session between 212.149.50.181 and 192.168.1.5 is authenticated by the chain shown below it. The chain is :  www.commerzbaking.de is signed by TC Trust Centre which is in turn signed by Cybertrust Global Root which is in turn signed by GTE Cybertrust Global Root.

Where do we get this information ? As part of the SSL/TLS handshake the remote server sends its certificate chain in a protocol message called Server Certificate. Our Ruby script will look for these messages and print out the chain in the following format..

Using the Unsniff Scripting API

We will write a tiny Ruby script and the Unsniff Scripting API to accomplish this task.  (full code available here)

1. Pull out all the PDUs containing a Server Certificate.
2. For each cert in chain; navigate and print the commonName and organizationName of the subject and issuer.

Key methods in the script

How to pull out all reassembled SSL/TLS PDU records which contain a Server Certificate?

A quick note : Users of Wireshark maybe a bit confused here. In Wireshark the unit of analysis is the link layer packet, i.e Ethernet or Wireless frames. Typically the final packet in the stream contains a link to reassembled content. Unsniff monitors PDUs as top level units. What you see in the PDU sheet are reassembled messages without regard for packet boundaries. TLS is a message layer built on top of a bytestream layer TCP. So we need to work with PDU’s for this example.

Luckily for us, Unsniff sets the Description field of each PDU to contain the names of handshake messages.  So we can just select the PDUs which contains “Server Certificate” anywhere in its description..

Collect all the certificates in the stack

Frequently a Server Hello + Server Certificate + Server Hello Done are packed intoa single PDU. We only need to work with the “Server Certificate” its easy to select this as the code shows below.

In the above example, we are wrapping the pdu.Fields method in an Enumerable wrapper . This allows us to mix-in methods like Find and Select to the Unsniff Scripting objects which are backed by C++ classes.

How to pull out issuer and subject names ?

At this point, we now have a handle to each certificate in the chain. Our next and final task is to print the issuer and subject details. Our friend is the FindField method

Running the script

The complete script (xcert.rb) is available at the Unsniff Scripting Samples pages on our new Wiki.

To run this script.

1. Download Unsniff Network Analyzer (its a free download). Note that Unsniff is a Windows app.
2. Download and install the latest Ruby Windows One Click Installer 
3. Download the xcert.rb script
4. Capture some packets and save it as USNF format. You can also work with PCAP files directly, but you have to modify the script to import the PCAP file into USNF format first. See samples in Import / Export section for hints.
5. Run the script

For the analyst with some scripting skillz ?

The philosophy of both Unsniff and Trisul is to put powerful tools in the hands of the analyst. With mid-level skills in Ruby (or even VBScript) you can do amazing things automatically. Take out the tedium of clicking through to perform repeatable tasks. As an exercise you can extend this script to do the following:

• download the root certificates included with Firefox and compare the CAs in your chain for validity

Part 2 : Add Trisul scripting

We have seen how you can do such deep analysis with Unsniff scripting. But this requires you to have a capture file of a manageable size. What if you wanted to

• check all of your traffic during 9AM to 11AM yesterday and print the cert stack of each SSL session
• analyze all SSL sessions during the last week from a workstation 192.168.1.22 – Say you found malware in this machine and want to audit its past activity
• flag all TLS sessions which did not have a valid root certificate, yet the user overrode the Firefox warning and proceeded to complete his/her transaction

For this kind of analysis involving long time ranges and multi-gigabytes of data, Trisul must be used in conjunction with Unsniff. Our latest release of Trisul 2.4 features a very powerful remote scripting API called TRP (Trisul Remote Protocol). Part – 2 will extend this sample to show how you can perform this analysis over large time ranges.

Till next time. Happy packet hunting.