Endace DAG Packet Capture Cards: Part 1

If you ever need to do high-bandwidth packet capture, you should know about FPGA packet-capture cards. An FPGA, or Field Programmable Gate Array, is an integrated circuit made up of a large number of logic gates that can be reprogrammed electronically. It has many of the advantages of an ASIC, or Application-Specific Integrated Circuit, without the large tooling costs. Since it can be reprogrammed in the field, it is much easier to make fixes or enhancements. An FPGA makes it possible to put a lot more intelligence in the network card, allowing it to filter and timestamp packets before they reach the computer. This, combined with a circular buffer and Direct Memory Access, vastly reduces the number of I/O interrupts the computer must processing, making it possible for a given computer to handle much higher data rates.

There are several companies that make FGPA packet-capture cards, including Endace, Napatech, and Silicom. (However, Endace has announced that they are no longer selling packet-capture cards after August 31, 2020, and that end of support for existing cards is August 31, 2021.)

I am most familiar with Endace cards, so I am going to talk about them. Endace is a New Zealand company that has been making high-speed DAG (Data Acquisition and Generation) packet capture cards since 2001. The picture above is their DAG 7.5G4 card, which has four interfaces for 10/100/1000Base-T, or 1 GBe optical Ethernet. (It uses small form-factor pluggable, or SFP, transceivers, allowing the same card to be used for copper or optical connections.) Other Endace DAG cards can handle up to 40 GBe optical, and there are special cards for directly processing Synchronous Optical Networking, or SONET, streams.

Rather than generating an interrupt for each packet received, Endace DAG cards write captured packets to a circular buffer in memory using Direct Memory Access (DMA). This greatly reduces the load on the CPU. The captured records are prefixed with a header that contains a 64-bit timestamp, which of the card’s interfaces the packet was captured on, and other, optional information. 

Endace supplies two utilities for capturing packets and writing them to a file: dagsnap writes captured packets to a single file, while dagbis creates a new file every time a specified size is reached. The records are written ERF format, which includes the timestamp header. This format is supported directly by Wireshark. The dagconvert utility will convert it them to standard PCAP format, for programs that do not support ERF format.

Extra features are available when Endace DAG cards are accessed via the DAG Streams C API.

One feature is Hash Load Balancing, which allows the processing of packets to be spread over several CPU cores. A CRC-based hash code is generated for each packet, based on its source and destination addresses, resulting hash codes are divided up among as many buckets as there are cores. The result is that packets are pseudo-randomly distributed to the various cores, but all the packets for a particular conversation all go to the same core.

It is also possible to assign a color (really just a number) to each packet, based on a number of characteristics about the packet. Each packet is then steered to one or more streams, based on its color. This makes it possible to have streams to analyze particular types of packets. And being able to steer a packet to multiple streams makes it possible, for example, to capture full packets with one stream, and capture just the headers for analysis in another stream.

Many network analysis tools use the libpcap network traffic capture library. If the Endace-DAG-enabled version is used, these programs can support reading streams from Endace DAG cards. The different streams appear as individual devices, so stream 2 on the first DAG card would be dag0:2. This allows programs like the Wireshark packet analyzer and the Snort network intrusion detection and prevention system to read a particular stream, while other streams are used by other programs.

Some network analysis tools, like Zeek (formerly Bro) network security monitor with the zeek-dag package, and the Suricata intrusion detection system with DAG support, can run multiple threads to make use of the Hash Load Balancing features of the Endace DAG cards.

Usually, DAG cards are behind a network TAP. They do not have MAC addresses, and they are just used to receive packets. However, they can also be used to re-transmit packets captured in an ERF file, for network capacity testing. There is utility, dagfwddemo, that receives packets, filters them based on Berkeley Packet Filter (BPF) expressions loaded into the card, and sends them back out another port, without them ever having to go through the CPU of the computer.

And if you find you have more ports on your card than you need to capture from, another utility, dagnetdev, will make the spare ports work like regular Network Interface Cards (NICs).

Endace DAG cards have have timing inputs for IRIG-B code or Pulse Per Second signals, such as those produced by GPS devices. Some of the later cards also support IEEE-1588 Precision Time Protocol. For forensic investigations, this makes it easier to correlate logs from various cards.

For stock traders that need to correlate logs from many Endace DAG cards in order to comply with the regNMS rule 613 or MiFID II regulations, Endace has a device called the TDS-24 that will distribute timing signals to 24 DAG cards, and can be daisy-chained to support over 500 DAG cards. And the Provenance feature can insert extra data in the log to indicate which device the records came from, the state of its clock synchronization, and several other fields that are useful when combining logs from several devices.

Endace DAG cards are supported on most Linux distributions.

Part 2 will show an example of setting up an Endace DAG card to route packets to multiple streams.


Leave a Reply

Your email address will not be published. Required fields are marked *