high performance packet captures...

[BLinux]

just wondering if anyone here has experience with high performance packet captures?

i'm researching this topic for a project and have come across tools like netmap, intel dpdk, pfring zc, etc. how do they compare? i was also looking at n2disk for packet indexing, but what are some comparable alternatives i should also look at?

 

[Evan]

Aside from the capture product..

Gigamon. Use the gigamon to filter the packets first to get what you want
Of course assuming you don't for whatever need to get the full contents of every last packet.

 

[BLinux]

Aside from the capture product..

Gigamon. Use the gigamon to filter the packets first to get what you want
Of course assuming you don't for whatever need to get the full contents of every last packet.

actually, i do need to record every packet for post-analysis. was looking at n2disk since it also indexes the pcap files and can later extract the packets based on BPF filter using the index.

 

[Evan]

https://www.net.in.tum.de/publications/papers/gallenmueller_ancs2015.pdf

https://sharkfest.wireshark.org/assets/presentations/I3: Sharkfest_2014_ABrown - Copy.pdf


Above link may help with some info, if I see one of the guys at work who will know about this I will ask for you.

 

[BLinux]

playing with n2disk today, but results are disturbing....

source test pcap file:


# capinfos test.pcap
File name: test.pcap
File type: Wireshark/tcpdump/... - pcap
File encapsulation: Ethernet
Packet size limit: file hdr: 65535 bytes
Number of packets: 59 k
File size: 57 MB
Data size: 56 MB
Capture duration: 598 seconds
Start time: Thu Jun 15 20:18:15 2017
End time: Thu Jun 15 20:28:14 2017
Data byte rate: 94 kBps
Data bit rate: 755 kbps
Average packet size: 944.06 bytes
Average packet rate: 100 packets/sec
SHA1: 487ab0f860c654f658823cc81c97b55b9166c833
RIPEMD160: 476164cfbe7fadcf7265d332f35571dd984d42a2
MD5: 8304068e903baf9f8cd1b4213d822da9
Strict time order: False

going to try extracting just the "port 80"/http traffic. so source has:


# tcpdump -r test.pcap | grep -c http
reading from file test.pcap, link-type EN10MB (Ethernet)
21834

21834 packets that are HTTP.

now, extracting with npcapextract:

# time npcapextract -a test.pcap -i test.idx -f "port 80" -o /var/tmp/results.pcap
15/Jun/2017 21:23:38 [npcapextract.c:878] Welcome to npcapextract (r4812) [Penryn] - (C) 2012-17 ntop.org
Error: invalid packet length 17442 [offset: 1021077]
15/Jun/2017 21:23:38 [npcapextract.c:925] 391 packets (522971 bytes) matched the filter in 0.058 sec.
15/Jun/2017 21:23:38 [npcapextract.c:937] Dumped into 1 different output files.
15/Jun/2017 21:23:38 [npcapextract.c:947] Total processing time: 0.058 sec.

real 0m0.072s
user 0m0.006s
sys 0m0.021s

WHAT??? only 391 packets?? seriously?:

# capinfos /var/tmp/results.pcap
File name: /var/tmp/results.pcap
File type: Wireshark/tcpdump/... - pcap
File encapsulation: Ethernet
Packet size limit: file hdr: 65535 bytes
Number of packets: 391
File size: 522 kB
Data size: 516 kB
Capture duration: 12 seconds
Start time: Thu Jun 15 20:20:45 2017
End time: Thu Jun 15 20:20:58 2017
Data byte rate: 41 kBps
Data bit rate: 335 kbps
Average packet size: 1321.46 bytes
Average packet rate: 31 packets/sec
SHA1: 7899e0d6287738f845ec827732e937bbcdc72a64
RIPEMD160: 106872a82f37c7718faaf733ae82efb477f7a6f4
MD5: 8fd54c63d4053d7e7e7048c83e34585b
Strict time order: True

Do the same thing with tcpdump:

# time tcpdump -r test.pcap -nn -w /var/tmp/results2.pcap "port 80"
reading from file test.pcap, link-type EN10MB (Ethernet)

real 0m0.181s
user 0m0.039s
sys 0m0.142s

# capinfos /var/tmp/results2.pcap
File name: /var/tmp/results2.pcap
File type: Wireshark/tcpdump/... - pcap
File encapsulation: Ethernet
Packet size limit: file hdr: 65535 bytes
Number of packets: 21 k
File size: 46 MB
Data size: 46 MB
Capture duration: 153 seconds
Start time: Thu Jun 15 20:20:45 2017
End time: Thu Jun 15 20:23:19 2017
Data byte rate: 303 kBps
Data bit rate: 2,424 kbps
Average packet size: 2160.56 bytes
Average packet rate: 140 packets/sec
SHA1: 49c6589081604cd1dc254ec89d621410cc37da30
RIPEMD160: 054acd7ac1742329bc1fb65a25566099e21225de
MD5: eddcee37a116eead44ae3ad1264754e1
Strict time order: True

Ok, those results look more accurate... but, what am I doing wrong with npcapextract?