How to read .cap files other than Pyshark that is faster than Scapy's rdpcap ()?

Question

I have been looking for a way to get 802.11 Packets from a .cap file into an Array. So far I have found:

• Scapy: which is kind of nice, documentation available, but too slow, when I try to open a file with size > 40 Mb, I just keeps hanging on until it consumes all my Ram (all 16 gigs of it) at which point my pc just blocks and I have to reboot it

• Pyshark: doesn't have any of Scapy's problems, but documentation is too scarce, I can't find a way to handle and get attributes for 802.11 Packets

So I was thinking maybe there are better solutions out there, or maybe someone does have some experience with pyshark?

from scapy.all import *
import pyshark
from collections import defaultdict
import sys
import math
import numpy as np
counter=0
Stats = np.zeros((14))
filename='cap.cap'

a = rdpcap(filename)
print len(a)
for p in a:
        pkt = p.payload
        #Management packets
        if p.haslayer(Dot11) and p.type == 0:
                ipcounter = ipcounter +1
                Stats[p.subtype] = Stats[p.subtype] + 1

print Stats

Note: when I launch the program with a 10 Mega bytes input (for instance) it takes about 20 seconds or so, but it does work, I wonder why is that, why is it so different from pyshark and what kind of computations is it doing?

Answer 1

You can patch scapy file named utils.py so that it won't load everything into memory

change :

def read_all(self,count=-1):
    """return a list of all packets in the pcap file
    """
    res=[]
    while count != 0:
        count -= 1
        p = self.read_packet()
        if p is None:
            break
        res.append(p)
    return res

to

def read_all(self,count=-1):
    """return an iterable of all packets in the pcap file
    """
    while count != 0:
        count -= 1
        p = self.read_packet()
        if p is None:
            break
        yield p
    return

credit goes to : http://comments.gmane.org/gmane.comp.security.scapy.general/4462

But link is now dead

Answer 2

Scapy will load all the packets to your memory and create a packetList instance. I think there are two solutions to your problem.

1. Capture packets with a filter. In my work, I have never captured more than 2MB packets since I only capture on one wireless channel once.
2. Divide the huge packet file into several smaller parts. And then deal with them.

Hope it helps.

Answer 3

If pyshark suits your needs, you can use it like so:

cap = pyshark.FileCapture('/tmp/mycap.cap')
for packet in cap:
    my_layer = packet.layer_name # or packet['layer name'] or packet[layer_index]

To see what available layers you have and what attributes they have, just print them (or use layer/packet.pretty_print()) or use autocomplete or look at packet.layer._all_fields. For instance packet.udp.srcport.

What is missing in the documentation?

Note that you can also apply a filter as an argument to the FileCapture instance (either a display filter or a BPF filter, see docs)

Answer 4

Have you tried dpkt? It has a nice Reader interface which seems to lazy-load packets (I have loaded 100MB+ pcap files with it, no problem).

Sample:

from dpkt.pcap import Reader

with open(...) as f:
    for pkt in Reader(f):
        ...

Answer 5

Thanks to @KimiNewt and After spending some time with the pyshark Source code, I got some understanding of the nuts and bolts of it

PS : opening a 450 MB file using pyShark doesn't take any time at all, and the data access is fairly easy. I don't see any downsides of using it at the moment, but I will try to keep this post up to date as I advance in my project.

This is a sample code of 802.11 packet parsing using pyShark, I hope it will help those working on similar projects.

import pyshark

#Opening the cap file
filename='data-cap-01.cap'
cap = pyshark.FileCapture(filename)

#Getting a list of all fields of this packet on the level of this specific layer
#looking somthing like this :['fc_frag', 'fc_type_subtype',..., 'fc_type']
print cap[0]['WLAN']._field_names

#Getting the value of a specific field, the packet type in
#this case (Control, Management or Data ) which will be represented by an Integer (0,1,2)
print cap[0]['WLAN'].get_field_value('fc_type')

I will be later on working on packet decryption for WEP and WPA and getting 3rd layer headers, so I might add that too.

Answer 6

with PcapReader('filename.pcapng') as pcap_reader:
    for pkt in pcap_reader:
        #do something with the packet
        ...

this works GOOD!

PcapReader just like xrange() to range()