How to capture metadata for https traffic using Bro

Question

I have capture some data using following tcpdum command.

tcpdump -i eth1 -w eth1_data.pcap -X

After that I have run following command to analyze eth1_data.pcap file using Bro.

bro -r eth1_data.pcap local "Site::local_nets += { 10.0.0.0/8 }"

I am using Bro 2.4.1 version. I have not changed any configuration other than above in command. There are a lot of files that are generated after above command execution. Now I have to find bytes transfered by some social websites e.g. htts://www.twitter.com. I have not found correct information about above sites in http.log.

I know that content for https sites is encrypted but metadata can be extracted (as app_stats.log also gives some info).

Is it correct that I should pick UID from ssl.log and then find resp_ip_bytes transfered from conn.log with same uid ?

Or any alternate way to get meta data information of https sites ?

Answer 1

It sounds like you are on the right path already.

One small note to keep in mind with using resp_ip_bytes is that the size on that will include the IP and TCP header from each packet. Also, there is no TCP reassembly being accounted for in that number so packet retransmits will increase the number even though no new data was sent. If you are looking for the content body size, you should use the resp_bytes field but keep in mind that this will still have all of the SSL/TLS framing and the content of that count be compressed.

Another small note I wanted to make is that we removed the app_stats script from 2.5 due to lack of maintenance and general issues with the approach.

Is there something in particular that you are looking for?