How can I analyse ~13GB of data?

Question

I have ~300 text files that contain data on trackers, torrents and peers. Each file is organised like this:

tracker.txt

time torrent
            


        

Answer 1

If you could use C++, you should take a look at Boost flyweight.

Using flyweight, you can write your code as if you had strings, but each instance of a string (your tracker name, etc.) uses only the size of a pointer.

Regardless of the language, you should convert the IP address to an int (take a look at this question) to save some more memory.