We need to read and count different types of messages/run some statistics on a 10 GB text file, e.g a FIX engine log. We use Linux, 32-bit, 4 CPUs, Intel, coding in Perl but the
Have you thought of streaming the file and filtering out to a secondary file any interesting results? (Repeat until you have a manageble size file).