Sorting gigantic binary files with C#

Question

I have a large file of roughly 400 GB of size. Generated daily by an external closed system. It is a binary file with the following format:

byte[8]byte[4]byte[n         


        

Answer 1

At great way to speed up this kind of file access is to memory-map the entire file into address space and let the OS take care of reading whatever bits from the file it needs to. So do the same thing as you're doing right now, except read from memory instead of using a BinaryReader/seek/read.

You've got lots of main memory, so this should provide pretty good performance (as long as you're using a 64-bit OS).