What is the fastest way to read 10 GB file from the disk?

后端 未结 13 3642
你的背包
你的背包 2021-02-20 01:19

We need to read and count different types of messages/run some statistics on a 10 GB text file, e.g a FIX engine log. We use Linux, 32-bit, 4 CPUs, Intel, coding in Perl but the

13条回答
  •  我在风中等你
    2021-02-20 01:51

    Parse the file once, reading line by line. Put the results in a table in a decent database. Run as many queries as you wish. Feed the beast regularly with new incoming data.

    Realize that manipulating a 10 Gb file, transferring it across the (even if local) network, exploring complicated solutions etc all take time.

提交回复
热议问题