Julia - Parallelism for Reading a Large file

问题

In Julia v1.1, assume that I have a very large text file (30GB) and I want parallelism (multi-threads) to read eachline, how can I do ?

This code is an attempt to do this after checking Julia's documentation on multi-threading, but it's not working at all

open("pathtofile", "r") do file
    # Count number of lines in file
    seekend(file)
    fileSize = position(file)
    seekstart(file)

    # skip nseekchars first characters of file
    seek(file, nseekchars)

    # progress bar, because it's a HUGE file
    p = Progress(fileSize, 1, "Reading file...", 40)
    Threads.@threads for ln in eachline(file)
        # do something on ln
        u, v = map(x->parse(UInt32, x), split(ln))
        .... # other interesting things
        update!(p, position(file))
    end    
end

Note 1 : you need using ProgressMeter (I want my code to show a progress bar while parallelism the file reading)

Note 2 : nseekchars is an Int and the number of characters I want to skip in the beginning in my file

Note 3 : the code is working but doesn't do parellelism without Threads.@threads macro next to the for loop

回答1:

For the maximum I/O performance:

1. Parallelize the hardware - that is use disk arrays rather than a single drive. Try searching for raid performance for many excellent explanations (or ask a separate question)

2. Use the Julia memory mapping mechanism

s = open("my_file.txt","r")
using Mmap
a = Mmap.mmap(s)

3. Once having the memory mapping, do the processing in parallel. Beware of false sharing for threads (depends on your actual scenario).

来源：https://stackoverflow.com/questions/54874366/julia-parallelism-for-reading-a-large-file