Access File through multiple threads

Question

I want to access a large file (file size may vary from 30 MB to 1 GB) through 10 threads and then process each line in the file and write them to another file through 10 threads

Answer 1

I would start with three threads.

1. a reader thread that reads the data, breaks it into "lines" and puts them in a bounded blocking queue (Q1),
2. a processing thread that reads from Q1, does the processing and puts them in a second bounded blocking queue (Q2), and
3. a writer thread that reads from Q2 and writes to disk.

Of course, I would also ensure that the output file is on a physically different disk than the input file.

If processing tends to be faster slower than the I/O (monitor the queue sizes), you could then start experimenting with two or more parallell "processors" that are synchronized in how they read and write their data.