Access File through multiple threads

Question

I want to access a large file (file size may vary from 30 MB to 1 GB) through 10 threads and then process each line in the file and write them to another file through 10 threads

Answer 1

Since order need to be maintained, so problem in itself says that reading and writing cannot be done in parallel as it is sequential process, the only thing that you can do in parallel is processing of records but that also doesnt solve much with only one writer.

Here is a design proposal:

1. Use One Thread t1 to read file and store data into a LinkedBlockingQueue Q1
2. Use another Thread t2 to read data from Q1 and put into another LinkedBlockingQueue Q2
3. Thread t3 reads data from Q2 and writes into a file.
4. To make sure that you dont encounter OutofMemoryError you should initialize Queues with appropriate size
5. You can use a CyclicBarrier to ensure all thread complete their operation
6. Additionally you can set an Action in CyclicBarrier where you can do your post processing tasks.

Good Luck, hoping you get the best design.

Cheers !!