Access File through multiple threads

Question

I want to access a large file (file size may vary from 30 MB to 1 GB) through 10 threads and then process each line in the file and write them to another file through 10 threads

Answer 1

I have faced similar problem in past. Where i have to read data from single file, process it and write result in other file. Since processing part was very heavy. So i tried to use multiple threads. Here is the design which i followed to solve my problem:

• Use main program as master, read the whole file in one go (but dont start processing). Create one data object for each line with its sequence order.
• Use one priorityblockingqueue say queue in main, add these data objects into it. Share refernce of this queue in constructor of every thread.
• Create different processing units i.e. threads which will listen on this queue. When we add data objects to this queue, we will call notifyall method. All threads will process individually.
• After processing, put all results in single map and put results against with key as its sequence number.
• When queue is empty and all threads are idle, means processing is done. Stop the threads. Iterate over map and write results to a file