Queueing Multiple Downloads, looking for a producer consumer API

荒凉一梦 提交于 2019-12-08 05:13:28

问题


I have an application (a servlet, but that's not very important) that downloads a set of files and parse them to extract information. Up to now, I did those operations in a loop : - fetching new file on the Internet - analyzing it.

A multi-threaded download-manager seems a better solution for this problem and I would like to implement it in the fastest way possible.

Some of the downloads are dependant from others (so, this set is partially ordered).

Mutli-threaded programming is hard and if I could find an API to do that I would be quite happy. I need to put a group of files (ordered) in a queue and get the first group of files that is completely downloaded.

Do you know of any library I could use to achieve that ?

Regards, Stéphane


回答1:


You could do something like:

BlockingQueue<Download> queue = new BlockingQueue<Download>();
ExecutorService pool = Executors.newFixedThreadPool(5);
Download obj = new Download(queue); 
pool.execute(obj); //start download and place on queue once completed
Data data = queue.take(); //get completely downloaded item

You may have to use a different kind of queue if the speed of each download is not the same. BlockingQueue is first in first out I believe.

You may want to look into using a PriorityBlockingQueue which will order the Download objects according to their Comparable method. See the API here for more details.

Hope this helps.



来源:https://stackoverflow.com/questions/6480726/queueing-multiple-downloads-looking-for-a-producer-consumer-api

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!