File downloading using python with threads

Question

I\'m creating a python script which accepts a path to a remote file and an n number of threads. The file\'s size will be divided by the number of threads, when each thread c

Answer 1

You can use a thread safe "semaphore", like this:

class Counter:
  counter = 0
  @classmethod
  def inc(cls):
    n = cls.counter = cls.counter + 1 # atomic increment and assignment
    return n

Using Counter.inc() returns an incremented number across threads, which you can use to keep track of the current block of bytes.

That being said, there's no need to split up file downloads into several threads, because the downstream is way slower than the writing to disk, so one thread will always finish before the next one is downloading.

The best and least resource hungry way is simply to have a download file descriptor linked directly to a file object on disk.