Python equivalent of a given wget command

Question

I\'m trying to create a Python function that does the same thing as this wget command:

wget -c --read-timeout=5 --tries=0 \"$URL\"

-c

Answer 1

urllib.request should work. Just set it up in a while(not done) loop, check if a localfile already exists, if it does send a GET with a RANGE header, specifying how far you got in downloading the localfile. Be sure to use read() to append to the localfile until an error occurs.

This is also potentially a duplicate of Python urllib2 resume download doesn't work when network reconnects

Answer 2

For Windows and Python 3.x, my two cents contribution about renaming the file on download :

1. Install wget module : pip install wget
2. Use wget :

import wget
wget.download('Url', 'C:\\PathToMyDownloadFolder\\NewFileName.extension')

Truely working command line example :

python -c "import wget; wget.download(""https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.17.2.tar.xz"", ""C:\\Users\\TestName.TestExtension"")"

Note : 'C:\\PathToMyDownloadFolder\\NewFileName.extension' is not mandatory. By default, the file is not renamed, and the download folder is your local path.

Answer 3

easy as py:

class Downloder():
    def download_manager(self, url, destination='Files/DownloderApp/', try_number="10", time_out="60"):
        #threading.Thread(target=self._wget_dl, args=(url, destination, try_number, time_out, log_file)).start()
        if self._wget_dl(url, destination, try_number, time_out, log_file) == 0:
            return True
        else:
            return False


    def _wget_dl(self,url, destination, try_number, time_out):
        import subprocess
        command=["wget", "-c", "-P", destination, "-t", try_number, "-T", time_out , url]
        try:
            download_state=subprocess.call(command)
        except Exception as e:
            print(e)
        #if download_state==0 => successfull download
        return download_state

Answer 4

Let me Improve a example with threads in case you want download many files.

import math
import random
import threading

import requests
from clint.textui import progress

# You must define a proxy list
# I suggests https://free-proxy-list.net/
proxies = {
    0: {'http': 'http://34.208.47.183:80'},
    1: {'http': 'http://40.69.191.149:3128'},
    2: {'http': 'http://104.154.205.214:1080'},
    3: {'http': 'http://52.11.190.64:3128'}
}


# you must define the list for files do you want download
videos = [
    "https://i.stack.imgur.com/g2BHi.jpg",
    "https://i.stack.imgur.com/NURaP.jpg"
]

downloaderses = list()


def downloaders(video, selected_proxy):
    print("Downloading file named {} by proxy {}...".format(video, selected_proxy))
    r = requests.get(video, stream=True, proxies=selected_proxy)
    nombre_video = video.split("/")[3]
    with open(nombre_video, 'wb') as f:
        total_length = int(r.headers.get('content-length'))
        for chunk in progress.bar(r.iter_content(chunk_size=1024), expected_size=(total_length / 1024) + 1):
            if chunk:
                f.write(chunk)
                f.flush()


for video in videos:
    selected_proxy = proxies[math.floor(random.random() * len(proxies))]
    t = threading.Thread(target=downloaders, args=(video, selected_proxy))
    downloaderses.append(t)

for _downloaders in downloaderses:
    _downloaders.start()