Python: Need to request only 20 times per minute

问题

I have made a python code that uses a api to request some data, but the api only allows for 20 requests per minute. I am using urllib to request the data. Also I am using a for loop because the data is located in a file:

for i in hashfile:
    hash = i
    url1 = "https://hashes.org/api.php?act=REQUEST&key="+key+"&hash="+hash
    print(url1)
    response = urllib.request.urlopen(url2).read()
    strr = str(response)

    if "plain" in strr:
        parsed_json = json.loads(response.decode("UTF-8"))
        print(parsed_json['739c5b1cd5681e668f689aa66bcc254c']['plain'])
        writehash = i+parsed_json
        hashfile.write(writehash + "\n")
    elif "INVALID HASH" in strr:
        print("You have entered an invalid hash.")
    elif "NOT FOUND" in strr:
        print("The hash is not found.")
    elif "LIMIT REACHED" in strr:
        print("You have reached the max requests per minute, please try again in one minute.")
    elif "INVALID KEY!" in strr:
        print("You have entered a wrong key!")
    else:
        print("You have entered a wrong input!")

Is there a way to make it just do 20 requests per minute? or if that isn't possible can I make it timeout after 20 tries? (btw, it's just part of the code)

回答1:

You want to use the time module. Add a time.sleep(3) at the end of each loop and you will get 20 requests a minute max.

回答2:

time.sleep(3) guarantees that you won't make more than 20 requests per minute with your code but it may delay allowed requests unnecessarily: imagine you need to make only 10 requests: time.sleep(3) after each request makes the loop to run half a minute but the api allows you to make all 10 requests at once (or at least one immediately after another) in this case.

To enforce limit of 20 requests per minute without delaying initial requests, you could use RatedSemaphore(20, period=60):

rate_limit = RatedSemaphore(20, 60)
for hash_value in hash_file:
    with rate_limit, urlopen(make_url(hash_value)) as response:
        data = json.load(response)

You could even make several requests at once while obeying the rate limit:

from multiprocessing.pool import ThreadPool

def make_request(hash_value, rate_limit=RatedSemaphore(20, 60)):
    with rate_limit:
        try:
            with urlopen(make_url(hash_value)) as response:
                return json.load(response), None
        except Exception as e:
                return None, e


pool = ThreadPool(4) # make 4 concurrent requests
for data, error in pool.imap_unordered(make_request, hash_file):
    if error is None:
        print(data)

来源：https://stackoverflow.com/questions/33201867/python-need-to-request-only-20-times-per-minute

标签

python

timer

request

timeout

urllib