Python - Example of urllib2 asynchronous / threaded request using HTTPS

Question

I\'m having a heck of a time getting asynchronous / threaded HTTPS requests to work using Python\'s urllib2.

Does anyone out there have a basic example that implements u

Answer 1

here is an example using urllib2 (with https) and threads. Each thread cycles through a list of URL's and retrieves the resource.

import itertools
import urllib2
from threading import Thread


THREADS = 2
URLS = (
    'https://foo/bar',
    'https://foo/baz',
    )


def main():
    for _ in range(THREADS):
        t = Agent(URLS)
        t.start()


class Agent(Thread):
    def __init__(self, urls):
        Thread.__init__(self)
        self.urls = urls

    def run(self):
        urls = itertools.cycle(self.urls)
        while True:
            data = urllib2.urlopen(urls.next()).read()


if __name__ == '__main__':
    main()