grequests

How can I send like 1000 requests the fastest way? I know that you can send multiple request with grequests : urls = [ 'sample.url/1', 'sample.url/2', ... ] request = (grequests.get(u) for u in urls) print grequests.map(request) But the return is not the content. What I need is to get the json data, so for example something like this: request = (grequests.get(u) for u in urls) content = grequests.json(request) The items returned are not the content, but they do include the content. You can fetch all of the content like so: result = grequests.map(request) content = '\n'.join(r.content for r in

I have small code snippet as below: import requests import multiprocessing header = { 'X-Location': 'UNKNOWN', 'X-AppVersion': '2.20.0', 'X-UniqueId': '2397123', 'X-User-Locale': 'en', 'X-Platform': 'Android', 'X-AppId': 'com.my_app', 'Accept-Language': 'en-ID', 'X-PushTokenType': 'GCM', 'X-DeviceToken': 'some_device_token' } BASE_URI = 'https://my_server.com/v2/customers/login' def internet_resource_getter(post_data): stuff_got = [] response = requests.post(BASE_URI, headers=header, json=post_data) stuff_got.append(response.json()) return stuff_got tokens = [{"my_token":

I need to make 100k head requests, and I'm using gevent on top of requests. My code runs for a while, but then eventually hangs. I'm not sure why it's hanging, or whether it's hanging inside requests or gevent. I'm using the timeout argument inside both requests and gevent. Please take a look at my code snippet below, and let me know what I should change. import gevent from gevent import monkey, pool monkey.patch_all() import requests def get_head(url, timeout=3): try: return requests.head(url, allow_redirects=True, timeout=timeout) except: return None def expand_short_urls(short_urls, chunk

I am using the grequests python library to send GET requests asynchronously to our server. I cant figure out how to get the server response time for each individual request within the pool of requests being sent out? unsentrequests=(grequests.get(u) for u in self.urls) # make a pool of requests responses=grequests.map(unsentrequests) # send the requests asynchronously To get the start time of a request-response pair I could do the following: grequests.get(u,headers={'start':time.time()) print responses[0].request.headers['start_time'] But how can I grap the time that the response was received

I am using python request.post() to send data to a remote database. How should I send more than one request(around 20-30) on same URL using different data using python ? Also, will sequential work fine for this case or do I need to make the requests in parallel ? You should look at grequests which uses requests and gevent import grequests urls = [ 'http://www.heroku.com', 'http://python-tablib.org', 'http://httpbin.org', 'http://python-requests.org', 'http://kennethreitz.com' ] rs = (grequests.get(u) for u in urls) grequests.map(rs) [<Response [200]>, <Response [200]>, <Response [200]>,

I'm running an AWS Lambda script with a Python 2.7 runtime. However, whenever it initializes it begins to import the grequests library but fails because of it's dependency on gevent: Gevent is required for grequests. It seems it is successfully finding the grequests library (since it knows it needs gevent) but fails. What I've tried so far: pip install --ignore-installed grequests -t . pip install --ignore-installed grequests -t ./lib pip install --ignore-installed gevent -t . pip install --ignore-installed gevent -t ./lib And then I compress the contents of the directory and upload to AWS per

I am very new to all of this; I need to obtain data on several thousand sourceforge projects for a paper I am writing. The data is all freely available in json format at the url http://sourceforge.net/api/project/name/[project name]/json. I have a list of several thousand of these URL's and I am using the following code. import grequests rs = (grequests.get(u) for u in ulist) answers = grequests.map(rs) Using this code I am able to obtain the data for any 200 or so projects I like, i.e. rs = (grequests.get(u) for u in ulist[0:199]) works, but as soon as I go over that, all attempts are met