pycurl only geting part of the response

问题

I'm making a request in python using pycurl to a URL which returns a reasonably large json formatted response. When I goto the URL in a browser I see the entire contents, but if I use pycurl and print the received data, I only see about half of what I see when I browse to the URL, and I get an error parsing the data using the json library stating :

ValueError: Unterminated string starting at: line 1 column 16078 (char 16078)

The pycurl request is this :

conn = pycurl.Curl()
conn.setopt(pycurl.URL, myUrl)
conn.setopt(pycurl.WRITEFUNCTION, on_receive)
conn.setopt(pycurl.CONNECTTIMEOUT, 30)
conn.setopt(pycurl.TIMEOUT, 30)
conn.setopt(pycurl.NOSIGNAL, 10)
conn.perform()

with the on_receive function currently just printing the data.

Does anybody know why I am only getting part of the response? I have used massive timeouts just for trying to solve this, I had initially not specified any timeouts but was still getting this error.

回答1:

in pycurl, you could set this,

import pycurl
pycurl.CONTENT_LENGTH_DOWNLOAD

try using

import Curl, pycurl
con = Curl()
con.set_option(pycurl.CONTENT_LENGTH_DOWNLOAD, 9999999999)
con.get('url' ....

also try following until it works:

pycurl.SIZE_DOWNLOAD
pycurl.REQUEST_SIZE

回答2:

You could try to access those json data with curl tool.
When you're able to get data, just translate curl options to pycurl options.

curl --help | less

来源：https://stackoverflow.com/questions/11325871/pycurl-only-geting-part-of-the-response

标签

python

json

pycurl