Requests - get content-type/size without fetching the whole page/content

Question

I have a simple website crawler, it works fine, but sometime it stuck because of large content such as ISO images, .exe files and other large stuff. Guessing content-type using

Answer 1

Sorry, my mistake, I should read documentation better. Here is the answer: http://docs.python-requests.org/en/latest/user/advanced/#advanced (Body Content Workflow)

tarball_url = 'https://github.com/kennethreitz/requests/tarball/master'
r = requests.get(tarball_url, stream=True)
if int(r.headers['content-length']) > TOO_LONG:
  r.connection.close()
  # log request too long