python requests return a different web page from browser or urllib

问题

I use requests to scrape webpage for some content.
When I use

import requests  
requests.get('example.org')

I get a different page from the one I get when I use my broswer or using

import urllib.request
urllib.request.urlopen('example.org')

I tried using urllib but it was really slow.
In a comparison test I did it was 50% slower than requests !!

How Do you solve this??

回答1:

After a lot of investigations I found that the site passes a cookie in the header attached to the first visitor to the site only.

so the solution is to get the cookies with head request, then resend them with your get request

import requests  
# get the cookies with head(), this doesn't get the body so it's FAST
cookies = requests.head('example.com')
# send get request with the cookies
result = requests.get('example.com', cookies=cookies)

Now It's faster than urllib + the same result :)

来源：https://stackoverflow.com/questions/43301477/python-requests-return-a-different-web-page-from-browser-or-urllib