Python requests isn't giving me the same HTML as my browser is

Question

I am grabbing a Wikia page using Python requests. There\'s a problem, though: the requests request isn\'t giving me the same HTML as my browser is with the very

Answer 1

I had a similar issue:

• Identical headers with Python and through the browser
• JavaScript definitely ruled out as a cause

To resolve the issue, I ended up swapping out the requests library for urllib.request.

Basically, I replaced:

import requests

session = requests.Session()
r = session.get(URL)

with:

import urllib.request

r = urllib.request.urlopen(URL)

and then it worked.

Maybe one of those libraries is doing something strange behind the scenes? Not sure if that's an option for you or not.