Python Requests - Use navigate site by servers IP

Question

I want to crawl a site, however cloudflare was getting in the way. I was able to get the servers IP, so cloudflare won\'t bother me.

How can I utilize this in the re

Answer 1

You will have to set a custom header host with value of example.com, something like:

requests.get('http://127.0.0.1/foo.php', headers={'host': 'example.com'})

should do the trick. If you want to verify that then type in the following command (requires netcat): nc -l -p 80 and then run the above command. It will produce output in the netcat window:

GET /foo.php HTTP/1.1
Host: example.com
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.6.2 CPython/3.4.3 Windows/8