Python Requests - Use navigate site by servers IP

Question

I want to crawl a site, however cloudflare was getting in the way. I was able to get the servers IP, so cloudflare won\'t bother me.

How can I utilize this in the re

Answer 1

Answer for HTTPS/SNI support: Use the HostHeaderSSLAdapter in the requests_toolbelt module:

The above solution works fine with virtualhosts for non-encrypted HTTP connections. For HTTPS you also need to pass SNI (Server Name Identification) in the TLS header which as some servers will present a different SSL certificate depending on what is passed in via SNI. Also, the python ssl libraries by default don't look at the Host: header to match the server connection at connection time.

The above provides a straight-forward for adding a transport adapter to requests that handles this for you.

Example

import requests

from requests_toolbelt.adapters import host_header_ssl

# Create a new requests session
s = requests.Session()

# Mount the adapter for https URLs
s.mount('https://', host_header_ssl.HostHeaderSSLAdapter())

# Send your request
s.get("https://198.51.100.50", headers={"Host": "example.org"})