Force requests to use IPv4 / IPv6

Question

How to force the requests library to use a specific internet protocol version for a get request? Or can this be achieved better with another method in Python? I

Answer 1

This is totally untested and will probably require some tweaks, but combining answers from Using Python “requests” with existing socket connection and how to force python httplib library to use only A requests, it looks like you should be able to create an IPv6 only socket and then have requests use that for its connection pool with something like:

try:
    from http.client import HTTPConnection
except ImportError:
    from httplib import HTTPConnection

class MyHTTPConnection(HTTPConnection):
    def connect(self):
        print("This actually called called")
        self.sock = socket.socket(socket.AF_INET6)
        self.sock.connect((self.host, self.port,0,0))
        if self._tunnel_host:
            self._tunnel()

requests.packages.urllib3.connectionpool.HTTPConnection = MyHTTPConnection

Answer 2

I've written a runtime patch for requests+urllib3+socket that allows passing the required address family optionally and on a per-request basis.

Unlike other solutions there is no monkeypatching involved, rather you replace your imports of requests with the patched file and it present a request-compatible interface with all exposed classes subclassed and patched and all “simple API” function reimplemented. The only noticeable difference should be the fact that there is an extra family parameter exposed that you can use to restrict the address family used during name resolution to socket.AF_INET or socket.AF_INET6. A somewhat complicated (but mostly just LoC intensive) series of strategic method overrides is then used to pass this value all the way down to the bottom layers of urllib3 where it will be used in an alternate implementation of the socket.create_connection function call.

TL;DR usage looks like this:

import socket

from . import requests_wrapper as requests  # Use this load the patch


# This will work (if IPv6 connectivity is available) …
requests.get("http://ip6only.me/", family=socket.AF_INET6)
# … but this won't
requests.get("http://ip6only.me/", family=socket.AF_INET)

# This one will fail as well
requests.get("http://127.0.0.1/", family=socket.AF_INET6)

# This one will work if you have IPv4 available
requests.get("http://ip6.me/", family=socket.AF_INET)

# This one will work on both IPv4 and IPv6 (the default)
requests.get("http://ip6.me/", family=socket.AF_UNSPEC)

Full link to the patch library (~350 LoC): https://gitlab.com/snippets/1900824

Answer 3

This is a hack, but you can monkey-patch getaddrinfo to filter to only IPv4 addresses:

# Monkey patch to force IPv4, since FB seems to hang on IPv6
import socket
old_getaddrinfo = socket.getaddrinfo
def new_getaddrinfo(*args, **kwargs):
    responses = old_getaddrinfo(*args, **kwargs)
    return [response
            for response in responses
            if response[0] == socket.AF_INET]
socket.getaddrinfo = new_getaddrinfo

Answer 4

I took a similar approach to https://stackoverflow.com/a/33046939/5059062, but instead patched out the part in socket that makes DNS requests so it only does IPv6 or IPv4, for every request, which means this can be used in urllib just as effectively as in requests.

This might be bad if your program also uses unix pipes and other such things, so I urge caution with monkeypatching.

import requests
import socket
from unittest.mock import patch
import re

orig_getaddrinfo = socket.getaddrinfo
def getaddrinfoIPv6(host, port, family=0, type=0, proto=0, flags=0):
    return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET6, type=type, proto=proto, flags=flags)

def getaddrinfoIPv4(host, port, family=0, type=0, proto=0, flags=0):
    return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET, type=type, proto=proto, flags=flags)

with patch('socket.getaddrinfo', side_effect=getaddrinfoIPv6):
    r = requests.get('http://ip6.me')
    print('ipv6: '+re.search(r'\+3>(.*?)</',r.content.decode('utf-8')).group(1))

with patch('socket.getaddrinfo', side_effect=getaddrinfoIPv4):
    r = requests.get('http://ip6.me')
    print('ipv4: '+re.search(r'\+3>(.*?)</',r.content.decode('utf-8')).group(1))

and without requests:

import urllib.request
import socket
from unittest.mock import patch
import re

orig_getaddrinfo = socket.getaddrinfo
def getaddrinfoIPv6(host, port, family=0, type=0, proto=0, flags=0):
    return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET6, type=type, proto=proto, flags=flags)

def getaddrinfoIPv4(host, port, family=0, type=0, proto=0, flags=0):
    return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET, type=type, proto=proto, flags=flags)

with patch('socket.getaddrinfo', side_effect=getaddrinfoIPv6):
    r = urllib.request.urlopen('http://ip6.me')
    print('ipv6: '+re.search(r'\+3>(.*?)</',r.read().decode('utf-8')).group(1))

with patch('socket.getaddrinfo', side_effect=getaddrinfoIPv4):
    r = urllib.request.urlopen('http://ip6.me')
    print('ipv4: '+re.search(r'\+3>(.*?)</',r.read().decode('utf-8')).group(1))

Tested in 3.5.2

Answer 5

I've found a minimalistic solution to force urrlib3 to use either ipv4 or ipv6. This method is used by urrlib3 for creating new connection both for Http and Https. You can specify in it any AF_FAMILY you want to use.

import socket
import requests.packages.urllib3.util.connection as urllib3_cn


  def allowed_gai_family():
    """
     https://github.com/shazow/urllib3/blob/master/urllib3/util/connection.py
    """
    family = socket.AF_INET
    if urllib3_cn.HAS_IPV6:
        family = socket.AF_INET6 # force ipv6 only if it is available
    return family

urllib3_cn.allowed_gai_family = allowed_gai_family

Answer 6

After reading the previous answer, I had to modify the code to force IPv4 instead of IPv6. Notice that I used socket.AF_INET instead of socket.AF_INET6, and self.sock.connect() has 2-item tuple argument.

I also needed to override the HTTPSConnection which is much different than HTTPConnection since requests wraps the httplib.HTTPSConnection to verify the certificate if the ssl module is available.

import socket
import ssl
try:
    from http.client import HTTPConnection
except ImportError:
    from httplib import HTTPConnection
from requests.packages.urllib3.connection import VerifiedHTTPSConnection

# HTTP
class MyHTTPConnection(HTTPConnection):
    def connect(self):
        self.sock = socket.socket(socket.AF_INET)
        self.sock.connect((self.host, self.port))
        if self._tunnel_host:
            self._tunnel()

requests.packages.urllib3.connectionpool.HTTPConnection = MyHTTPConnection
requests.packages.urllib3.connectionpool.HTTPConnectionPool.ConnectionCls = MyHTTPConnection

# HTTPS
class MyHTTPSConnection(VerifiedHTTPSConnection):
    def connect(self):
        self.sock = socket.socket(socket.AF_INET)
        self.sock.connect((self.host, self.port))
        if self._tunnel_host:
            self._tunnel()
        self.sock = ssl.wrap_socket(self.sock, self.key_file, self.cert_file)

requests.packages.urllib3.connectionpool.HTTPSConnection = MyHTTPSConnection
requests.packages.urllib3.connectionpool.VerifiedHTTPSConnection = MyHTTPSConnection
requests.packages.urllib3.connectionpool.HTTPSConnectionPool.ConnectionCls = MyHTTPSConnection