Fetching a .onion domain with requests

问题

I'm trying to access the following domain nzxj65x32vh2fkhk.onion using requests.
I have tor running and I configured the session's object proxies correctly.

import requests
session = requests.session()
session.proxies = {'http':  'socks5://localhost:9050',
                   'https': 'socks5://localhost:9050'}
print(session.get('http://httpbin.org/ip').text) # prints {"origin": "67.205.146.164" }

print(requests.get('http://httpbin.org/ip').text) # prints {"origin": "5.102.254.76" }

However when I try to access the URL with the .onion domain I get the following error:

session.get('http://nzxj65x32vh2fkhk.onion/all')

ConnectionError: SOCKSHTTPConnectionPool(host='nzxj65x32vh2fkhk.onion', port=80): Max retries exceeded with url: /all (Caused by NewConnectionError('<requests.packages.urllib3.contrib.socks.SOCKSConnection object at 0x7f5e8c2dbbd0>: Failed to establish a new connection: [Errno -2] Name or service not known',))

I also tried to replace localhost with 127.0.0.1 as suggested in one of the answers. The result is the same unfortunately.

Performing the same request using urllib2 works just fine.

import socks, socket, urllib2

def create_connection(address, timeout=None, source_address=None):
  sock = socks.socksocket()
  sock.connect(address)
  return sock

socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', 9050)
socket.socket = socks.socksocket
socket.create_connection = create_connection

print(urllib2.urlopen('http://nzxj65x32vh2fkhk.onion/all').read()) # Prints the URL's contents

cURL also retrieves the contents of the page correctly.

I'm using Python 2.7.13, requests 2.13.0 & PySocks 1.6.7. Tor is running through a docker container with the following command:

sudo docker run -it -p 8118:8118 -p 9050:9050 -d dperson/torproxy

What am I doing wrong here? What do I need to do to make requests recognize the .onion URLs?

回答1:

The solution is to use the socks5h protocol in order to enable remote DNS resolving in case the local DNS resolving process fails. See https://github.com/kennethreitz/requests/blob/e3f89bf23c53b98593e4248054661472aacac820/requests/packages/urllib3/contrib/socks.py#L158

The following code works as expected:

import requests
session = requests.session()
session.proxies = {'http':  'socks5h://localhost:9050',
                   'https': 'socks5h://localhost:9050'}
print(session.get('http://httpbin.org/ip').text) # prints {"origin": "67.205.146.164" }

print(requests.get('http://httpbin.org/ip').text) # prints {"origin": "5.102.254.76" }

print(session.get('http://nzxj65x32vh2fkhk.onion/all').text) # Prints the contents of the page

来源：https://stackoverflow.com/questions/42971622/fetching-a-onion-domain-with-requests

标签

python

python-requests

urllib2

socks