General SOCKS server failure with python tor but working from tor browser

问题

I would like to use tor from python to automate requests. I made a test with a page to check IP and it works.

I then pointed to the site I want to, and apparently they avoid a tor endpoint because (see stack trace below) - but it works from tor browser.

Any better way to debug response of browser? (e.g. connection refused)

Which things I am missing to query from python instead than from browser?

I am trying with something like:

socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5 , "127.0.0.1", 9150, True)
socket.socket = socks.socksocket

socks.wrapmodule(requests)
url = "myexample.com"
# r = requests.Session()
s = requests.get(url)
print s


    Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1181, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1172, in connect
    self.timeout, self.source_address)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 562, in create_connection
    sock.connect(sa)
  File "/Library/Python/2.7/site-packages/socks.py", line 747, in connect
    negotiate(self, dest_addr, dest_port)
  File "/Library/Python/2.7/site-packages/socks.py", line 419, in _negotiate_SOCKS5
    CONNECT, dest_addr)
  File "/Library/Python/2.7/site-packages/socks.py", line 494, in _SOCKS5_request
    raise SOCKS5Error("{0:#04x}: {1}".format(status, error))
socks.SOCKS5Error: 0x01: General SOCKS server failure

回答1:

Even though you use tor some website are looking for User Agent. Try to had User Agent header to your request. I had the same problem.

This worked for me

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
import random
from fake_useragent import UserAgent
from torrequest import TorRequest
import time, socks, socket
from stem import Signal
from stem.control import Controller


ua = UserAgent()


with Controller.from_port(port = 9051) as controller:
    controller.authenticate(password = 'YourPasswordHere')
    socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
    socket.socket = socks.socksocket   

    controller.signal(Signal.NEWNYM)
    if controller.is_newnym_available() == False:
        print("Waitting time for Tor to change IP: "+ str(controller.get_newnym_wait()) +" seconds")
        time.sleep(controller.get_newnym_wait())
    req = Request(url)
    req.add_header('User-Agent', ua.random)
    req_doc= urlopen(req)#.read().decode('utf8')
    print(req_doc)

来源：https://stackoverflow.com/questions/41992581/general-socks-server-failure-with-python-tor-but-working-from-tor-browser