I\'m trying to scrape some pages that are on a website but to view the pages, I need to be connected to a VPN. My setup is as follows:
I see that https://www.privateinternetaccess.com/ has option to use SOCKS5 proxy. If you are using requests module for scraping you may use SOCKS5 like that:
pip install -U requests[socks]
and in the script:
import requests
proxies = {'http': 'socks5://user:pass@host:port',
'https': 'socks5://user:pass@host:port'}
resp = requests.get('http://example.com', proxies=proxies )