Verifying HTTPS certificates with urllib.request

Question

I am trying to open an https URL using the urlopen method in Python 3\'s urllib.request module. It seems to work fine, but the documentation warns that \"[i]f neither

Answer 1

I found a library that does what I'm trying to do: Certifi. It can be installed by running pip install certifi from the command line.

Making requests and verifying them is now easy:

import certifi
import urllib.request

urllib.request.urlopen("https://example.com/", cafile=certifi.where())

As I expected, this returns a HTTPResponse object for a site with a valid certificate and raises a ssl.CertificateError exception for a site with an invalid certificate.

Answer 2

Different Linux distributives have different pack names. I tested in Centos and Ubuntu. These certificate bundles are updates with system update. So you may just detect which bundle is available and use it with urlopen.

cafile = None
for i in [
    '/etc/ssl/certs/ca-bundle.crt',
    '/etc/ssl/certs/ca-certificates.crt',
]:
    if os.path.exists(i):
        cafile = i
        break
if cafile is None:
    raise RuntimeError('System CA-certificates bundle not found')

Answer 3

Works in python 2.7 and above

context = ssl.create_default_context(cafile=certifi.where())
req = urllib2.urlopen(urllib2.Request(url, body, headers), context=context)

Answer 4

import certifi
import ssl
import urllib.request
try:
    from urllib.request import HTTPSHandler
    context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
    context.options |= ssl.OP_NO_SSLv2
    context.verify_mode = ssl.CERT_REQUIRED
    context.load_verify_locations(certifi.where(), None)
    https_handler = HTTPSHandler(context=context,  check_hostname=True)
    opener = urllib.request.build_opener(https_handler)
except ImportError:
    opener = urllib.request.build_opener()

opener.addheaders = [('User-agent',  YOUR_USER_AGENT)]
urllib.request.install_opener(opener)

Answer 5

You can download the certificates Mozilla in a format usable for urllib (e.g. PEM format) at http://curl.haxx.se/docs/caextract.html

Answer 6

Elias Zamarias answer still works, but gives a deprecation warning:

DeprecationWarning: cafile, cpath and cadefault are deprecated, use a custom context instead.

I was able to solve the same problem this way instead (using Python 3.7.0):

import ssl
import urllib.request

ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
response = urllib.request.urlopen("http://www.example.com", context=ssl_context)