Can I supply a URL to lxml.etree.parse on Python 3?

Question

The documentation says I can:

lxml can parse from a local file, an HTTP URL or an FTP URL. It also auto-detects and reads gzip-compressed XML files

Answer 1

The issue is that lxml does not support HTTPS urls, and http://pypi.python.org/simple redirects to a HTTPS version.

So for any secure website, you need to read the URL yourself:

from lxml import etree
from urllib.request import urlopen

parser = etree.HTMLParser()

with urlopen('https://pypi.python.org/simple') as f:
    tree = etree.parse(f, parser)