问题
I'm trying to scrape this webpage using Python: https://fftoolbox.scoutfantasysports.com/football/rankings/PrintVersion.php
I've been using the requests package. I can "solve" the issue by setting verify=False, however I've read that that's not secure. In other threads, people said to point the requests.get() function to the filepath of the relevant certificate. I exported the certificate from my browser, and then tried that, but with no luck. This
requests.get('https://fftoolbox.scoutfantasysports.com/football/rankings/PrintVersion.php',verify='C:/Users/ericb/Desktop/fftoolboxscoutfantasysportscom.crt')
gives the SSL error still
SSLError: HTTPSConnectionPool(host='fftoolbox.scoutfantasysports.com', port=443): Max retries exceeded with url: /football/rankings/PrintVersion.php (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),))
And this
requests.get('https://fftoolbox.scoutfantasysports.com/football/rankings/PrintVersion.php',cert='C:/Users/ericb/Desktop/fftoolboxscoutfantasysportscom.crt')
yields
Error: [('PEM routines', 'PEM_read_bio', 'no start line'), ('SSL routines', 'SSL_CTX_use_PrivateKey_file', 'PEM lib')]
I've done a decent amount of webscraping before, but I've never had to deal with certificates until now. How can I get around this? I should also note that I'd like to put my final Python script and any files it uses onto a public GitHub repo. But I don't want do do anything that would jeopardize my security, like uploading keys or something.
回答1:
The server is misconfigured, it does not send the intermediate certificate it needs to send. See this report: https://www.ssllabs.com/ssltest/analyze.html?d=fftoolbox.scoutfantasysports.com&hideResults=on
Certificates provided 1 (1776 bytes)
Chain issues Incomplete
Or https://sslanalyzer.comodoca.com/?url=fftoolbox.scoutfantasysports.com
Trusted by Microsoft? No (unable to get local issuer certificate) UNTRUSTED
Trusted by Mozilla? No (unable to get local issuer certificate) UNTRUSTED
With openssl s_client -connect fftoolbox.scoutfantasysports.com:443 -showcerts you can see:
Certificate chain
0 s:/OU=Domain Control Validated/CN=fftoolbox.scoutfantasysports.com
i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
And the webserver should be configured to send the /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2 intermediary certificate but it does not.
So, you could contact the website and tells them they are misconfigured. You will not be the only one impacted by that, as the second link shows.
Alternatively, you could add the missing certificate locally as fully trusted, but this kind of lowers your security. You can also download the missing certificate (not the one of the website, the intermediary one) locally and add verify=/path/to/certificate in your requests.get call.
来源：https://stackoverflow.com/questions/51502329/python-getting-ssl-error-when-trying-to-scrape-a-webpage