Scripts broke after upgrading LWP “certificate verify failed”

Question

I have a lot of scripts, most of them based around WWW::Mechanize that scrape data off of misc hardware that is accessible via HTTPs. After upgrading most of my perl installation and its modules, all scripts using HTTPS:// broke because of "certificate verify failed"

This is a result of the fact that the newer versions of LWP does a proper check on the certificate and dies if something doesn't match.

In my case, the failed certificate authentication is expected due to the circumstances, so i needed to find a way of cleanly circumventing this check.

Answer 1

Say I want to tell you something, and I don't want anyone else to know it. We'd arrange a password, and I'd use it to encrypt the message, then I'd send you the message.

What if I didn't make sure the person to whom I gave the password and encrypted message was you? Then any number of people could simply impersonate you and the encryption would be for naught. That was the state of LWP's HTTPS support until recently.

Now, LWP actually checks to whom it's talking unless you ask LWP to behave as it once did. You can do that using:

my $ua = LWP::UserAgent->new(
   ssl_opts => { verify_hostname => 0 },
);

If you want to affect all LWP::UserAgent instances in your script without specifying the option all over the place, you can add the following to your script

$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;

Or you can launch your script as follows:

PERL_LWP_SSL_VERIFY_HOSTNAME=0 script.pl

Finally, if you want LWP to always be unsafe, you can add the following to your login script:

export PERL_LWP_SSL_VERIFY_HOSTNAME=0

However, I recommend none of the above. The far better option would be to provide the certificate for the host to which you are communicating. (This is the equivalent of adding an exception in Firefox, if you know what I mean.) See the documentation for $ua->ssl_opts.

Answer 2

For me, using:

my $ua = LWP::UserAgent->new(
  ssl_opts => { verify_hostname => 0 },
);

Yielded

Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER together with SSL_ca_file|SSL_ca_path for verification. If you really don't want to verify the certificate and keep the connection open to Man-In-The-Middle attacks please set SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.

Using this did not give any warnings:

my $ua = LWP::UserAgent->new(
  ssl_opts => { SSL_verify_mode => 'SSL_VERIFY_NONE'},
);

Answer 3

I prepended my code with:

$ENV{'PERL_LWP_SSL_VERIFY_HOSTNAME'} = 0;

This caused the script to circumvent the check in a clean and simple way.

Answer 4

@ikegami makes a good argument for why you don't want to disable SSL hostname verification, but doesn't directly mention how to avoid it.

If you're talking to a public system with a CA-signed certificate, you need to point LWP to your distribution's root certificate collection. Under a Debian-based system (Ubuntu, etc.), this is kept under /etc/ssl/certs/.

BEGIN {
    $ENV{HTTPS_CA_DIR} = '/etc/ssl/certs'
}

If you are talking to your own server with a self-signed certificate, you can save a copy of that certificate on the client, and point your script to that particular file.

BEGIN {
    $ENV{HTTPS_CA_FILE} = '/path/to/my/server-certificate.crt'
}

You could instead set these in the environment before running your script (e.g. export them from your shell), or you could apply the settings directly to your UserAgent object. See the LWP::UserAgent documentation for more details; search for ssl_opts (around halfway down the page).