lwp

I have a lot of scripts, most of them based around WWW::Mechanize that scrape data off of misc hardware that is accessible via HTTPs. After upgrading most of my perl installation and its modules, all scripts using HTTPS:// broke because of "certificate verify failed" This is a result of the fact that the newer versions of LWP does a proper check on the certificate and dies if something doesn't match. In my case, the failed certificate authentication is expected due to the circumstances, so i needed to find a way of cleanly circumventing this check. Say I want to tell you something, and I don't

If you try to login at https://orbit.theplanet.com/Login.aspx?url=/Default.aspx (use any username/password combination), you can see that the login credentials are sent as a non-traditional set of POST data: just a lonesome JSON string and no normal key=value pair. Specifically, instead of: username=foo&password=bar or even something like: json={"username":"foo","password":"bar"} There's simply: {"username":"foo","password":"bar"} Is it possible to perform such a request with LWP or an alternative module? I am prepared to do so with IO::Socket but would prefer something more high-level if

My symptom is that I cannot use a proxy with HTTPS requests with LWP. This seems to be a common problem, and the hints on Google and even here all suggest a work-around for setting the HTTPS_PROXY environment variable for use by Crypt::SSLeay. My specific problem appears to be that LWP::Protocol::https is loading IO::Socket::SSL rather than Crypt::SSLeay. How can I force Crypt::SSLeay's use instead? My code: #!/usr/bin/perl use strict; use warnings; $ENV{HTTPS_PROXY} = 'http://10.0.3.1:3128'; use LWP::UserAgent; my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new('GET','https://www

So I am using WWW::Mechanize to crawl sites. It works great, except if I request a url such as: http://www.levi.com/ I am redirected to: http://us.levi.com/home/index.jsp And for my script I need to know that this redirect took place and what the url I was redirected to is. Is there anyway to detect this with WWW::Mechanize or LWP and then get the redirected url? Thanks! use strict; use warnings; use URI; use WWW::Mechanize; my $url = 'http://...'; my $mech = WWW::Mechanize->new(autocheck => 0); $mech->max_redirect(0); $mech->get($url); my $status = $mech->status(); if (($status >= 300) && (

I am fetching some pages over the Web using Perl's LWP::UserAgent and would like to be as polite as possible. By default, LWP::UserAgent does not seamlessly handle compressed content via gzip. Is there an easy way to make it do so, to save everyone some bandwidth? LWP has this capability built in, thanks to HTTP::Message . But it's a bit hidden. First make sure you have Compress::Zlib installed so you can handle gzip . HTTP::Message::decodable() will output a list of allowed encodings based on the modules you have installed; in scalar context, this output takes the form a comma-delineated

How can I get LWP to verify that the certificate of the server I'm connecting to is signed by a trusted authority and issued to the correct host? As far as I can tell, it doesn't even check that the certificate claims to be for the hostname I'm connecting to. That seems like a major security hole (especially with the recent DNS vulnerabilities). Update: It turns out what I really wanted was HTTPS_CA_DIR , because I don't have a ca-bundle.crt. But HTTPS_CA_DIR=/usr/share/ca-certificates/ did the trick. I'm marking the answer as accepted anyway, because it was close enough. Update 2: It turns