lwp

Lets say i have this code: use strict; use LWP qw ( get ); my $content = get ( "http://www.msn.co.il" ); print STDERR $content; The error log shows something like "\xd7\x9c\xd7\x94\xd7\x93\xd7\xa4\xd7\xa1\xd7\x94" which i'm guessing it's utf-16 ? The website's encoding is with <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=windows-1255"> so why these characters appear and not the windows-1255 chars ? And, another weird thing is that i have two servers: the first server returning CP1255 chars and i can simply convert it to utf8, and the current server gives me these chars and i can

I'm doing some web scraping using Perl's LWP. I need to process a set of URLs, some of which may redirect (1 or more times). How can I get ultimate URL with all redirects resolved, using HEAD method? If you use the fully featured version of LWP::UserAgent , then the response that is returned is an instance of HTTP::Response which in turn has as an attribute an HTTP::Request . Note that this is NOT necessarily the same HTTP::Request that you created with the original URL in your set of URLs, as described in the HTTP::Response documentation for the method to retrieve the request instance within

I am using LWP to download content from web pages, and I would like to limit the amount of time it waits for a page. This is accomplished in lwp like this: my $ua = LWP::UserAgent->new; $ua->timeout(10); $ua->get($url); And this works fine, except for whenever the timeout reaches its limit, it just dies and I can't continue on with the script! I'd really like to handle this timeout properly so that I can record that the url had a timeout and then move on to my next one. Does anyone know how to do this? Thanks! LWP::Agent 's get() returns a HTTP::Response object that you can use for checking

I have a Perl script that automatically downloads content from various sources. It does the downloading in an eval block with an alarm so that the attempt will time out if it takes too long: eval { alarm(5); my $res = $ua->request($req); $status = $res->is_success; $rawContent = $res->content; $httpCode = $res->code; alarm(0); }; This has worked for years, but after doing some system updates, all of a sudden it quit working. Instead, the first source it hits that times out, I get the following error and the program terminates: Alarm clock What am I doing incorrectly that is preventing eval

I am having trouble with a perl screenscraper to an HTTPS site. In debugging, I ran the following: print $res->headers_as_string; and in the output, I have the following line: Client-SSL-Warning: Peer certificate not verified Is there a way I can auto-accept this certificate, or is that not the problem? #!/usr/bin/perl use LWP::UserAgent; use Crypt::SSLeay::CTX; use Crypt::SSLeay::Conn; use Crypt::SSLeay::X509; use LWP::Simple qw(get); my $ua = LWP::UserAgent->new; my $req = HTTP::Request->new(GET => 'https://vzw-cat.sun4.lightsurf.net/vzwcampaignadmin/'); my $res = $ua->request($req); print

This is my first post on SO, so be gentle. I'm not even sure if this belongs here, but here goes. I want to access some information on one of my personal accounts. The website is poorly written and requires me to manually input the date I want the information for. It is truly a pain. I have been looking for an excuse to learn more Perl so I thought this would be a great opportunity. My plan was to write a Perl script that would login to my account and query the information for me. However, I got stuck pretty quickly. my $ua = LWP::UserAgent->new; my $url = url 'https://account.web.site'; my

I tried to run this perl5 program: #!/usr/bin/env perl use strict; use warnings; use LWP; my $ua = LWP::UserAgent->new('Mozilla'); $ua->credentials("test.server.com:39272", "realm-name", 'user_name', 'some_pass'); my $res = $ua->get('http://test.server.com:39272/'); print $res->content; On other hand I have HTTP::Daemon: #!/usr/bin/env perl use strict; use warnings; use HTTP::Daemon; my $hd = HTTP::Daemon->new or die; print "Contact URL: ", $hd->url, "\n"; while (my $hc = $hd->accept) { while (my $hr = $hc->get_request) { if ($hr->method eq 'GET') { print $hr->as_string, "\n"; } } $hc->close;