Perl - Breaking a variable from the input into two for the url

问题

I'm loading data from a .txt for the purposes of scraping. However, the URL requires that I break that variable up and do +/- 2 to it. For example, if the value is 2342, I need to create 2340 and 2344 for the purposes of the URL.

I took a guess at how to break it up:

 $args{birth_year} = ($args{birth_year} - 2) . '-' . ($args{birth_year} + 2);

How do I then put it in the URL?

Here's the relevant part of the code:

  use strict;
  use warnings;
  use WWW::Mechanize::Firefox;
  use Data::Dumper;
  use LWP::UserAgent;
   use JSON;
  use CGI qw/escape/;
  use HTML::DOM;

  open(my $l, 'locations2.txt') or die "Can't open locations: $!";

 while (my $line = <$l>) {
    chomp $line;
     my %args;
     @args{qw/givenname surname birth_place birth_year gender race/} = split /,/, $line;
     $args{birth_year} = ($args{birth_year} - 2) . '-' . ($args{birth_year} + 2);
      my $mech = WWW::Mechanize::Firefox->new(create => 1, activate => 1);
     $mech->get("https://familysearch.org/search/collection/index#count=20&query=%2Bgivenname%3A$args{givenname}20%2Bsurname%3A$args{surname}20%2Bbirth_place%3A$args{birth_place}%20%2Bbirth_year%3A1910-1914~%20%2Bgender%3A$args{gender}20%2Brace%3A$args{race}&collection_id=2000219");

For Example

Input is:

Benjamin,Schuvlein,Germany,1912,M,White

Desired URL is:

https://familysearch.org/search/collection/index#count=20&query=%2Bgivenname%3ABenjamin%20%2Bsurname%3ASchuvlein%20%2Bbirth_place%3AGermany%20%2Bbirth_year%3A1910-1914~%20%2Bgender%3AM%20%2Brace%3AWhite&collection_id=2000219

回答1:

Why can't you just change this line:

$mech->get("https://familysearch.org/search/collection/index#count=20&query=%2Bgivenname%3A$args{givenname}20%2Bsurname%3A$args{surname}20%2Bbirth_place%3A$args{birth_place}%20%2Bbirth_year%3A1910-1914~%20%2Bgender%3A$args{gender}20%2Brace%3A$args{race}&collection_id=2000219");

to this:

$mech->get("https://familysearch.org/search/collection/index#count=20&query=%2Bgivenname%3A$args{givenname}20%2Bsurname%3A$args{surname}20%2Bbirth_place%3A$args{birth_place}%20%2Bbirth_year%3A$args(birth_year)~%20%2Bgender%3A$args{gender}20%2Brace%3A$args{race}&collection_id=2000219");

NOTE: I changed this bit:

%3A1910-1914~%20

to this:

%3A$arg(birth_year)~%20

回答2:

One way to do it:

file content:
link1
link2
...
linkn

use Data::Dumper;
use strict;
use warnings;

local $/=undef;
open(FILE,'<',$filename) or die $filename;
my $i = 1;
while (my $line = <FILE>){
  chomp($line);
  print "line: $line\n";
  my $tempfile = './$i.html';$i++;
  $mech->get( $line, ':content_file' => $tempfile, synchronize => 1 );
}

回答3:

This answer doesn't consider whether the data in the input needs to be URL-encoded, i.e. somewhere along the way if a surname is "von Schtupp" it needs to become "von%20Schtupp"

I didn't test this, so there may be a typo or minor error. Nevertheless, it's the approach that I would use. My answer also assumes that you don't care what order in which the search criteria appear.

my %query_params = (
    givenname => $args{givenname},
    surname   => $args{surname},
    birth_place => $args{birth_place},
    birth_year => sprintf("%d-%d", $args{birth_year} - 2, $args{birth_year} + 2),
    gender     => $args{gender},
    race       => $args{race},
);
my $query_parameter = join '%20',
                      map { "%2B$_%3A$query_params{$_}" }
                      keys %query_params;
my $url = "https//familysearch.org/search/collection/index#count=20&query=" .
          $query_parameter . "&collection_id=2000219";

来源：https://stackoverflow.com/questions/14793092/perl-breaking-a-variable-from-the-input-into-two-for-the-url

标签

ajax

json

perl

web-scraping

mozrepl