How do I fetch just the beginning of a Web page with LWP?

Question

Does anyone know the best way to fetch just 50% of the Web page on a GET or POST request? The Web page I fetch takes me 10, 20 seconds to completely load, and I only need to filter just a few lines from the beginning of the page.

Answer 1

use 5.010;
use strictures;
use LWP::UserAgent qw();

my $content;
LWP::UserAgent->new->get(
    $url,
    ':content_cb' => sub {
        my ($chunk, $res) = @_;
        state $length = $res->header('Content-Length');
        $content .= $chunk;
        die if length($content) / $length > 0.5;
    },
);

Answer 2

If the web site in question suppplies the Content-Length header you can just ask how much data is going to be sent and request only half of it.

This code demonstrates.

use strict;
use warnings;

use LWP;

my $ua = LWP::UserAgent->new;
my $url = 'http://website.test';

my $resp = $ua->head($url);
my $half = $resp->header('Content-Length') / 2;

$resp = $ua->get($url, Range => "bytes=1-$half");
my $content = $resp->content;

Answer 3

If the web application needs a long time to render the page you will usually have no possibility to accellerate the process by fetching 'half' of the page.

The page will be delivered after all the database queries and the actual rendering is done. And those are likely the reason for the long delays.