perl - array of integers using way too much memory?

Question

When I run the following script:

my @arr = [1..5000000];

for($i=0; $i<5000000; $i++) {
        $arr[$i] = $i;
        if($i % 1000000 == 0) {
                    


        

Answer 1

Or, implicitly handling the pack for you, there is Tie::Array::PackedC:

use Tie::Array::PackedC;
# make @arr use $arr_storage for storing packed elements, by default using 'l!' pack format
tie my @arr, 'Tie::Array::PackedC', my $arr_storage;

vec may also be of interest.

Answer 2

Rather than creating an array, you may create a binary string that is exactly 5000000 * 4 characters long using pack:

my $numbers = "";
$numbers .= pack("N", $_) for (1..5000000);

It should definitely take less space. Consequently, you can fetch the values with substr and unpack.

Answer 3

All Perl values are represented internally as perl scalars, which consume way more memory than a simple int. Even if the scalar is only holding an int. Even if the scalar is undef!

As others have suggested, PDL may be something to look at if you really want to work with huge arrays of this sort.

Answer 4

Complete revision of my answer. Looking at what you have in your code, I see some strange things.

my @arr = [1..5000000];

Here, you assign an anonymous array-reference to $arr[0]. This array only holds one value: The array reference. The hidden anonymous array holds the 5 million numbers.

for($i=0; $i<5000000; $i++) {
        $arr[$i] = $i;
        if($i % 1000000 == 0) {
                print "$i\n";
        }
}

Here, you fill the array with 5 million sequential numbers, overwriting the array reference in the declaration.

A much shorter way to do it would be:

my @arr = (1 .. 5_000_000);

Perhaps that will save you some memory.

Answer 5

You can always use C or C++ in Perl.This will probably give you a small footprint in some hard jobs. Just an idea using C!

#!/usr/bin/perl
use Inline C;
use strict;

for(my $i=0; $i<5000000; $i++) {
        set_array_index($i,$i);
        if($i % 1000000 == 0) {
                #print "$i\n";
                print get_array_index($i)."\n";
        }
}

__END__
__C__

int array[5000000];

void set_array_index(int index,int value) {
    array[index]=value;
}

int get_array_index(int index) {

    if (array[index]==NULL)
        return 0;

    return array[index];
}

Answer 6

Maybe you could use an iterator instead of such a big list of integers.

The iterator pays an overhead of a function call for each new value, but saves memory. Check MJD Higher Order Perl Chapter 4 ( 4.2.1 ).

If I remember it right, range operator dont' build such a huge list in latest perls.