perl - array of integers using way too much memory?

Question

When I run the following script:

my @arr = [1..5000000];

for($i=0; $i<5000000; $i++) {
        $arr[$i] = $i;
        if($i % 1000000 == 0) {
                    


        

Answer 1

Here are a few lines from an interactive pdl2 session showing how this could be done using basic PDL constructs:

pdl> $arr = sequence(long, 5000000) + 1;  # create pdl data array (a.k.a. a piddle)

pdl> help vars                            # see, it is only ~19MB
PDL variables in package main::

Name         Type   Dimension       Flow  State          Mem
----------------------------------------------------------------
$arr           Long D [5000000]            P           19.07MB
$Pi          Double D []                   P            0.01KB

pdl> p which( $arr%1000000 == 0 )         # which returns indexes which are true
[999999 1999999 2999999 3999999 4999999]

See the on-line PDL book for a good introduction to what PDL can be used for. The PDL mailing lists are the best source of information on PDL usage and development. The responses are often rapid.

Answer 2

If you are dealing with such large arrays, you might want to use a toolkit like the PDL.

(Oh, and yes, you are correct: It takes so much memory because it is an array of Perl scalars.)