“Too many indices” big matrix vector length issue in R

问题

Hello and thanks in advance. I'm using Rx64 version 3.1.2 on a Windows Server and have a file-backed big matrix generated from the package bigmemory that I'm trying to use in a linear programming problem. The matrix is 7062 rows by 364520 columns for a total of 2574240240 entries (of integers).

When I run the line for the linear program, I get the following error:

Error in GetElements.bm(x, i, j) : 
  Too many indices (>2^31-1) for extraction.

That number, 2147483647, from what I read is the maximum number of entries R allows for any object even on 64-bit processes. I've read here that I can use the 'experimental version' of R to get around this but I was hoping there's a recently available solution. I have supporting output below to confirm my R version:

> version
               _                           
platform       x86_64-w64-mingw32          
arch           x86_64                      
os             mingw32                     
system         x86_64, mingw32             
status                                     
major          3                           
minor          1.2                         
year           2014                        
month          10                          
day            31                          
svn rev        66913                       
language       R                           
version.string R version 3.1.2 (2014-10-31)
nickname       Pumpkin Helmet 

> Sys.getenv("R_ARCH")
[1] "/x64"

回答1:

It would be best if you could also provide the 'line for the linear program' but my comment is too long to put above.

@James is correct that 32-bit is hard-coded in to bigmemory at the moment. After glancing at the source on github I see the author did remove the constraint and you can download that newest version with:

library(devtools)
install_github("kaneplusplus/bigmemory")

Without that limitation I believe you will still have a problem. If I assume correctly, you are also using the complementary package bigalgebra. There is a way to use 64-bit with bigalgebra by reintstalling the downloaded tar ball and installing with:

REFBLAS=1 R CMD INSTALL bigalgebra_0.8.4.tar.gz

NOTE - the functions available for 64-bit are very limited currently and will be slower than the non-64-bit versions because of the currently workarounds.

I have been in contact with the original author and I am in the process of updating the packages to utilize RcppArmadillo which will provide a cleaner interface and deal with the 64-bit problem. This is currently in progress at my fork of the package on my github account.

回答2:

The best you can do is basically divide and conquer, meaning splitting your matrix and do as many steps (more) as necessary to reach your goal.

来源：https://stackoverflow.com/questions/28836410/too-many-indices-big-matrix-vector-length-issue-in-r