Example of bigmemory and friends with file backing

六眼飞鱼酱① 提交于 2019-12-19 08:01:18

问题


I am interested in exploring how R can handle data out-of-memory. I've found the bigmemory package and friends (bigtabulate and biganalytics), but was hoping that someone could point me to a worked out example that uses file backing with these packages. Any other out-of-memory tips would also be appreciated.


回答1:


Charlie, just email Mike and Jay, they have a number of examples working around the ASA 'flights' database example from a year or two ago.

Edit: In fact, the Documentation tab has what I had in mind; the scripts are also on the site.




回答2:


Take a look at "CRAN Task View: High-Performance and Parallel Computing with R". There is a chapter "Large memory and out-of-memory data" where severel solutions are mentioned. For example package ff.




回答3:


Any other out-of-memory tips would also be appreciated.

I frequently work with large datasets. Even though my code has been optimized, I still launch Amazon EC2 instances from time to time because it gives me access to far more resources than on my desk. For example, an instance with 26 ECUs, 8 cores, and 68 gigs of RAM only costs about a $0.80-1.00 per hour (spot instance pricing).

If that seems reasonable, you can launch a public machine image that already has R and do this job in no time.



来源:https://stackoverflow.com/questions/4997612/example-of-bigmemory-and-friends-with-file-backing

标签
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!