Read lines by number from a large file

狂风中的少年 提交于 2019-11-28 17:16:24

The trick is to use connection AND open it before read.table:

con<-file('filename')
open(con)

read.table(con,skip=5,nrow=1) #6-th line
read.table(con,skip=20,nrow=1) #27-th line
...
close(con)

You may also try scan, it is faster and gives more control.

Ari B. Friedman

If it's a binary file

Some discussion is here: Reading in only part of a Stata .DTA file in R

If it's a CSV or other text file

If they are contiguous and at the top of the file, just use the ,nrows argument to read.csv or any of the read.table family. If not, you can combine the ,nrows and the ,skip arguments to repeatedly call read.csv (reading in a new row or group of contiguous rows with each call) and then rbind the results together.

If your file has fixed line lengths then you can use 'seek' to jump to any character position. So just jump to N * line_length for each N you want, and read one line.

However, from the R docs:

 Use of seek on Windows is discouraged.  We have found so many
 errors in the Windows implementation of file positioning that
 users are advised to use it only at their own risk, and asked not
 to waste the R developers' time with bug reports on Windows'
 deficiencies.

You can also use 'seek' from the standard C library in C, but I don't know if the above warning also applies!

Before I was able to get an R solution/answer, I've done it in Ruby:

#!/usr/bin/env ruby

NUM_SEQS = 14024829

linenumbers = (1..10).collect{(rand * NUM_SEQS).to_i}

File.open("./data/uniprot_2011_02.tab") do |f|
  while line = f.gets
    print line if linenumbers.include? f.lineno 
  end
end

runs fast (as fast as my storage can read the file).

I compile a solution based on the discussions here.

scan(filename,what=list(NULL),sep='\n',blank.lines.skip = F)

This will only show you number of lines but will read in nothing. If you really want to skip the blank lines, you could just set the last argument to TRUE.

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!