delete rows ff package

Question

Since a while now I´ve been using ff package in order to work with big data. The R object I´ve worked with has about 130.000.000 rows and 14 columns. Two of those columns, Temperature and Precipitation have missing values “NA” so I need to delete those rows in order to move forward with my work. I´ve been trying to do it like I would in a normal R object:

data<-data[!is.na(data$temp),]

But I keep getting an error:

Error: vmode(index) == "integer" is not TRUE

Does anyone have been able to delete rows in a ffdf object? I´d appreciate any help.

Answer 1

Indexing based on a logical ff_vector is not possible in ff, you need to supply a vector of ff integers. That is what the error message is trying to tell you. So you can do the subsetting like this

require(ffbase)
idx <- !is.na(data$temp)
idx <- ffwhich(idx, idx == TRUE)
data <- data[idx, ]

or (using version 6.3 of ffbase)

require(ffbase)
data <- subset(data, !is.na(temp))