问题
I have a big table similar to datadf
with 3000 thousand columns and rows, I saw some methods to obtain my expected summary in stack overflow (Frequency of values per column in table), but even the fastest is very slow for my table. EDIT: thx to comments, several methods are currently satisfactory.
library(data.table)
library(tidyverse)
library(microbenchmark)
datadf <- data.frame(var1 = rep(letters[1:3], each = 4), var2 = rep(letters[1:4], each = 3), var3 = rep('m', 12), stringsAsFactors = F )
datadf <- datadf[sample(1:nrow(datadf), 1000, T),sample(1:ncol(datadf), 1000, T)]
dataDT <- as.data.table(datadf)
lev<-unique(unlist(datadf))
microbenchmark(
#base EDITED based on comment
sapply(datadf, function(x) table(factor(x, levels=lev, ordered=TRUE))), #modified based on comment
#tidyverse EDITED based on comment
datadf %>% gather() %>% count(key, value) %>% spread(key, n, fill = 0L), # based on comment
#data.table
dcast(melt(dataDT, id=1:1000, measure=1:1000)[,1001:1002][, `:=` (Count = .N), by=.(variable,value)], value ~ variable ,
value.var = "value", fun.aggregate = length),
# EDITED, In Answer below
dcast.data.table(
melt.data.table(dataDT, measure.vars = colnames(dataDT))[, .N, .(variable, value)],
value ~ variable,
value.var = "N",
fill = 0
),
times=1
)
min lq mean median uq max neva
86.8522 86.8522 86.8522 86.8522 86.8522 86.8522 1
207.6750 207.6750 207.6750 207.6750 207.6750 207.6750 1
12207.5694 12207.5694 12207.5694 12207.5694 12207.5694 12207.5694 1
46.3014 46.3014 46.3014 46.3014 46.3014 46.3014 1 # Answer
回答1:
This is about double the speed of the data.table method you provided, and should scale very well with the size of the dataset:
setDT(datadf)
dcast.data.table(
melt.data.table(datadf, measure.vars = colnames(datadf))[, .N, .(variable, value)],
value ~ variable,
value.var = "N",
fill = 0
)
I'd be interested to see the benchmarks for your full dataset, because not all of these methods will scale similarly.
来源:https://stackoverflow.com/questions/52283303/summarize-by-column-efficiently