draw a huge sample (1e09) from a multinomial distribution with sample

问题

I would like to sample from a multinomial distribution. I would do this by using sample and specifying some probabilites. E.g: I have 3 categories, and I want to sample 10 times.

> my_prob = c(0.2, 0.3, 0.5)
> x = sample(c(0:2), 100, replace = T, prob = my_prob)
> head(x)
[1] 2 0 2 1 1 2

My setting is now only different in the following aspect: I want to sample a lot (e.g. 1e09) numbers. And actually I am only interested in the frequency of each category. So in the above mentioned example this would mean:

> table(x)
x
 0  1  2 
27 29 44

Does anybody have an idea how to compute this as efficient as possible?

thanks, steffi

回答1:

You need rmultinom.

my_prob <- c(0.2,0.3,0.5)
number_of_experiments <- 10
number_of_samples <- 100
experiments <- rmultinom(n=number_of_experiments, size=number_of_samples, prob=my_prob)
experiments

     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
     [1,]   14   18   15   19   14   17   23   18   24    15
     [2,]   33   34   36   30   40   30   27   38   24    30
     [3,]   53   48   49   51   46   53   50   44   52    55

回答2:

If the problem is that you can't fit a vector of length 1e9 into RAM, then you can repeatedly calculate the table for a smaller number of samples and add up the totals.

n_total <- 1e9
n_chunk <- 1e6
n_iter <- n_total / n_chunk
my_prob = c(0.2, 0.3, 0.5)
totals <- numeric(3)
for(i in seq_len(n_iter))
{
  totals <- totals + table(sample(0:2, n_chunk, replace = TRUE, prob = my_prob))
}
totals
stopifnot(sum(totals) == n_total)

Like Max said, you might prefer rmultinom over sample. Take the rowSums of his experiments variable.

来源：https://stackoverflow.com/questions/7915668/draw-a-huge-sample-1e09-from-a-multinomial-distribution-with-sample

标签

random-sample