How many simulations need to do?

Question

Hello my problem is more related with the validation of a model. I have done a program in netlogo that i\'m gonna use in a report for my thesis but now the question is, how

Answer 1

There are two aspects to this (1) How many parameter combinations (2) How many runs for each parameter combination.

(1) Generally you would do experiments, where you vary some of your input parameter values and see how some model output changes. Take the well known Schelling segregation model as an example, you would vary the tolerance value and see how the segregation index is affected. In this case you might vary the tolerance from 0 to 1 by 0.01 (if you want discrete) or you could just take 100 different random values in the range [0,1]. This is a matter of experimental design and is entirely affected by how fine you wish to examine your parameter space.

(2) For each experimental value, you also need to run multiple simulations so that you can can calculate the average and reduce the impact of randomness in the simulation run. For example, say you ran the model with a value of 3 for your input parameter (whatever it means) and got a result of 125. How do you know whether the 'real' answer is 125 or something else. If you ran it 10 times and got 10 different numbers in the range 124.8 to 125.2 then 125 is not an unreasonable estimate. If you ran it 10 times and got numbers ranging from 50 to 500, then 125 is not a useful result to report.

The number of runs for each experiment set depends on the variability of the output and your tolerance. Even the 124.8 to 125.2 is not useful if you want to be able to estimate to 1 decimal place. Look up 'standard error of the mean' in any statistics text book. Basically, if you do N runs, then a 95% confidence interval for the result is the average of the results for your N runs plus/minus 1.96 x standard deviation of the results / sqrt(N). If you want a narrower confidence interval, you need more runs.

The other thing to consider is that if you are looking for a relationship over the parameter space, then you need fewer runs at each point than if you are trying to do a point estimate of the result.

Answer 2

1) The number of simulations for each parameter setting can be decided by studying the coefficient of variance Cv = s / u, here s and u are standard deviation and mean of the result respectively. It is explained in detail in this paper Coefficient of variance.

2) The simulations where parameters are changed can be analyzed using several methods illustrated in the paper Testing methods.

These papers provide scrupulous analyzing methods and refer to other papers which may be relevant to your question and your research.

Answer 3

Not sure exactly what you mean, but maybe you can check the books of Hastie and Tishbiani

http://web.stanford.edu/~hastie/local.ftp/Springer/OLD/ESLII_print4.pdf

specially the sections on resampling methods (Cross-Validation and bootstrap).

They also have a shorter book that covers the possible relevant methods to your case along with the commands in R to run this. However, this book, as a far as a I know, is not free.

http://www.springer.com/statistics/statistical+theory+and+methods/book/978-1-4614-7137-0

Also, could perturb the initial conditions to see you the outcome doesn't change after small perturbations of the initial conditions or parameters. On a larger scale, sometimes you can break down the space of parameters with regard to final state of the system.