Design experiments and analysis for a simulation

[menahg]

Hi,
I am building an agent-based simulation,
There is no real data to construct the model, a data is used to generate some values helo to define agents' decision making. There is also some randomness added to the model. I need to know the best methods that allow me to design experiments and quantify uncertainty.

I should run a sensitivity analysis to fully understand the dynamics of the model since I don't have real data for validation, but I don't have a solid statistical background.

Some questions:

• How do I know the required number of repetition of the simulation?
• Most of the modellers consider sensitivity analysis to analyze model output, but I don't know what techniques in sensitivity analysis are proper?
• How can I select more representative visualizations to show the results?

Thanks and Regards,
Nada

 

[hlsmith]

I haven't run a ABM yet, but have run many Monte Carlo simulations at the group level.

-Number of repetitions: Look for convergence, so when you increase sample size the effect doesn't change much. So you just make sure that you have enough samples that the effect converges.

-Sensitivity analysis: You typically take the most important or questionable inputs and change them to see how much the effect is sensitive to the input values.

 

[menahg]

I haven't run a ABM yet, but have run many Monte Carlo simulations at the group level.

-Number of repetitions: Look for convergence, so when you increase sample size the effect doesn't change much. So you just make sure that you have enough samples that the effect converges.

-Sensitivity analysis: You typically take the most important or questionable inputs and change them to see how much the effect is sensitive to the input values.

Thanks for your response. How to know about which variables to test for the sensitivity analysis, and How to deal with the randomness you added to the model!

 

[hlsmith]

This takes context knowledge. Many times inputs may be directed by prior research results or empirical data. So if the sample sizes for those estimates and/or the estimates were not precise - those may be triggers to target those inputs. In addition, if you were simulating a policy change or intervention, it may be reasonable to adjust it's value as well.

So any pivotal variable or variable that may not be well established in the literature. You could even remove the variables with assumptions and see what that does as well. Ideally, you write out the protocol you want to execute prior to running analyses to prevent what may appear as just data mining.