Question about using statistics (t-test or Chi square?) for computer gaming

Hi I'm not a student of statistics but I have tried to learn a little of statistics myself. I have used an online chi square calculator for some work and I think I know when to use a t-test (but then there's multiple types IIRC...). So please forgive my limited stats knowledge (I do think it's interesting and useful though!).

So I play a mod for Civ 4 (a turn based strategy computer game) and I got thinking that you can use statistics to check if different civs (when controlled by AI alone) are balanced in overall power even though they are different in their abilities. You can use a 'cheat' to make only AI play (which removes the human influence on the results). My main problem is determining the appropriate sample size (number of games to be run) but I would also appreciate it if you could confirm that I have chosen the correct statistical test.

Rules of the game:
Only one civilisation can win but possibly noone may achieve a victory condition and often there is a time victory (the person who has the most points is the winner). However if you run the game long enough there probably will be a clear winner (perhaps run it until 3000 years instead of 500). The game will end before the time runs out if someone achieves a victory condition. There are a few different types of victory conditions.

I can make it so that only AI will compete in a game.

Scores don't always represent who wins in the end but they give a fairly good estimate of who is winning and likely to win overall (therefore I am thinking Chi square analysis is appropriate using the overall winner from each game and totaling the number of wins over lots of games). EDIT: I mentioned t-test because I think it could show if there is a statistically significant difference between scores of different civilizations.

Maybe there are too many civs (most have 2 leaders who are a bit different in their effects on the civilisation they lead) for one analysis? If so I could just do an analysis on half a dozen of them or so - even that would be very useful and interesting!

Aim of the statistical analysis
To determine if one AI civilisation (or rather leader) wins significantly more than other AI civilisations.

My Questions
How can I determine the appropriate sample size (number of games to run)?

What number of different civilisations could I use without it making the sample size equal a billion ;) or there-abouts?

Am I right in wanting to use Chi square analysis to determine the answer to my question?

Thanks for this forum!:)
Last edited: