SSDF often uses the term “margin of error”. What factors influence the size of this margin?
More than anything else, the number of games played decides the margin of error (confidence range, see below). Once upon a time, we thought that 40 games between two computers was a lot. Nowadays, we know more about statistics. After so few games, you can almost never say for sure which computer is better. Of course, in most situations your result after 40 games looks similar to what you will see after 1,000 games. But it happens often enough, that the picture is different. Even if the two are of equal strength, you may well get a result of 28-12 in the first series of games and 12-28 in the next. The margin of error also depends on the relative strength of the two computers. A big difference in strength results in a larger margin of error. From a statistical point of view, the optimal solution is thus to play a large number of games against opponents of similar strength.
