The Experiment-wise Error Rate Quickly Becomes Problematic! What’s the error rate for six comparisons? Unfortunately, as you’ll see next, the experiment-wise error rate snowballs based on the number of groups in your experiment. Each comparison represents a roll of the dice for obtaining a false positive. Our experiment includes this family of six comparisons. For our example ANOVA with four groups (A B C D), we’ll need to make the following six comparisons. The more groups you have, the more comparison tests you need to perform. In the ANOVA context, you want to compare the group means. Related posts: Types of Errors in Hypothesis Testing and Significance Levels and P-values Family Error Rates in ANOVA With enough rolls, it becomes inevitable.

With 25 rolls, snake eyes become more likely than not (50.8%). However, the more times you roll the dice, the more likely you’ll get two ones. If you roll the dice just once, your chances of rolling snake eyes aren’t too bad. The probability of snake eyes for a single roll is ~2.8% rather than 5%, but you get the idea. Imagine you’re rolling a pair of dice and rolling two ones (known as snake eyes) represents a Type I error. If you perform enough tests, you’re virtually guaranteed to get a false positive! The error rate for a family of tests is always higher than an individual test. However, as you conduct more and more tests, your chance of a false positive increases. When you perform only one test, the type I error rate equals your significance level, which is often 5%. In other words, there’s a chance that you’ll reject a null hypothesis that is actually true-it’s a false positive. What is this experiment-wise error rate? For every hypothesis test you perform, there is a type I error rate, which your significance level (alpha) defines.

Jmp two way anova