1. MSE is an estimate of the population variance based on the deviation of scores around their respective treatment means. It is a weighted average of the treatment variances (see p.80)
2. MST is also an estimate of the population variance if the null hypothesis is true. It is based upon the deviations of group means about the grand mean. Since it is influenced by treatment effects, it is only an estimate of the same population variance if the treatment effects are zero; i.e., when the null hypothesis is true.
3. It turns out that if the null hypothesis is true, the ratio of these two variance estimates is distributed as an F-distribution:
F = MST / MSE
4. Since under the null hypothesis the two mean squares are estimating the same population value, this ratio should be close to 1 when the null is true. The observed value of F is compared to the sampling distribution of F to get a p-value (or empirical p-value via permutation test) to look for departures from the null hypothesis.
5. If the observed F ratio is "large", then perhaps the assumption of the null hypothesis of no treatment effect is false, and we should reject the null.