Family-Wise Error Rate

A Family-Wise Error Rate is an error rate measure that quantifies the probability of making at least one Type I error when conducting multiple statistical hypothesis tests simultaneously in a multiple testing problem.

• AKA: FWER, Familywise Error Rate, Experiment-Wise Error Rate, Multiple Comparison Error Rate, Family-Wise Type I Error Rate.
• Context:
  • It can typically equal 1 - (1 - α)^n for n independent tests at individual significance level α.
  • It can typically increase rapidly with the number of hypothesis tests unless controlled through multiple testing correction.
  • It can typically be controlled at a desired level using methods like Bonferroni correction, Holm-Bonferroni method, or Šidák correction.
  • It can typically be more conservative than false discovery rate control, providing stronger protection against any false positives.
  • It can often be preferred in confirmatory studies where any Type I error would invalidate conclusions.
  • It can often conflict with statistical power when strictly controlled, creating a trade-off between error types.
  • It can often be calculated exactly for independent tests but requires approximation for dependent tests.
  • It can often determine the choice of post-hoc test following ANOVA or other omnibus tests.
  • It can range from being a Weak FWER Control to being a Strong FWER Control, depending on its null hypothesis configuration.
  • It can range from being a Marginal FWER to being a Joint FWER, depending on its test dependency structure.
  • It can range from being a Per-Family FWER to being a Per-Comparison FWER, depending on its error unit definition.
  • It can range from being an Asymptotic FWER to being a Finite-Sample FWER, depending on its sample size assumption.
  • It can range from being a Conservative FWER Control to being a Exact FWER Control, depending on its correction method.
  • ...
• Example(s):
  • Clinical Trial FWER Controls, such as:
    • FWER ≤ 0.05 for primary endpoints in Phase III trials.
    • FWER control for dose-response studies with multiple doses.
    • FWER adjustment in interim analysis with multiple looks.
  • Genomic Study FWER Controls, such as:
    • FWER for candidate gene studies testing specific hypotheses.
    • FWER in linkage analysis with multiple markers.
    • FWER for pathway analysis with predefined gene sets.
  • Post-Hoc Testing FWER Controls, such as:
    • Tukey's HSD controlling FWER for all pairwise comparisons.
    • Dunnett's Test controlling FWER for comparisons to control.
    • Scheffé's Method controlling FWER for all possible contrasts.
  • Multiple Endpoint FWER Controls, such as:
    • FWER for co-primary endpoints requiring all significant.
    • FWER in hierarchical testing procedures with ordered hypotheses.
    • FWER for gatekeeping procedures with logical relationships.
  • ...
• Counter-Example(s):
  • False Discovery Rate, which controls expected proportion of false positives among discoveries.
  • Per-Comparison Error Rate, which ignores multiple testing issues.
  • False Discovery Proportion, which is a random variable rather than expected value.
  • Type II Error Rate, which concerns false negatives rather than false positives.
  • Local False Discovery Rate, which focuses on individual hypotheses.
• See: Multiple Testing Problem, Multiple Comparisons Inference Task, Type I Error, Type II Error, Bonferroni Correction, Holm-Bonferroni Method, Šidák Correction, False Discovery Rate, Multiple Hypothesis Testing Framework, Post-Hoc Analysis, Tukey's Range Test, Fisher's Least Significant Difference Test, Statistical Power.

---

---

References

2016

• (Wikipedia, 2016) ⇒ https://en.wikipedia.org/wiki/Family-wise_error_rate Retrieved:2016-9-14.
  • In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses tests.

1979

• (Holm, 1979) ⇒ Sture Holm. (1979). "A Simple Sequentially Rejective Multiple Test Procedure." Scandinavian Journal of Statistics.
  • NOTE: The Bonferroni procedure, while controlling FWER, is often too conservative. Sequential procedures that test hypotheses in order can maintain FWER control while providing more power to reject false null hypotheses.

1961

• (Tukey, 1961) ⇒ John W. Tukey. (1961). "The Problem of Multiple Comparisons." Princeton University.
  • NOTE: When making multiple comparisons, we must distinguish between the error rate per comparison and the error rate per family of comparisons. The latter, which we now call family-wise error rate, is crucial for maintaining the overall validity of our statistical inferences.

---