Significance Level

A Significance Level is a probability threshold that specifies the maximum acceptable Type I error probability for rejecting a null hypothesis in statistical hypothesis testing.

• AKA: Alpha Level, Alpha, Type I Error Rate, Significance Threshold, Size of Test.
• Context:
  • It can typically be set before data collection to control false positive error rate in hypothesis testing procedures.
  • It can often determine the hypothesis test rejection region by defining critical values for test statistics.
  • It can be defined mathematically as P(reject H₀ | H₀ is true) = α.
  • It can be calculated from confidence levels as α = 1 - (confidence level/100).
  • It can range from being a Conservative Significance Level to being a Liberal Significance Level, depending on its error tolerance.
  • It can influence statistical power measures through the Type I-Type II error tradeoff.
  • It can require adjustment in multiple hypothesis testing through family-wise error rate or false discovery rate control.
  • It can determine whether a test statistic produces a statistically significant result.
  • It can vary by research field, with particle physics using α = 3×10⁻⁷ and social sciences using α = 0.05.
  • It can affect sample size determination tasks through power analysis calculations.
  • It can be compared with p-values to make null hypothesis rejection decisions.
  • ...
• Example(s):
  • Standard Significance Levels, such as:
    • 0.05 Significance Level for general hypothesis testing and social science research.
    • 0.01 Significance Level for medical research requiring stringent error control.
    • 0.001 Significance Level for high-stakes testing with severe consequences.
    • 0.10 Significance Level for exploratory analysis and pilot studys.
  • Field-Specific Significance Levels, such as:
    • 5-Sigma Significance Level (α ≈ 3×10⁻⁷) for particle physics discovery claims.
    • Genome-Wide Significance Level (α = 5×10⁻⁸) for genetic association studys.
    • 0.20 Significance Level for preliminary screening tests.
  • Adjusted Significance Levels, such as:
    • Bonferroni-Corrected Significance Level = α/n for n simultaneous tests.
    • Šidák-Corrected Significance Level = 1-(1-α)^(1/n) for independent tests.
    • FDR-Adjusted Significance Level for false discovery rate control.
  • Statistical Test Significance Levels, such as:
    • t-Test Significance Level for mean comparisons.
    • Chi-Square Test Significance Level for categorical data analysis.
    • ANOVA Significance Level for multiple group comparisons.
  • ...
• Counter-Example(s):
  • P-Value, which is the calculated probability from data rather than the predetermined threshold.
  • Statistical Significance Level, which is the ordinal classification result rather than the probability threshold.
  • Confidence Level, which equals (1 - α) and represents the complement probability.
  • Effect Size, which measures practical importance rather than statistical threshold.
  • Statistical Power, which equals (1 - β) and relates to Type II error.
  • Test Statistic, which is the calculated value rather than the threshold.
• See: Statistical Hypothesis Testing Task, Type I Hypothesis Testing Error, Null Hypothesis, Rejected Null Hypothesis, Hypothesis Test Rejection Region, Hypothesis Test Acceptance Region, Test Statistic, P-Value, Statistical Power Measure, Sample Size Determination Task, Family-Wise Error Rate, False Discovery Rate.

---

---

References

2017a

• (Wikipedia, 2017) ⇒ http://en.wikipedia.org/wiki/Type_I_and_type_II_errors#Type_I_error
  • A type I error occurs when the null hypothesis (H₀) is true, but is rejected. It is asserting something that is absent, a false hit. A type I error may be likened to a so-called false positive (a result that indicates that a given condition is present when it actually is not present).

The type I error rate or significance level is the probability of rejecting the null hypothesis given that it is true. It is denoted by the Greek letter α (alpha) and is also called the alpha level. Often, the significance level is set to 0.05 (5%), implying that it is acceptable to have a 5% probability of incorrectly rejecting the null hypothesis.

2017b

• (Stat Treak, 2017) ⇒ http://stattrek.com/statistics/dictionary.aspx?definition=P-value Retrieved: 2017-03-07
  • A Type I error occurs when the researcher rejects a null hypothesis when it is true. The probability of committing a Type I error is called the significance level, and is often denoted by α.

---