Coverage Probability Validation Method

A Coverage Probability Validation Method is a statistical validation method that assesses whether confidence intervals achieve their nominal coverage level through empirical testing or theoretical analysis.

• AKA: CI Coverage Assessment Method, Actual Coverage Estimation Method, Coverage Probability Testing Method, Interval Coverage Validation Method.
• Context:
  • It can typically compute actual coverage as proportion of intervals containing true parameter.
  • It can typically compare actual coverage against nominal coverage (e.g., 95%) to detect undercoverage or overcoverage.
  • It can typically use Monte Carlo simulations with known truth to evaluate interval methods.
  • It can often reveal that Wald intervals undercover (e.g., 89% actual for 95% nominal) in small samples.
  • It can often identify when Wilson intervals achieve closer to nominal coverage.
  • It can often guide selection between competing interval methods based on coverage performance.
  • It can range from being a Simulation-Based Coverage Probability Validation Method to being a Theoretical Coverage Probability Validation Method, depending on its validation approach.
  • It can range from being a Point Coverage Probability Validation Method to being a Uniform Coverage Probability Validation Method, depending on its parameter space coverage.
  • It can range from being a Conditional Coverage Probability Validation Method to being a Unconditional Coverage Probability Validation Method, depending on its conditioning strategy.
  • It can range from being a Exact Coverage Probability Validation Method to being a Approximate Coverage Probability Validation Method, depending on its precision requirement.
  • ...
• Example(s):
  • Monte Carlo Coverage Studys, such as:
    • 10,000 simulations: Wald CI achieves 87.3% coverage for n=20, F1=0.8.
    • Wilson CI achieves 94.1% coverage for same scenario.
    • BCa bootstrap achieves 93.8% coverage with B=1000 replicates.
  • Parameter Space Evaluations, such as:
    • Coverage heatmap: F1 ∈ [0,1] × n ∈ [10,200].
    • Poor coverage near boundaries (F1<0.1 or F1>0.9).
    • Adequate coverage (±1%) for F1 ∈ [0.3,0.7] and n>50.
  • Method Comparison Studys, such as:
    • Five methods tested: Wald, Wilson, Score, Bootstrap, Profile.
    • Wilson best overall: average 94.8% coverage across scenarios.
    • Trade-off analysis: coverage vs interval width.
  • ...
• Counter-Example(s):
  • Interval Width Evaluation, which focuses on precision not coverage.
  • Bias Assessment Method, which evaluates point estimates.
  • Power Analysis Method, which evaluates hypothesis test rejection rates.
• See: Statistical Validation Method, Coverage Probability, Confidence Interval, Monte Carlo Simulation, Wilson Score F1 Confidence Interval Method, Wald F1 Confidence Interval Method, Coverage Empirical Studies Catalog Method, F1 Confidence Interval Construction Method, Nominal Coverage, Actual Coverage, Undercoverage, Small Sample Inference, Interval Estimation Theory, F1 Interval Selection Guide Method.

---