Bootstrap Confidence Intervals and Hypothesis Testing

Mathematical formulation

Nonparametric unit bootstrap, three causal estimators, confidence intervals, and bootstrap-assisted hypothesis testing.

NATIVE MATHML

Nonparametric bootstrap map

{\hat{F}}_{n} = \frac{1}{n} \sum_{i = 1}^{n} δ_{O_{i}} ⟶ O_{1}^{* (b)}, \dots, O_{n}^{* (b)} \overset{iid}{\sim} {\hat{F}}_{n} ⟶ {\hat{Δ}}_{m}^{* (b)}

Observed unit

O_{i} = (Y_{i}, T_{i}, X_{i})

Resample the complete subject record. Never resample Y, T, and X independently.

Multiplicity representation

(N_{1}^{*}, \dots, N_{n}^{*}) \sim Multinomial (n; \frac{1}{n}, \dots, \frac{1}{n})

Duplicates and omissions are expected. Approximately 63.2% of subjects are distinct in one large bootstrap sample.

Refit the full procedure

{\hat{Δ}}_{m}^{* (b)} = 𝒜 (O_{1}^{* (b)}, \dots, O_{n}^{* (b)})

The algorithm 𝒜 includes refitting propensity and outcome models—not merely reusing original fitted predictions.

IPW, Horvitz–Thompson

\hat{Δ} = \frac{1}{n} \sum [\frac{T_{i} Y_{i}}{{\hat{e}}_{i}} - \frac{(1 - T_{i}) Y_{i}}{1 - {\hat{e}}_{i}}]

Each bootstrap sample refits ê(X). Fitted probabilities are bounded numerically to [0.005, 0.995].

Outcome regression

\hat{Δ} = \frac{1}{n} \sum {\hat{m} (1, X_{i}) - \hat{m} (0, X_{i})}

Standardize both predicted treatment worlds over the same empirical X distribution.

Doubly robust / AIPW

\hat{Δ} = \frac{1}{n} \sum [\hat{m} (1, X_{i}) - \hat{m} (0, X_{i}) + \frac{T_{i}}{{\hat{e}}_{i}} {Y_{i} - \hat{m} (1, X_{i})} - \frac{1 - T_{i}}{1 - {\hat{e}}_{i}} {Y_{i} - \hat{m} (0, X_{i})}]

Consistency requires at least one nuisance model to be correctly specified under standard regularity and positivity conditions.

Bootstrap spread and local bootstrap bias

\begin{matrix} B_{s} = number of successful bootstrap fits \\ {\bar{Δ}}_{m}^{*} = \frac{1}{B_{s}} \sum_{b = 1}^{B_{s}} {\hat{Δ}}_{m}^{* (b)}, {\hat{Bias}}_{boot, m} = {\bar{Δ}}_{m}^{*} - {\hat{Δ}}_{m} \\ {\hat{SE}}_{boot, m} = \sqrt{\frac{1}{B_{s} - 1} \sum_{b = 1}^{B_{s}} {{\hat{Δ}}_{m}^{* (b)} - {\bar{Δ}}_{m}^{*}}^{2}} \end{matrix}

Percentile interval

[q_{α / 2, m}^{*}, q_{1 - α / 2, m}^{*}]

The endpoints are empirical quantiles of the successful bootstrap estimates. The percentile interval respects strictly monotone transformations, but it is generally only first-order accurate and can miscover when estimator bias or skewness is substantial.

Normal / Wald interval

{\hat{Δ}}_{m} \pm z_{1 - α / 2} {\hat{SE}}_{boot, m}

This interval is symmetric around the original estimate. With the same standard error and critical value, it is algebraically dual to the corresponding two-sided Wald test.

Interpretation of the bootstrap bias estimate: mean(Δ̂*) − Δ̂ estimates local finite-sample bias under the empirical resampling law. It is not an estimate of causal bias from unmeasured confounding, poor overlap, or persistent model misspecification.

Null hypothesis and supported alternatives

H_{0} : Δ = Δ_{0} versus H_{1} : Δ \neq Δ_{0}, Δ > Δ_{0}, or Δ < Δ_{0}

Bootstrap-SE Wald test

z = \frac{\hat{Δ} - Δ_{0}}{{\hat{SE}}_{boot}}

Two-sided: p = 2{1−Φ(|z|)}
Greater: p = 1−Φ(z)
Less: p = Φ(z)

The test uses normal calibration with the bootstrap-estimated standard error.

Centered, unstudentized bootstrap test

\begin{matrix} D_{obs} = \hat{Δ} - Δ_{0}, D_{b}^{*} = {\hat{Δ}}_{b}^{*} - \hat{Δ} \\ p_{cent} = \frac{1 + \sum 𝟙 {| D_{b}^{*} | \geq | D_{obs} |}}{B_{s} + 1} \end{matrix}

The displayed formula is for the two-sided alternative; one-sided versions use the corresponding signed inequality. This unstudentized test is approximate and is not generally dual to the percentile interval.

CI/test duality

Reject H_{0} \Leftrightarrow Δ_{0} \notin C_{1 - α}

This equivalence requires a matching dual pair. The two-sided Wald test matches the two-sided normal/Wald interval; directional Wald tests match the corresponding one-sided lower or upper confidence bound.

Independent resampling units

The unit bootstrap assumes subjects are i.i.d. If data are clustered, longitudinal, paired, or survey sampled, resample the appropriate cluster or use a design-respecting method.

Regular estimator

The ordinary bootstrap can fail for nonregular estimators, boundary parameters, extreme model selection, or some highly adaptive machine-learning procedures without additional theory.

Causal identification precedes inference

Bootstrap uncertainty does not repair unmeasured confounding, poor overlap, measurement error, nuisance-model bias, or a mismatch between the estimand and analysis.

Interpretation: a narrow bootstrap interval can be precisely centered on the wrong causal value. The simulation-only sampling-distribution panel makes that failure visible when hidden confounding or model misspecification is introduced.
Failed fits: summaries use only the B_s successful bootstrap analyses. Frequent failures are themselves a warning about instability or nonregularity; simply discarding many failed fits can distort the empirical bootstrap distribution.

9. Does the bootstrap mimic repeated sampling?

Compare bootstrap errors from the observed study with estimator errors across independently generated studies from the known DGP.

Method

Run the simulation-only comparison after completing a bootstrap.

Inspect and export numerical results

Bootstrap b	IPW RD	OR RD	DR RD

Bootstrap Confidence Intervals and Hypothesis Testing — Se Yoon Lee, Ph.D.

Mathematical formulation

Observed unit

Multiplicity representation

Refit the full procedure

IPW, Horvitz–Thompson

Outcome regression

Doubly robust / AIPW

Percentile interval

Normal / Wald interval

Bootstrap-SE Wald test

Centered, unstudentized bootstrap test

CI/test duality

Independent resampling units

Regular estimator

Causal identification precedes inference

1. The observed study

Original point estimates

2. One bootstrap resample

Selected resample versus original analysis

3. Bootstrap distribution

4. Monte Carlo stability as B increases

5. Confidence-interval methods

6. Hypothesis test

7. Normal-approximation power curve

8. All adjusted methods

9. Does the bootstrap mimic repeated sampling?