Hypothesis Testing for Causal Effects

Define the test

Load a lecture result or construct a custom Wald test on the risk-difference scale.

Lecture result

The observed comparison is an association test. The adjusted tests target a causal risk difference only under consistency, conditional exchangeability, positivity, and adequate estimation.

Estimate and uncertainty

Estimated risk difference

−6 pp12 pp

Estimated standard error

0.20 pp4.00 pp

Show the lecture’s stored 95% interval when a lecture preset is active.

Hypothesis

Null risk difference

−6 pp10 pp

Alternative hypothesis

Significance level α

0.010.10

The test direction and α should be chosen before examining the result. Switching to a favorable one-sided test afterward is not valid.

Operating characteristics

Assumed true RD for power

−6 pp12 pp

Repeated samples

Simulation seed

Actions

Mathematical formulation

Estimand, null hypothesis, Wald statistic, p-values, matching confidence sets, Type I error, and power.

RISK-DIFFERENCE SCALE

Potential outcomes → marginal causal risk difference → point null

{RD}_{ATE} = E {Y (1)} - E {Y (0)}; H_{0} : RD = {RD}_{0}

Two-sided alternative

H_{A} : RD \neq {RD}_{0}

Evidence can arise from either a sufficiently positive or sufficiently negative departure from the null.

Directional alternatives

H_{A} : RD > {RD}_{0} or RD < {RD}_{0}

A one-sided direction should be scientifically justified and specified before seeing the estimate.

Observed versus causal contrast

E (Y | T = 1) - E (Y | T = 0) \neq RD

The crude comparison generally targets an association rather than the adjusted causal estimand.

Estimate standardized relative to the null

Z = \frac{\hat{RD} - {RD}_{0}}{\hat{SE} (\hat{RD})} \approx N (0, 1) under H_{0}

Two-sided p-value

p = 2 {1 - Φ (| Z_{obs} |)}

Probability under the null reference model of a statistic at least as extreme in either direction.

Greater-than p-value

p = 1 - Φ (Z_{obs})

Large positive values count against the null.

Less-than p-value

p = Φ (Z_{obs})

Large negative values count against the null.

Interpretation: a p-value is not P(H₀ | data), not the probability the result occurred “by chance,” and not the probability that the causal effect is real.

Matching Wald confidence set

Two-sided: \hat{RD} \pm z_{1 - α / 2} \hat{SE} Greater: [\hat{RD} - z_{1 - α} \hat{SE}, \infty)

Exact procedural duality

A level-α Wald test rejects a null value exactly when that value lies outside the matching Wald confidence set.

Lecture percentile interval

The adjusted lecture analyses display percentile bootstrap intervals but compute p-values from a Wald z-statistic using the bootstrap SE.

Important consequence

A percentile bootstrap interval and a bootstrap-SE Wald p-value are not mathematically exact duals, even when their conclusions happen to agree.

Repeated-sampling operating characteristics

α = P (reject H_{0} | H_{0} true) β (θ) = P (do not reject | θ) Power = 1 - β (θ)

Type I error

Rejecting the chosen point null when it is true. Nominal α is achieved only when the reference approximation and standard error are adequately calibrated.

Type II error

Failing to reject the null at a particular non-null effect. It depends on effect size, standard error, α, and test direction.

Power is not evidence magnitude

Power is a design/operating characteristic, not the posterior probability that the alternative is true after observing data.

Consistency

Y = T Y (1) + (1 - T) Y (0)

Observed treatment versions must correspond to the potential outcomes being compared.

Exchangeability

{Y (1), Y (0)} ⊥ T | X

Measured adjustment must adequately block confounding paths for a causal interpretation.

Positivity and estimation

0 < P (T = 1 | X) < 1

The estimator and uncertainty procedure must also be appropriate for the target population.

Separation of questions: a small p-value addresses sampling compatibility with a null under the fitted analysis. It does not establish exchangeability, repair model misspecification, or quantify unmeasured confounding.

1. Null reference distribution and observed test statistic

The shaded region is the rejection region for the selected alternative and α.

standard-normal density under H₀observed zcritical value

The reference curve is an approximation for the Wald statistic. It is not the distribution of the causal effect itself.

2. Interpretation of the current test

Statistical conclusion, effect magnitude, interval, estimand, and assumptions.

3. Effect estimate, null value, and confidence intervals

The matching Wald interval is shown separately from the lecture’s stored interval.

matching Wald intervallecture intervalestimatenull value

4. Confidence curve: p-value versus the hypothesized null

Move the null slider or inspect every possible point null on the same scale.

Wald p-value functionα thresholdcurrent null

For the matching Wald procedure, point-null values with p ≥ α form the corresponding confidence set.

5. Power curve under the selected standard error

Power is computed under the normal approximation while varying the true risk difference.

powerαassumed true RD

At the null, rejection probability is approximately α. Power rises when the true effect moves into the direction detected by the chosen test.

6. Repeated-sampling experiment

Simulate estimates from the working model \(\widehat{RD}\sim N(RD_{true},SE^2)\).

Plot

The simulation uses the currently selected standard error and test rule.

This simulation illustrates the working reference model. It does not diagnose confounding, poor overlap, model failure, or bootstrap validity.

7. Lecture results: crude and adjusted tests

Click a method to load its estimate, standard error, stored interval, and inferential interpretation.

Method	Estimate (pp)	SE (pp)	z for H₀: RD=0	two-sided p	lecture 95% interval	SE source

8. Full-estimator bootstrap workflow for adjusted methods

The lecture’s adjusted p-values use a bootstrap standard error inside a normal/Wald test.

Resample unitsDraw n observed rows with replacement.

Refit nuisance modelsRefit the treatment and/or outcome model.

Recompute estimatorRebuild IPW, OR, or DR risk difference.

Estimate SEUse the standard deviation of successful bootstrap estimates.

Form Wald zz = (estimate − null)/bootstrap SE.

Important: refitting only the final arithmetic while holding nuisance fits fixed understates full-procedure uncertainty.

9. Four layers that must remain separate

Hypothesis testing occupies only one layer of a causal analysis.

Causal question

What would happen to the target population under smoking versus non-smoking?

Identification & estimation

Which assumptions and estimator connect observed data to the causal risk difference?

Statistical uncertainty

How far is the estimate from the null relative to sampling uncertainty?

Causal-validity uncertainty

How sensitive is the result to unmeasured confounding and other violations?

A small p-value in the statistical layer cannot repair a failure in the identification layer.

10. Hypothesis-testing checklist

Use this checklist before interpreting “significance” causally.

1. EstimandIs the test about a crude association or an adjusted causal estimand?

2. Null value and scaleIs the null defined on the RD, RR, log-RR, odds-ratio, or another scale?

3. Standard errorDoes uncertainty include all fitted nuisance components and estimation steps?

4. Reference approximationIs the Wald/normal approximation adequate, or is another procedure needed?

5. IdentificationAre consistency, exchangeability, and positivity plausible?

6. MagnitudeIs the estimated difference scientifically meaningful, not merely detectable?

7. DirectionWas a one-sided alternative justified and specified in advance?

8. SensitivityWhat happens under unmeasured confounding, poor overlap, or modeling changes?

9. ReportingReport estimate, interval, p-value, method, target, and assumptions together.

Export and inspect

All plots use resolution-independent inline SVG.