Randomized Experiments: Design and Inference

Why Randomization Works, and How to Get the Most from It

Fei Huang, UNSW Sydney

• Randomization is the benchmark every other method in these notes tries to imitate.
• This chapter covers why it works, how to estimate effects and quantify uncertainty, and the design choices that determine whether an experiment can answer its question: the unit of randomization, blocking, covariates, and power.
• Case study: a randomized IRS outreach experiment with 4.5 million tax returns, used to estimate the effect of health insurance on mortality.

Where This Chapter Fits

Step 1: Can you randomly assign who is treated?

• You arrive here when: You control assignment, so you can make it random.
• This chapter: Why randomization removes selection bias, how to estimate and test effects, and the design choices that determine power.
• Next: Chapter 4: running experiments in real organizations.

See the design flowchart in Chapter 1.

In Practice: Randomized Experiments

Practitioner Summary

• Use it when: you, or a partner organization, control who gets the treatment, and withholding or delaying it for some units is acceptable
• What you need: the assignment list, baseline covariates (ideally past outcomes), and an outcome you can measure for nearly everyone
• What must hold: assignment really was random and was followed; no spillovers between treated and control units
• Checks to run: a power calculation before launch; a balance table with a joint test; attrition by arm
• In R: lm_robust(y ~ treat, data) from estimatr (HC2 by default, matching the formula above), or clusters = team if teams were randomized; a permutation loop for randomization inference (this chapter)
• Telling a manager: “A lottery decided who got it, so the only systematic difference between the groups is the program. The gap in outcomes is its effect, within this margin of error.”

Why Randomization Works

Randomization Removes Selection Bias

Independence by Design

If a lottery decides who is treated, treatment status cannot depend on anything about the units, including their potential outcomes:

\big(Y_i(0), Y_i(1)\big) \perp D_i

Then E[Y_i(0) \mid D_i = 1] = E[Y_i(0) \mid D_i = 0], the selection-bias term from Chapter 2 is zero, and

E[Y_i \mid D_i = 1] - E[Y_i \mid D_i = 0] = E[Y_i(1) - Y_i(0)] = \text{ATE}

What Randomization Balances

Randomization balances treatment and control groups in expectation on every characteristic: observed or unobserved, measured before treatment or never measured at all. No other design does this. That is why the rest of the notes ask, for each method, “what would make this comparison as good as random?”

Unbiased Is Not the Same as Exact

Any Single Experiment Can Be Unlucky

Unbiasedness is a property of the procedure: averaged over all possible random assignments, the difference in means equals the ATE. In any one assignment, chance imbalances move the estimate up or down.

• With 8 workers (Chapter 2), lottery estimates of a true effect of 0.25 vary with a standard deviation of about 0.4
• With thousands of units, chance imbalances shrink and the estimate concentrates around the truth

Quantifying this variation is the job of inference.

Estimation and Inference

The Difference in Means

The Workhorse Estimator

\hat\tau = \bar Y_1 - \bar Y_0, \qquad \widehat{SE}(\hat\tau) = \sqrt{\frac{s_1^2}{n_1} + \frac{s_0^2}{n_0}}

where \bar Y_d, s_d^2, and n_d are the mean, variance, and size of group d.

• Equivalent to OLS of Y_i on D_i; this standard error equals the heteroskedasticity-robust (HC2) standard error
• For the effect in the sample at hand, the formula is conservative: it cannot account for how individual effects vary, which would reduce the true variance. It is exactly right if the sample is a random draw from a larger population
• With cluster-level assignment, cluster standard errors at the level of randomization

Randomization Inference

Testing Without a Model of the Data

Take the sharp null hypothesis that treatment had no effect for anyone: Y_i(1) = Y_i(0) for all i. Under this null we know every unit’s outcome under any assignment, so we can:

1. Re-randomize treatment many times, following the original design
2. Recompute the estimate each time
3. Report the share of re-randomized estimates at least as extreme as the actual one: the p-value

The p-value is exact in finite samples if all possible assignments are used (with a large random sample of them, as below, it is exact up to simulation error). It makes no distributional assumptions and respects the design (blocks, clusters) when re-randomization follows the original design. It is especially useful in small experiments.

Randomization Inference in R

set.seed(2026)
n <- 100
d <- sample(rep(0:1, n / 2))                     # completely randomized design
y <- 50 + 4 * d + rnorm(n, 0, 8)                 # true effect = 4

observed <- mean(y[d == 1]) - mean(y[d == 0])

# Re-randomize under the sharp null of no effect
perm <- replicate(10000, {
  d_star <- sample(d)
  mean(y[d_star == 1]) - mean(y[d_star == 0])
})

c(estimate = round(observed, 2),
  p_randomization = mean(abs(perm) >= abs(observed)),
  p_t_test = round(t.test(y ~ d)$p.value, 4))

       estimate p_randomization        p_t_test 
         4.8400          0.0012          0.0011 

Designing an Experiment

Choosing the Unit of Randomization

Individuals or Clusters?

Randomize…	When	Cost
Individuals	Little risk of spillovers; treatment can be delivered person by person	Spillovers bias the comparison if they occur
Clusters (teams, schools, branches, villages)	Treatment is delivered to groups, or spillovers within groups are likely	Fewer independent units, so less power

The Design Effect

With clusters of size m and intra-cluster correlation \rho (how similar outcomes are within a cluster), the variance of the estimate is inflated by roughly

\text{DEFF} = 1 + (m - 1)\rho

Example: teams of 6 software developers with \rho = 0.2 give \text{DEFF} = 2, so the experiment behaves as if it had half as many developers. Cui et al. (2026, see Chapter 5) randomized some developers individually and others by team.

Blocking (Stratified Randomization)

Randomize Within Groups of Similar Units

Divide units into blocks by characteristics that predict the outcome (age, income, prior performance), then randomize within each block.

• Guarantees balance on the blocking variables
• Improves precision when blocking variables predict outcomes
• Makes subgroup analysis cleaner, since each subgroup is itself a small experiment
• Analysis: estimate the effect within each block and take a weighted average by block size; a regression with block fixed effects gives the same answer only when the treated share is the same in every block (otherwise it weights blocks by their treatment variance)

Goldin et al. (2021, case study below) stratified by age, gender, marital status, dependents, income, self-employment, prior penalty status, and state policy environment.

Adjusting for Covariates

Regression Adjustment in Experiments

Y_i = \alpha + \tau D_i + X_i'\beta + \varepsilon_i

• Randomization already makes the difference in means unbiased; covariates are about precision (regression adjustment has a small finite-sample bias that vanishes in large samples)
• Use pre-treatment covariates only (measured before assignment, or unaffected by it)
• Prognostic covariates, especially the baseline value of the outcome, can cut standard errors substantially
• Interacting D_i with demeaned covariates gives an estimator that is never less precise asymptotically (Lin 2013)
• Pre-specify the covariates to avoid searching for the “best” result

Power and Sample Size

The Minimum Detectable Effect

With a share P of N units treated, outcome standard deviation \sigma, significance level \alpha, and power 1 - \kappa:

\text{MDE} = \big(z_{1-\alpha/2} + z_{1-\kappa}\big) \sqrt{\frac{1}{P(1-P)}} \; \frac{\sigma}{\sqrt N}

• Power grows with \sqrt N: halving the MDE needs four times the sample
• Unequal allocation (P \neq 0.5) costs power; budgets or ethics sometimes require it
• Imperfect take-up dilutes the intent-to-treat effect: if only 10% of the treated group take up (and no one in the control group can), the ITT is one-tenth of the effect on those who do, and the required sample grows by a factor of about 100

Power for a Rare Outcome

# Illustration: annual mortality of 1.0%; we want to detect a fall to 0.94%
# (a 6% relative reduction) with 80% power at the 5% level
pw <- power.prop.test(p1 = 0.0100, p2 = 0.0094, power = 0.80, sig.level = 0.05)
ceiling(pw$n)                                   # required per group

[1] 418864

# Now suppose only 10% of those offered treatment actually take it up:
# the intent-to-treat difference is ten times smaller
pw_itt <- power.prop.test(p1 = 0.0100, p2 = 0.01 - 0.0006 * 0.10, power = 0.80)
ceiling(pw_itt$n)

[1] 43040636

• Rare outcomes plus low take-up are why credible experiments on insurance and mortality need millions of participants, or must focus on higher-risk groups. Goldin et al. do both.

Balance Checks

What a Balance Table Can and Cannot Tell You

• Purpose: to detect failures of randomization or implementation (wrong lists, manipulated assignment), and to describe the sample
• With many covariates, about 1 in 20 will differ at the 5% level by chance; a joint test is more informative than counting stars
• A chance imbalance does not make the estimator biased; adjusting for the imbalanced pre-treatment covariate can improve precision
• Do not re-run the randomization until the table “looks good” unless that rule was planned in advance

Multiple Arms and Factorial Designs

Testing More Than One Version

• Multiple arms: several versions of treatment against a shared control group (e.g., letters with and without personalized information)
• Factorial designs: cross two treatments (A only, B only, both, neither) to estimate each effect and their interaction
• Trade-off: each comparison uses a smaller share of the sample, so arms must be chosen with power in mind
• Multiple testing: with many arms and outcomes, pre-specify the main comparisons and adjust for multiple hypotheses

Case Study

Case Study: Health Insurance and Mortality

Goldin, Lurie & McCubbin (2021), Quarterly Journal of Economics (Goldin et al. 2021)

Question: Does health insurance coverage reduce mortality?

Why it is hard: people who buy insurance differ from those who do not (Chapter 1), and mortality is rare, so experiments need enormous samples.

The opportunity: in early 2017 the US Internal Revenue Service (IRS) planned to send letters encouraging people who had paid a penalty for being uninsured to look at coverage options. A budget limit meant not everyone could be sent one, so the agencies randomized.

Case Study: The Design

A Very Large, Stratified, Multi-Arm Experiment

• Sample: 4.5 million tax returns (8.9 million people) that paid the 2015 penalty for lacking coverage
• Assignment: 86% sent a letter, 14% control; randomization stratified by age, gender, marital status, dependents, income, self-employment, prior penalty, and state policy environment
• Arms: a baseline letter with a personalized penalty estimate; a non-personalized version; an earlier mailing date; versions with exemption information or a Spanish translation
• Outcomes: coverage and mortality from administrative records (tax forms and Social Security death records): no surveys and essentially no attrition
• Registered in the AEA RCT Registry and approved by a university ethics board

• Point out the unequal allocation (86/14): driven by the program’s goal of broad outreach and its budget, at a cost in power.

Case Study: Results

What the Letters Did

• Coverage: previously uninsured recipients were 1.1 percentage points more likely to enroll over two years (+1.9%), about 0.23 extra months of coverage on average
• Arms differ: the non-personalized letter was about 26% less effective than the personalized one; the earlier mailing about 40% more effective; exemption information and Spanish translation made no difference
• Mortality: among previously uninsured adults aged 45–64, two-year mortality fell by about 0.06 percentage points, roughly one fewer death per 1,587 people sent a letter
• No detectable effect for children or younger adults, where deaths are much rarer

From the Letter’s Effect to Insurance’s Effect

The mortality estimate is an intent-to-treat effect of the letter. Dividing by the effect on coverage (an instrumental variables estimate, Chapters 5–6) gives the effect of coverage itself for people induced to enroll by the letter: about −0.18 percentage points per month of coverage, imprecise but large enough to rule out the much smaller estimate from a naive regression of mortality on coverage.

Case Study: Design Lessons

What This Experiment Teaches

• Scale beats rare outcomes: a 0.06 pp effect is only detectable with millions of participants
• Target where power is: restricting to 45–64-year-olds, whose mortality is higher and whose coverage responded more, sharpened the estimate
• Administrative outcomes avoid attrition and self-report bias
• Arms answer design questions that matter for the next outreach campaign (personalize, send early)
• The control group is not “no outreach”: controls may have seen TV ads or other campaigns, so the estimate is the effect of the letter on top of everything else happening

Case Study: What to Copy

Practices to Reuse

• Piggyback on a decision already being made. A budget limit meant not everyone could get a letter, so randomizing who did cost the agency nothing.
• Measure outcomes from records. Tax and death records gave essentially no attrition and no self-report bias.
• Add cheap arms that answer operational questions. Personalized vs. generic letters and early vs. late mailing told the agency how to run the next campaign.

Case Study: Discussion Questions

1. Why did the IRS randomize at the tax return (household) level rather than the individual level?
2. The split was 86% treated and 14% control. What did this choice cost in power, and why might the agencies still prefer it?
3. Why does stratifying by income and age matter more for precision here than for bias?
4. The mortality effect is significant for 45–64-year-olds but not for younger adults. Is that heterogeneity, low power, or both? How would you tell?
5. Why is the IV estimate of the effect of coverage larger than the effect of the letter? Whose effect does it describe?

Case Study: Suggested Answers (1–2)

1. One letter was sent per tax return, addressed to the filers, so the return is the natural unit. People on the same return also make coverage decisions together; randomizing individuals within a return would put treated and control people in the same household and invite spillovers. Standard errors are clustered by household.
2. For a fixed sample, the variance of a difference in means scales with 1/[P(1-P)]. At P = 0.86 this is about 8.3, against 4 at an even split, so standard errors are about 1.4 times larger. The agencies’ aim was outreach, so they wanted as many people as possible to receive a letter.

Case Study: Suggested Answers (3–4)

3. Randomization already makes the arms comparable on income and age on average, so stratifying cannot remove bias. It removes chance imbalance on strong predictors of coverage and mortality, which makes estimates more precise.
4. Probably both. Deaths are rare among younger adults, so power is low, and their coverage response may also differ. Compare confidence intervals, not just significance: a wide interval that includes the older group’s effect points to low power; a precise estimate near zero points to a real difference.

Case Study: Suggested Answers (5)

5. The IV estimate divides the letter’s effect on mortality by its effect on coverage, which is far less than one month per person, so the ratio is larger. It is the effect per month of coverage for compliers: people who enrolled because they received a letter.

Summary

Summary

• Randomization makes treatment independent of potential outcomes, so the difference in means is unbiased for the ATE
• Unbiased ≠ exact: inference quantifies chance variation; randomization inference gives exact p-values under the sharp null
• Design choices determine what an experiment can learn: unit of randomization, blocking, covariates, arms, and sample size
• Power is often the binding constraint, especially with rare outcomes or low take-up
• Large administrative experiments, like the IRS letters, can answer questions observational data cannot

Next Steps

1. Read: Goldin et al. (2021), especially the sections on the outreach intervention and the mortality results
2. Practice: In the randomization-inference code, set the true effect to 0 and re-run 20 times. How often is p < 0.05?
3. Practice: Use power.prop.test to find the sample needed if mortality were 2% instead of 1%
4. Next Class: Running experiments in practice: firms, online platforms, and pre-registration (Chapter 4)

References

Cui, Zheyuan Kevin, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. 2026. “The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers.” Management Science, ahead of print. https://doi.org/10.1287/mnsc.2025.00535.

Goldin, Jacob, Ithai Z Lurie, and Janet McCubbin. 2021. “Health Insurance and Mortality: Experimental Evidence from Taxpayer Outreach.” The Quarterly Journal of Economics 136 (1): 1–49. https://doi.org/10.1093/qje/qjaa029.

Lin, Winston. 2013. “Agnostic Notes on Regression Adjustments to Experimental Data: Reexamining Freedman’s Critique.” The Annals of Applied Statistics 7 (1): 295–318. https://doi.org/10.1214/12-AOAS583.