Causal Inference Framework

Potential Outcomes, Estimands, and Assignment

Instructor

Fei Huang, UNSW Sydney

  • Chapter 1 introduced the fundamental problem: we never see both potential outcomes for the same unit.
  • This chapter builds the language used in the rest of the notes: potential outcomes, estimands, the assumptions about treatments (SUTVA), assignment mechanisms, and causal diagrams for deciding what to control for.
  • The running example is a firm giving some workers access to a generative AI assistant.

Potential Outcomes

Units, Treatments, Outcomes

NoteThe Ingredients of a Causal Question
  • Units i = 1, \dots, N: the entities affected (workers, households, firms, postcodes)
  • Treatment D_i: the intervention whose effect we want, e.g., access to an AI assistant (D_i = 1) or not (D_i = 0)
  • Outcome Y_i: what the intervention might change, e.g., tickets resolved per hour
  • Potential outcomes Y_i(1) and Y_i(0): the outcome unit i would have under each treatment
  • Unit-level effect \tau_i = Y_i(1) - Y_i(0)
TipA Treatment Must Be Something We Could Change

“What is the effect of a worker’s experience?” is vague. “What is the effect of giving a worker six months of on-the-job training?” is a well-defined intervention with two clear potential outcomes. Precise policy questions start with a treatment that someone could actually manipulate.

A Potential Outcomes Table

NoteEight Customer-Support Workers (a Thought Experiment)
Worker Y_i(0): without AI Y_i(1): with AI \tau_i
A (new) 1.0 1.6 +0.6
B (new) 1.2 1.7 +0.5
C (new) 1.4 1.8 +0.4
D 1.8 2.1 +0.3
E 2.0 2.2 +0.2
F 2.2 2.3 +0.1
G (expert) 2.6 2.6 0.0
H (expert) 2.8 2.7 −0.1

Average effect: \frac{1}{8}\sum_i \tau_i = +0.25 tickets per hour. Effects differ across workers, and in reality we only ever see one column for each.

  • The pattern (larger gains for newer workers) mirrors the findings of Brynjolfsson, Li and Raymond (2025), studied in Chapters 8 and 9.

What Happens When Managers Choose

Code 
po <- data.frame(
  worker = LETTERS[1:8],
  y0 = c(1.0, 1.2, 1.4, 1.8, 2.0, 2.2, 2.6, 2.8),
  y1 = c(1.6, 1.7, 1.8, 2.1, 2.2, 2.3, 2.6, 2.7)
)
ate <- mean(po$y1 - po$y0)

# Scenario 1: managers give AI to their four strongest workers
d_mgr <- as.integer(po$worker %in% c("E", "F", "G", "H"))
naive_mgr <- mean(po$y1[d_mgr == 1]) - mean(po$y0[d_mgr == 0])

# Scenario 2: AI assigned by lottery, repeated many times
set.seed(2026)
lottery <- replicate(10000, {
  d <- sample(rep(0:1, 4))
  mean(po$y1[d == 1]) - mean(po$y0[d == 0])
})

round(c(true_ATE = ate,
        managers_choose = naive_mgr,
        lottery_average = mean(lottery),
        lottery_sd = sd(lottery)), 3)
       true_ATE managers_choose lottery_average      lottery_sd 
          0.250           1.100           0.249           0.375
  • When managers choose the strongest workers, the comparison overstates the effect even though those workers gain the least: their high baseline does the work.
  • Under a lottery, the comparison is right on average; any single lottery can still be off (the standard deviation). Chapter 3 is about that sampling variation.

Estimands

Which Average Effect?

NoteCommon Estimands
Estimand Definition Policy question it answers
ATE E[Y_i(1) - Y_i(0)] What if everyone got the treatment?
ATT E[Y_i(1) - Y_i(0) \mid D_i = 1] Did the program help the people who actually received it?
ATC E[Y_i(1) - Y_i(0) \mid D_i = 0] What if we extended it to those not yet covered?
CATE E[Y_i(1) - Y_i(0) \mid X_i = x] Who benefits most? How should we target?
TipIn the AI Example

If managers give the tool to the experts (E–H), the ATT is +0.05, while the ATC (new and mid-level workers) is +0.45. Both are “the effect of AI”, and they differ by a factor of nine. Always say whose effect you are estimating.

Estimands in Later Chapters

NoteMany Designs Recover a Specific, Local Effect
Design What it recovers Chapter
Randomized experiment ATE (or intent-to-treat effect) 3–5
Experiment with imperfect take-up, or an instrument LATE: effect for compliers, whose treatment responds to the instrument 5–6
Regression discontinuity Effect for units at the cutoff 7
Panel fixed effects A variance-weighted average of within-unit effects 8
Difference-in-differences ATT for treated units, after treatment 9

A credible local effect is often more useful than a non-credible global one. But it should be described as what it is.

Assumptions About the Treatment

SUTVA: No Interference

WarningOne Unit’s Treatment Must Not Change Another’s Outcome

The Stable Unit Treatment Value Assumption (SUTVA) has two parts. The first is no interference: Y_i depends only on D_i, not on D_j.

When it fails:

  • Vaccines: my vaccination protects you
  • Platforms and markets: an incentive that makes some ride-hail drivers work more lowers earnings for other drivers
  • Teams: if half a support team gets an AI assistant, the untreated half may receive harder tickets, or learn from treated colleagues
  • Information: treated households tell untreated neighbours about a program

Interference biases simple comparisons and changes what the estimand means. Designs that randomize whole clusters (teams, villages, markets) are one response (Chapter 5).

SUTVA: No Hidden Versions of Treatment

Warning“The Treatment” Must Mean One Thing

The second part: there is only one version of each treatment level.

  • “Health insurance” may mean Medicaid or a private plan with a high deductible
  • “An AI assistant” in 2021 (GPT-3) is not the same treatment as in 2025
  • “Training” delivered by an expert and by a novice trainer may have different effects

If versions differ, define the treatment more narrowly, or measure and analyze versions separately.

NoteAssignment Is Not Receipt

Being offered a treatment is not the same as taking it. Many evaluations randomize or study the offer (a letter, an invitation, eligibility). The effect of the offer is the intent-to-treat effect; the effect of actually taking the treatment requires extra assumptions (Chapters 5–6).

The Assignment Mechanism

Why Assignment Is Everything

TipRecall the Decomposition from Chapter 1

E[Y_i \mid D_i = 1] - E[Y_i \mid D_i = 0] = \text{ATT} + \underbrace{E[Y_i(0) \mid D_i = 1] - E[Y_i(0) \mid D_i = 0]}_{\text{selection bias}}

Selection bias is zero when treatment is unrelated to untreated potential outcomes. Whether that holds depends entirely on how units came to be treated: the assignment mechanism.

Five Assignment Mechanisms

NoteHow Treatment Was Assigned Determines the Method
Assignment mechanism Key condition Methods
The researcher randomizes (Y_i(0), Y_i(1)) \perp D_i Experiments (Ch. 3–5)
An outside shock shifts treatment The shock is as good as random and affects outcomes only through treatment Instrumental variables (Ch. 6)
A rule based on a score Treatment switches at a known cutoff c of a running variable Regression discontinuity (Ch. 7)
Units choose, observed over time Untreated outcomes evolve similarly, or confounders are fixed within units Fixed effects, DiD, synthetic control (Ch. 8–10)
Units choose, observed once (Y_i(0), Y_i(1)) \perp D_i \mid X_i, with overlap Regression, matching (Ch. 11)
  • Ask students to place familiar policies (a lottery-based housing program, a means-tested subsidy, a state law, a firm’s choice of who gets new software) into one of the five rows.

Causal Diagrams

Drawing Your Assumptions

NoteThree Basic Structures

A causal diagram draws variables as nodes and causal effects as arrows. Three patterns matter most for deciding what to control for:

Structure Picture Example Control for it?
Confounder D \leftarrow X \rightarrow Y Firm size drives both AI adoption and productivity Yes: controlling for it blocks a non-causal (back-door) path
Mediator D \rightarrow M \rightarrow Y Insurance → doctor visits → health No if you want the total effect
Collider D \rightarrow C \leftarrow Y AI adoption and productivity both make a firm “featured” in the press No: conditioning on it creates bias

Good and Bad Controls

WarningControlling for More Is Not Always Better
  • Confounders measured before treatment: good controls
  • Mediators (affected by treatment): controlling for them removes part of the effect you want to measure
  • Colliders, including sample selection on an outcome-related variable: conditioning on them creates spurious associations
  • Pre-treatment variables that only predict the outcome: harmless, and they improve precision
  • Pre-treatment variables that only predict treatment (instrument-like): they add no bias protection, reduce precision, and can amplify bias from any remaining unmeasured confounder

The rule of thumb “only control for variables determined before treatment” avoids most mistakes, but not all: some pre-treatment variables are also bad controls (the instrument-like case above, and pre-treatment colliders, known as M-bias). Causal diagrams handle these cases (Cinelli et al. 2024).

Simulation: Collider Bias

Code 
set.seed(2026)
n <- 5000
ai_adoption  <- rbinom(n, 1, 0.5)          # randomly adopted: NO effect on productivity
productivity <- rnorm(n)

# A firm is "featured" in a business magazine if it adopted AI or is highly productive
featured <- (ai_adoption == 1 & runif(n) < 0.3) | productivity > 1

c(all_firms      = coef(lm(productivity ~ ai_adoption))["ai_adoption"],
  featured_firms = coef(lm(productivity ~ ai_adoption, subset = featured))["ai_adoption"])
     all_firms.ai_adoption featured_firms.ai_adoption 
                0.01276997                -1.11199358
  • Across all firms, the estimated effect is close to zero, as it should be.
  • Among featured firms only, AI adopters look far less productive: non-adopters had to be highly productive to be featured. Studying only the firms that appear in the news, survived a crisis, or published a case study creates this bias.

Identification, Estimation, Inference

Three Separate Questions

TipKeep Them Apart
  1. Identification: if we had infinite data, would our comparison reveal the estimand? This depends on the assignment mechanism and the assumptions we are willing to make. It is never solved by more data.
  2. Estimation: which formula do we apply to a finite sample (difference in means, regression, matching, two-stage least squares)?
  3. Inference: how uncertain is the estimate? Uncertainty comes from sampling (who is in the data) and from design (which units happened to be treated).

Most disagreements about a study are about step 1. Most of these notes are too.

Summary

  • A causal effect compares potential outcomes for the same unit; only one is ever observed
  • Estimands (ATE, ATT, ATC, CATE, LATE, local effects) answer different policy questions; say which one you target
  • SUTVA requires no interference and a single, well-defined version of treatment
  • The assignment mechanism determines whether a comparison is biased, and which method fits
  • Causal diagrams clarify which variables to control for: confounders yes; mediators and colliders no
  • Separate identification from estimation and inference

Next Steps

  1. Read: Holland (1986); Cinelli et al. (2024)
  2. Practice: In the eight-worker simulation, let managers give AI to the four newest workers. Is the naive comparison now too high or too low? Why?
  3. Think: For a policy you know well, write down the treatment, the outcome, the estimand, and the assignment mechanism
  4. Next Class: Randomized experiments: design and inference (Chapter 3)

References

Cinelli, Carlos, Andrew Forney, and Judea Pearl. 2024. “A Crash Course in Good and Bad Controls.” Sociological Methods & Research 53 (3): 1071–104. https://doi.org/10.1177/00491241221099552.
Holland, Paul W. 1986. “Statistics and Causal Inference.” Journal of the American Statistical Association 81 (396): 945–60. https://doi.org/10.1080/01621459.1986.10478354.