Regression and Matching

Selection on Observables

Fei Huang, UNSW Sydney

• This is the last step of the design flowchart: units chose their treatment, we observe them only once (or have no credible variation over time), and we must argue that we observe every factor that drives both treatment and outcome.
• Regression, matching, and propensity scores all rest on that same assumption. They differ in how they use the observed characteristics, not in what they assume.
• Two case studies: LaLonde’s comparison of non-experimental estimates with an experiment (the classic warning), and a study of customers who join a retailer’s social media page (matching combined with difference-in-differences).

Where This Chapter Fits

Step 5: Do you observe every factor that drives both treatment and the outcome?

• You arrive here when: Units chose their treatment, you observe them only once, and you can argue that all confounders are measured.
• This chapter: Selection on observables, regression adjustment, matching, propensity scores, balance checks, and sensitivity analysis.
• Next: Chapter 12: backing up any single design with others that have different weaknesses.

See the design flowchart in Chapter 1.

In Practice: Regression and Matching

Practitioner Summary

• Use it when: no design-based source of variation exists, and you can argue you observe what drove treatment
• What you need: rich pre-treatment covariates, especially past outcomes, and untreated units similar to the treated ones
• What must hold: no unmeasured confounders, and overlap
• Checks to run: an overlap plot; standardized differences before and after (below 0.1); several estimators; a sensitivity analysis
• In R: MatchIt::matchit() and match.data(), sensemakr for sensitivity
• Telling a manager: “Among customers with the same purchase history, joiners visited 5% more. A hidden factor would need to be this strong to explain that away.”

Selection on Observables

The Assumption

Conditional Independence and Overlap

Conditional independence (no unmeasured confounders): among units with the same observed characteristics X, treatment is as good as random

\big(Y_i(0), Y_i(1)\big) \perp D_i \mid X_i

Overlap (common support): for every value of X, there are both treated and untreated units: 0 < P(D_i = 1 \mid X_i) < 1. For the ATT it is enough that every treated unit has comparable untreated units: P(D_i = 1 \mid X_i) < 1, and Y_i(0) \perp D_i \mid X_i

Under both, the effect is identified by comparing like with like and averaging:

\text{ATT} = E\big[\, E[Y \mid D=1, X] - E[Y \mid D=0, X] \;\big|\; D = 1 \big]

The Strongest Untestable Assumption in These Notes

Every other design in these notes uses a source of variation in treatment (a lottery, a shock, a cutoff, timing). Here, the claim is simply that we have measured everything that matters. Data cannot confirm that claim, so the case must be made with knowledge of how treatment was chosen.

When Is It Plausible?

Arguments That Help

• Treatment was assigned by a known process based on recorded information (an underwriting rule, a targeting algorithm, a caseworker’s checklist)
• Rich pre-treatment data, especially past values of the outcome, which capture much of what drives future outcomes
• Institutional knowledge of why some units were treated and others not
• Results are insensitive to adding or removing controls, and survive formal sensitivity analysis

Signs of Trouble

• Units chose treatment based on private information (motivation, health, expected gains) that is not in the data
• Treated and untreated units look very different on observables, suggesting they differ on unobservables too
• Estimates move a lot as controls are added

Three Ways to Compare Like with Like

Regression Adjustment

Controlling for X in a Regression

Y_i = \alpha + \tau D_i + X_i'\beta + \varepsilon_i

• Simple and easy to check; the default first step
• Assumes the outcome depends on X in the specified form (usually linear)
• Where treated and untreated units do not overlap, regression extrapolates from the functional form without warning
• With heterogeneous effects and a dummy for every value of X, OLS weights each covariate group by the variance of treatment within it, P(D=1 \mid X)\,[1 - P(D=1 \mid X)], which need not give the ATT or ATE. With other specifications the weights are harder to describe, and can be negative
• Only control for pre-treatment variables (Chapter 2: no mediators, no colliders)

Matching

Pair Each Treated Unit with Similar Untreated Units

Approach	Idea
Exact matching	Match on identical values of X (feasible with few, discrete covariates)
Coarsened exact matching	Bin continuous variables, then match exactly on the bins
Nearest-neighbour matching	Match each treated unit to the closest untreated unit(s) by a distance on X
Caliper	Discard matches that are farther apart than a set tolerance
With or without replacement	Reusing controls gives closer matches but higher variance

Matching makes overlap visible: treated units without good matches are dropped, and the estimand becomes the effect for the units that remain.

The Propensity Score

Collapsing Many Covariates into One Number

The propensity score is the probability of treatment given covariates: e(X_i) = P(D_i = 1 \mid X_i).

Key result: if treatment is as good as random given X, it is also as good as random given e(X) (Rosenbaum and Rubin 1983). So we can match or weight on a single score instead of many covariates.

Ways to use it:

• Matching treated and untreated units with similar scores
• Inverse probability weighting (IPW): weight untreated units by e(X) / (1 - e(X)) to estimate the ATT
• Doubly robust estimators combine a propensity model with an outcome regression; consistent if either model is right

Diagnostics: Overlap and Balance

Check, Then Estimate

• Overlap: plot the propensity score distributions for treated and untreated units; trim units outside the common range
• Balance: after matching or weighting, compare covariates using standardized mean differences (difference in means divided by the pooled standard deviation). A common rule of thumb is below 0.1
• Variance ratios near 1 for continuous covariates
• Do not judge balance with p-values: they shrink with sample size and say nothing about bias
• Balance on observables is necessary, not sufficient: it tells us nothing about unobservables

Simulation

Matching and Weighting in R

suppressMessages(library(MatchIt))
set.seed(2026)
n <- 4000

# Customers decide whether to join a retailer's loyalty app.
# Frequent and loyal customers are more likely to join, and visit more anyway.
past_visits <- rpois(n, 6)
loyalty     <- runif(n)
join   <- rbinom(n, 1, plogis(-3 + 0.3 * past_visits + 2 * loyalty))
visits <- 2 + 0.8 * past_visits + 3 * loyalty + 1.0 * join + rnorm(n)   # true effect = 1
dat <- data.frame(visits, join, past_visits, loyalty)

# Nearest-neighbour propensity score matching with a caliper
m  <- matchit(join ~ past_visits + loyalty, data = dat, method = "nearest",
              distance = "glm", caliper = 0.1)
md <- match.data(m)

# Inverse probability weighting for the ATT
ps <- glm(join ~ past_visits + loyalty, family = binomial, data = dat)$fitted
w  <- ifelse(dat$join == 1, 1, ps / (1 - ps))

round(c(true_effect = 1,
        naive       = mean(visits[join == 1]) - mean(visits[join == 0]),
        regression  = unname(coef(lm(visits ~ join + past_visits + loyalty, dat))["join"]),
        PS_matching = unname(coef(lm(visits ~ join, md, weights = weights))["join"]),
        IPW         = weighted.mean(visits[join == 1], w[join == 1]) -
                      weighted.mean(visits[join == 0], w[join == 0])), 2)

true_effect       naive  regression PS_matching         IPW 
       1.00        2.64        1.02        1.09        1.08 

Checking Balance

smd <- cbind(before = summary(m)$sum.all[, "Std. Mean Diff."],
             after  = summary(m)$sum.matched[, "Std. Mean Diff."])
round(smd, 2)

            before after
distance      0.83  0.05
past_visits   0.63  0.05
loyalty       0.47  0.01

• Before matching, joiners differ from non-joiners by about half a standard deviation or more on past visits and loyalty. After matching, the differences are below 0.1.

When the Assumption Fails

# Now an unobserved trait (enthusiasm) drives both joining and visiting
enthusiasm <- rnorm(n)
join2   <- rbinom(n, 1, plogis(-3 + 0.3 * past_visits + 2 * loyalty + 1.2 * enthusiasm))
visits2 <- 2 + 0.8 * past_visits + 3 * loyalty + 1.0 * join2 + 1.5 * enthusiasm + rnorm(n)
dat2 <- data.frame(visits = visits2, join = join2, past_visits, loyalty)

m2 <- matchit(join ~ past_visits + loyalty, data = dat2, method = "nearest",
              distance = "glm", caliper = 0.1)
c(true_effect = 1,
  PS_matching = round(unname(coef(lm(visits ~ join, match.data(m2), weights = weights))["join"]), 2))

true_effect PS_matching 
       1.00        2.54 

• Balance on past visits and loyalty can look perfect, and the estimate is still badly biased: matching cannot balance what it cannot see.

Real Data

Real Data: Can Matching Recover the Experiment?

data("lalonde", package = "MatchIt")   # 185 NSW trainees and 429 comparison men from a national survey (PSID)
f <- treat ~ age + educ + race + married + nodegree + re74 + re75

m_ll <- matchit(f, data = lalonde, method = "nearest", distance = "glm", caliper = 0.1, replace = TRUE)
md_ll <- match.data(m_ll)

round(c(experimental_benchmark = 1794,   # Dehejia & Wahba (1999), same 185 trainees
        naive      = with(lalonde, mean(re78[treat == 1]) - mean(re78[treat == 0])),
        regression = unname(coef(lm(update(f, re78 ~ treat + .), lalonde))["treat"]),
        PS_matching = unname(coef(lm(re78 ~ treat, md_ll, weights = weights))["treat"])))

experimental_benchmark                  naive             regression 
                  1794                   -635                   1548 
           PS_matching 
                  2039 

Reading the Result

The raw comparison says training lowered 1978 earnings by about $600, because the survey men earned far more to begin with. Controlling for earnings in 1974 and 1975 moves the estimate close to the experimental $1,794; matching does as well. The experiment is what tells us so: with observational data alone, we could not have known which estimate to trust (Dehejia and Wahba 1999).

• lalonde ships with MatchIt. Note the effective sample: 183 of 185 trainees are matched, but only a few dozen distinct comparison men are used, because matching is with replacement.

Sensitivity Analysis

How Strong Would a Hidden Confounder Have to Be?

Measuring Sensitivity to Hidden Confounding

• Rosenbaum bounds (Rosenbaum 2002): how much could an unobserved factor change the odds of treatment, between otherwise matched units, before the conclusion changes?
• Omitted variable bias benchmarks: how strongly would an unobserved confounder need to be related to treatment and outcome, compared with an observed covariate, to explain away the estimate? (Cinelli and Hazlett 2020)
• Coefficient stability: how much do estimates move as controls are added, relative to how much explanatory power they add? (Oster 2019)

A result that survives only a very weak hidden confounder should not be presented as causal.

Combining Matching with Other Designs

Matching Is Often Most Useful as a Helper

• Matching + difference-in-differences: match treated units to similar untreated units before treatment, then compare changes. Fixed differences on unobservables cancel out (Chapter 9); matching makes parallel trends more plausible
• Matching within an RDD bandwidth or an IV sample to improve balance and precision
• Synthetic control (Chapter 10) is itself a form of matching on the pre-treatment path of the outcome

Case Study 2 uses this combination.

Case Study 1: The LaLonde Benchmark

Case Study 1: Can Non-Experimental Methods Recover an Experiment?

LaLonde (1986), American Economic Review (LaLonde 1986)

Setting: the National Supported Work (NSW) demonstration, a US job-training program in the 1970s for disadvantaged workers (women receiving welfare benefits, and disadvantaged men), run as a randomized experiment.

The test: keep the treated group, but replace the randomized control group with comparison groups drawn from national surveys (the Panel Study of Income Dynamics and the Current Population Survey). Then apply standard non-experimental methods and compare their estimates with the experimental benchmark.

Case Study 1: What LaLonde Found

Non-Experimental Estimates Often Missed

• The survey comparison groups were very different from program participants. In 1979, women in the experiment’s control group earned about $3,800; women in the broadest survey comparison group earned about $8,000
• Many standard estimators (regressions with controls, before–after comparisons, two-step selection models) produced estimates far from the experimental result
• Specification tests did not reliably flag which estimates were wrong

The lesson: without a design, controlling for observables can leave large biases, and the data may not tell you so.

Case Study 1: The Debate That Followed

Can Propensity Scores Do Better?

• Dehejia and Wahba (1999) showed that propensity score methods, with careful attention to overlap (dropping comparison units unlike any participant) and to pre-program earnings, came much closer to the experimental benchmark
• Smith and Todd (2005) found that this success was sensitive to the sample and specification, and that difference-in-differences matching performed more reliably
• Synthesis: matching helps when you have the right covariates (especially lagged outcomes) and enforce overlap; it cannot rescue a comparison group drawn from a different population

Case Study 1: What to Copy

Practices to Reuse

• Benchmark against an experiment when you can. It is the only direct test of whether an observational method works in your setting.
• Draw comparison units from the same population, and drop those with no comparable treated unit.
• Condition on past outcomes (earnings before the program); demographics alone are not enough.

Case Study 1: Discussion Questions

1. Why does replacing the experimental control group with a national survey sample create such large differences in earnings?
2. Why are pre-program earnings so important as covariates in job-training evaluations?
3. What does “specification tests did not flag the problem” imply for applied work?
4. Why might matching on changes in earnings (DiD matching) be more robust than matching on levels?

Case Study 1: Suggested Answers (1–3)

1. The national surveys sample the wider population: older, more often employed, and with much higher earnings than the disadvantaged people the program enrolled. Those differences dwarf the program’s effect.
2. Past earnings are the strongest predictor of future earnings, and people often enter training after an earnings drop. Without earnings history, comparisons confuse this dip and recovery with the program’s effect.
3. Passing standard specification tests does not show an estimate is right. Applied work needs a design argument, benchmarks where possible, and sensitivity analysis.

Case Study 1: Suggested Answers (4)

4. Differencing removes fixed differences in earning ability that matching on observed characteristics misses, so only differences in earnings growth remain.

Case Study 2: Social Media and Customer Value

Case Study 2: Does Joining a Brand’s Social Media Page Change Customers?

Rishika, Kumar, Janakiraman & Bezawada (2013), Information Systems Research (Rishika et al. 2013)

Question: Do customers who join a retailer’s social media page (becoming a Facebook “fan”) visit more often and become more profitable?

Setting: a wine retailer in the northeastern United States that launched a Facebook page.

Data: transaction records for each customer from January 2008 to March 2011, linked to social media participation. Participants were identified through a customer survey (sent to 5,000 customers) and matched to the firm’s records of page members: 394 participants with enough data.

The selection problem: customers choose to join. Loyal, frequent buyers are more likely to join, and would visit often anyway.

Case Study 2: The Design

Propensity Score Matching, Then Difference-in-Differences

1. Propensity score: a logit model of joining, using 82 weeks of transaction and demographic data from before the page launched
2. Optimal pair matching: each participant paired with a similar non-participant; 287 matched pairs
3. Diagnostics: overlap of propensity scores (histograms and a Kolmogorov–Smirnov test); covariate balance before and after matching; Rosenbaum sensitivity analysis for hidden bias
4. Difference-in-differences on the matched sample, with customer fixed effects: compare the change in monthly visits after joining for participants with the change for their matched non-participants

• This is the “matching as a helper” idea: matching makes the comparison group similar on observables; the DiD and customer fixed effects then remove any remaining fixed differences.

Case Study 2: Results and Checks

Main Findings

• Joining the page raises visit frequency by about 5.2% relative to matched non-participants
• Profits per customer rise by a similar proportion (an estimated elasticity of about 5.6%)
• Effects are larger when the page is more active, and for customers who are more loyal, buy premium products, and are less deal-sensitive

Robustness

• Estimates are positive and significant when the post-period is restricted to 3, 6, 9, 12, or 15 months, and converge towards the full-period estimate
• Alternative matching methods give similar results
• Placebo launch dates within the 18 months before the real launch give estimates indistinguishable from zero

Case Study 2: Threats

What Matching Plus DiD Cannot Rule Out

• Time-varying unobservables: a customer whose enthusiasm for wine is rising may both join the page and start visiting more; DiD removes fixed differences, not changing ones
• Who is in the sample: participants were identified through a survey (about a quarter of the 5,000 customers surveyed responded), so the matched sample may not represent all page members
• Single retailer and category: effects for a wine retailer may differ elsewhere

Case Study 2: What to Copy

Practices to Reuse

• Draw controls from the same firm’s customers and match on long pre-period behaviour (82 weeks).
• Follow matching with difference-in-differences to remove remaining fixed differences.
• Add placebo launch dates and a Rosenbaum sensitivity analysis.

Case Study 2: Discussion Questions

1. Why match on 82 weeks of pre-launch behaviour rather than only on demographics?
2. What does the DiD step add after matching? What would remain if the authors had only compared matched customers after launch?
3. Describe a confounder that would survive both matching and DiD.
4. How would you design an experiment to answer the same question? What would it cost the retailer?

Case Study 2: Suggested Answers (1–3)

1. Past buying shows loyalty, interest in wine, and how often a customer visits, which drive both joining the page and future visits. Demographics capture these only weakly.
2. DiD removes any remaining differences that are constant over time. A comparison of matched customers after launch only would still include differences that matching did not balance, such as unobserved loyalty.
3. A change that starts around the time of joining, such as a new interest in wine, higher income, or moving closer to a store, could raise both joining and visits.

Case Study 2: Suggested Answers (4)

4. Randomly invite customers to join the page, for example by email with a small incentive, and use the invitation as an instrument for joining. Costs include the incentives, reaching fewer customers through the page during the test, and a large sample because take-up would be low.

Comparing the Two Case Studies

Two Lessons About Observables

What the Two Studies Show

	LaLonde (1986)	Rishika et al. (2013)
Role	Benchmark against an experiment	Applied study without an experiment
Comparison group	National survey samples	Matched customers of the same retailer
Key covariates	Demographics, some earnings history	82 weeks of transaction history
Design	Regression, difference-in-differences, and selection models	Propensity score matching + DiD
Lesson	Observables-only estimates can be far off	Same population, rich history, and DiD make matching more credible

Best Practices

Selection-on-Observables Checklist

Before and After Estimating

1. Explain how treatment was chosen and why the observed covariates capture it
2. Use only pre-treatment covariates, including past outcomes where possible
3. Check overlap and trim units without comparable counterparts; say how the estimand changes
4. Report balance with standardized differences, before and after
5. Compare estimators (regression, matching, weighting, doubly robust); large disagreements signal problems
6. Run a sensitivity analysis for unobserved confounding
7. Where possible, combine with a design that uses another source of variation (DiD, an instrument, a cutoff)

Common Pitfalls

Mistakes to Avoid

• Treating good balance on observables as evidence of no hidden bias
• Controlling for post-treatment variables (mediators) or colliders
• Ignoring lack of overlap and letting regression extrapolate
• Searching over covariates and matching settings until the estimate is significant
• Judging balance with p-values
• Presenting an observables-only estimate as causal without sensitivity analysis

Summary

• Regression, matching, and propensity scores share one assumption: no unmeasured confounders, plus overlap
• They differ in how they use covariates: functional form (regression), similar units (matching), a single score (propensity), or weights
• Check overlap and balance; quantify how strong hidden confounding would have to be
• The LaLonde (1986) benchmark is the standing warning: observables-only estimates can be badly wrong
• Matching is most credible as a helper to designs with another source of variation

Next Steps

1. Read: LaLonde (1986) Sections I–III; Rishika et al. (2013) Sections 3–6 and 8
2. Practice: In the simulation, drop loyalty from the matching model. How much bias appears, and does the balance table warn you?
3. Next Class: Putting it together: combining designs, accumulating evidence, and reporting (Chapter 12)

References

Cinelli, Carlos, and Chad Hazlett. 2020. “Making Sense of Sensitivity: Extending Omitted Variable Bias.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82 (1): 39–67. https://doi.org/10.1111/rssb.12348.

Dehejia, Rajeev H, and Sadek Wahba. 1999. “Causal Effects in Nonexperimental Studies: Reevaluating the Evaluation of Training Programs.” Journal of the American Statistical Association 94 (448): 1053–62.

LaLonde, Robert J. 1986. “Evaluating the Econometric Evaluations of Training Programs with Experimental Data.” American Economic Review 76 (4): 604–20.

Oster, Emily. 2019. “Unobservable Selection and Coefficient Stability: Theory and Evidence.” Journal of Business & Economic Statistics 37 (2): 187–204. https://doi.org/10.1080/07350015.2016.1227711.

Rishika, Rishika, Ashish Kumar, Ramkumar Janakiraman, and Ram Bezawada. 2013. “The Effect of Customers’ Social Media Participation on Customer Visit Frequency and Profitability: An Empirical Investigation.” Information Systems Research 24 (1): 108–27. https://doi.org/10.1287/isre.1120.0460.

Rosenbaum, Paul R. 2002. “Observational Studies.” Springer.

Rosenbaum, Paul R, and Donald B Rubin. 1983. “The Central Role of the Propensity Score in Observational Studies for Causal Effects.” Biometrika 70 (1): 41–55.

Smith, Jeffrey A, and Petra E Todd. 2005. “Does Matching Overcome LaLonde’s Critique of Nonexperimental Estimators?” Journal of Econometrics 125 (1–2): 305–53. https://doi.org/10.1016/j.jeconom.2004.04.011.