Putting It Together

Combining Designs, Accumulating Evidence, and Reporting

Instructor

Fei Huang, UNSW Sydney

  • The final chapter steps back from individual methods. Strong evaluations rarely rely on a single design; strong evidence rarely comes from a single study.
  • We look at how the case studies in these notes combined designs, how to combine estimates across studies, what can go wrong at the level of the literature, and how to report an evaluation so that decision-makers can use it.
  • The generative AI studies from across the notes serve as a worked example of evidence synthesis.

Where This Chapter Fits

NoteAfter the flowchart
  • You arrive here when: You have a main design, and want to know how much to trust it and how to report it.
  • This chapter: Combining designs, pooling evidence across studies, external validity, and reporting.
  • Next: Your own evaluation.

See the design flowchart in Chapter 1.

Choosing and Combining Designs

Where Each Method Finds Its Counterfactual

NoteA Recap of the Toolkit
Method The counterfactual comes from… Key assumption Chapter
Randomized experiment A lottery Random assignment; no spillovers 3–5
Instrumental variables An outside shock that shifts treatment Relevance; independence; exclusion; monotonicity 6
Regression discontinuity Units just across a cutoff Continuity at the cutoff 7
Panel fixed effects The same unit at other times No time-varying confounders (strict exogeneity) 8
Difference-in-differences The change in an untreated group Parallel trends 9
Synthetic control A weighted mix of untreated units Pre-treatment fit carries over 10
Regression, matching Similar units, by observed characteristics No unmeasured confounders 11

The order follows the design flowchart in Chapter 1.

The Best Papers Combine Designs

TipTriangulation in the Case Studies
Paper Main design Backed up by
Goldin et al. (2021, see Ch. 3) Randomized letters IV to scale to the effect of coverage
Cui et al. (2026, see Ch. 4–5) Three field experiments IV for take-up; fixed effects; pooling across experiments
Künn et al. (2023, see Ch. 8) Player fixed effects IV using wind-driven outdoor pollution; replication in another league
Brynjolfsson et al. (2025, see Ch. 8–9) Staggered DiD Team-level IV; a small randomized pilot; robust estimators
Miller et al. (2021, see Ch. 9) Event-study DiD Placebo groups; power analysis; Wald-style scaling
Demirci et al. (2025, see Ch. 9) DiD on a common shock Synthetic DiD; dose-response with search data; placebo dates

Each design rests on a different assumption. When designs with different weaknesses agree, the conclusion is much harder to dismiss.

Experimental and Observational Estimates

WarningChecking Designs Against Each Other
  • LaLonde’s benchmark: standard non-experimental estimates of a job-training program often missed the experimental answer (LaLonde 1986)
  • Within one paper: Cui et al. (2026) report both DiD and experimental (IV) estimates for each firm. The DiD estimates are sometimes larger and sometimes smaller: the direction of selection differs across firms
  • Across papers: an AI assistant raised support-agent productivity by about 15% in a staggered rollout (Brynjolfsson et al. 2025), and a coding assistant raised completed tasks by about 26% in randomized experiments (Cui et al. 2026). Different settings and estimands, but the same direction and similar size

Lesson: observational designs are most credible when their assumptions can be checked against an experiment, or against a design with different assumptions.

Accumulating Evidence

Combining Estimates Across Studies

NotePrecision Weighting

Given estimates \hat\tau_1, \dots, \hat\tau_K with standard errors s_1, \dots, s_K of the same estimand:

\hat\tau_{\text{pooled}} = \frac{\sum_k w_k \hat\tau_k}{\sum_k w_k}, \qquad w_k = \frac{1}{s_k^2}, \qquad SE(\hat\tau_{\text{pooled}}) = \frac{1}{\sqrt{\sum_k w_k}}

  • Precise studies get more weight
  • This is how Cui et al. (2026) combine their three noisy experiments into one informative estimate
  • If true effects differ across settings, a random-effects model adds a between-study variance \tau^2, which widens the pooled interval and shifts weight towards smaller studies

Pooling in R

Code 
# Three hypothetical experiments estimating the same effect (in %)
est <- c(firm_A = 22, firm_B = 41, firm_C = 18)
se  <- c(firm_A = 12, firm_B = 25, firm_C = 15)

w <- 1 / se^2
pooled    <- sum(w * est) / sum(w)
pooled_se <- 1 / sqrt(sum(w))

# Test for heterogeneity across studies (Cochran's Q)
Q <- sum(w * (est - pooled)^2)

round(c(pooled = pooled, se = pooled_se,
        lower_95 = pooled - 1.96 * pooled_se, upper_95 = pooled + 1.96 * pooled_se,
        Q = Q, p_heterogeneity = pchisq(Q, df = length(est) - 1, lower.tail = FALSE)), 2)
         pooled              se        lower_95        upper_95               Q 
          22.97            8.77            5.77           40.17            0.64 
p_heterogeneity 
           0.73
  • None of the three hypothetical experiments is significant on its own at the 5% level, but the pooled estimate is.
  • The heterogeneity test has little power with three studies; a non-significant Q does not prove effects are identical.

What Goes Wrong at the Level of a Literature

WarningSelective Reporting and Publication Bias
  • If studies with significant results are more likely to be published, the published literature overstates effects, even if every individual study is unbiased
  • Small studies that happen to find large effects are especially likely to be published
  • Symptoms: many estimates just past the significance threshold; small studies reporting larger effects (an asymmetric “funnel plot”)

Remedies: pre-registration (Chapter 4); registered reports; publishing null results; replication packages with data and code; reporting failed experiments (Cui et al. (2026) report the abandoned Accenture experiment).

From Evidence to Policy

External Validity

NoteWill It Work Here?

Questions to ask before applying a result elsewhere:

  • Population: are the people or firms similar? Compliers in an IV or experiment may differ from the target population
  • Setting: institutions, prices, and alternatives differ across countries and sectors
  • Scale: effects in a pilot may change when everyone is treated (market prices adjust; supply of trainers runs out; others respond)
  • Time: technologies change. An AI tool from 2021 is not the tool of 2026
  • Mechanism: if we know why something works, we can judge whether that reason holds elsewhere

Synthesis Example: Generative AI at Work

TipWhat Four Studies, Four Designs, Say Together
Study Design Finding
Cui et al. (2026) Field experiments (Ch. 4–5) Coding assistant raises completed tasks ~26%; larger gains and adoption for newer developers
Brynjolfsson et al. (2025) Staggered rollout, FE/DiD (Ch. 8–9) Support productivity +15%; +30% for less skilled and less experienced agents; small gains for the most experienced
Demirci et al. (2025) DiD on ChatGPT’s release (Ch. 9) Freelance job posts in writing and coding fall ~21% relative to manual work
Shaw and Nave (2026) (preprint) Lab and online experiments (Ch. 4) When they consult the AI, people follow its wrong answers ~80% of the time (Study 1)

Consistent pattern: within firms, AI complements less experienced workers; across markets, it substitutes for some outsourced tasks; and people over-rely on its output. A decision-maker needs all three.

Reporting an Evaluation

TipA Checklist for the Write-Up
  1. Question and estimand: the effect of what, on what, for whom
  2. Design: where the counterfactual comes from, stated in plain language
  3. Assumptions and the evidence for them (balance, first stage, pre-trends, placebo tests)
  4. Main estimate with uncertainty, in units a decision-maker understands
  5. Robustness to alternative specifications and estimators
  6. Heterogeneity: who gains and who loses
  7. Limitations: what the design cannot rule out; external validity
  8. Costs and benefits where possible
  9. Replication materials: data access, code, pre-registration

Communicating to Decision-Makers

NoteTranslate Estimates into Decisions
  • Use meaningful units: “one fewer death for every 1,587 people sent a letter” (Goldin et al. 2021); “about 15,600 deaths averted had all states expanded Medicaid” (Miller et al. 2021)
  • Report uncertainty: a range as well as a point estimate
  • Say whose effect it is (compliers, treated units, units at a cutoff)
  • Separate what the evidence shows from what you recommend
  • Be explicit about what is not known

Summary

Summary

  • Every method in these notes finds a counterfactual in a different place, and rests on a different assumption
  • Strong studies combine designs whose weaknesses differ; strong conclusions rest on several studies
  • Pool comparable estimates with precision weights; check for heterogeneity
  • Guard against publication bias with pre-registration, replication, and reporting null and failed results
  • External validity needs as much attention as internal validity
  • Report evaluations so that a decision-maker can see the estimand, the assumptions, the uncertainty, and the limits

Final Exercise

  1. Choose a policy question you care about (insurance, finance, technology, operations)
  2. Identify the estimand and the assignment mechanism
  3. Propose a primary design and a second design with different assumptions
  4. Write the six questions from Chapter 1 for your own proposal
  5. Draft a one-page summary for a decision-maker

References

Brynjolfsson, Erik, Danielle Li, and Lindsey Raymond. 2025. “Generative AI at Work.” The Quarterly Journal of Economics 140 (2): 889–942. https://doi.org/10.1093/qje/qjae044.
Cui, Zheyuan Kevin, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. 2026. “The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers.” Management Science, ahead of print. https://doi.org/10.1287/mnsc.2025.00535.
Demirci, Ozge, Jonas Hannane, and Xinrong Zhu. 2025. “Who Is AI Replacing? The Impact of Generative AI on Online Freelancing Platforms.” Management Science 71 (10): 8097–108. https://doi.org/10.1287/mnsc.2024.05420.
Goldin, Jacob, Ithai Z Lurie, and Janet McCubbin. 2021. “Health Insurance and Mortality: Experimental Evidence from Taxpayer Outreach.” The Quarterly Journal of Economics 136 (1): 1–49. https://doi.org/10.1093/qje/qjaa029.
Künn, Steffen, Juan Palacios, and Nico Pestel. 2023. “Indoor Air Quality and Strategic Decision Making.” Management Science 69 (9): 5354–77. https://doi.org/10.1287/mnsc.2022.4643.
LaLonde, Robert J. 1986. “Evaluating the Econometric Evaluations of Training Programs with Experimental Data.” American Economic Review 76 (4): 604–20.
Miller, Sarah, Norman Johnson, and Laura R Wherry. 2021. “Medicaid and Mortality: New Evidence from Linked Survey and Administrative Data.” The Quarterly Journal of Economics 136 (3): 1783–829. https://doi.org/10.1093/qje/qjab004.
Shaw, Steven D, and Gideon Nave. 2026. Thinking—Fast, Slow, and Artificial: How AI Is Reshaping Human Reasoning and the Rise of Cognitive Surrender. SSRN Working Paper 6097646. https://doi.org/10.2139/ssrn.6097646.