When Experiments Go Wrong

Non-Compliance, Attrition, and Spillovers

Fei Huang, UNSW Sydney

• Real experiments rarely go exactly to plan. People offered a treatment do not take it; people in the control group find a way to get it; outcomes go missing; treated units affect untreated ones.
• This chapter covers the three most common problems and what to do about each. The running case study is the set of AI coding-assistant experiments at Microsoft, Accenture, and a Fortune 100 firm, which ran into all three.
• Non-compliance leads naturally to instrumental variables, the topic of Chapter 6.

Where This Chapter Fits

Step 1, when it goes wrong

• You arrive here when: You randomized, but people did not comply, outcomes went missing, or treatment spilled over.
• This chapter: Intent-to-treat effects, scaling by take-up, attrition bounds, and designs for spillovers.
• Next: Chapter 6: a randomized offer is an instrument. The same logic works for shocks you did not create.

See the design flowchart in Chapter 1.

In Practice: When the Experiment Goes Wrong

Practitioner Summary

• Use it when: people did not take what they were offered, outcomes are missing, or the treatment leaked to the control group
• What you need: take-up in both arms, attrition by arm, and the team or cluster structure of the units
• What must hold: exclusion (the offer affects outcomes only through use); no defiers if controls can get the treatment; for Lee bounds, treatment changes whether outcomes are observed in one direction only
• Checks to run: take-up and attrition by arm; baseline traits of those who leave; control-group take-up over time
• In R: intent-to-treat feols(y ~ z, data), effect of use feols(y ~ 1 | d ~ z, data)
• Telling a manager: “Offering the tool raised output by X. Among the people who used it because we offered it, the gain was Y.”

Non-Compliance

Assigned Is Not the Same as Treated

Two Treatment Variables

• Z_i: assignment, which the researcher randomizes (offered the AI tool, sent a letter, invited to a program)
• D_i: receipt, which the unit chooses (actually used the tool, actually enrolled)

Experiment	Assignment Z	Receipt D	Take-up
IRS outreach (Chapter 3)	Letter sent	Enrolled in coverage	+1.1 percentage points (difference between arms)
AI coding assistant (this chapter)	Access offered	Developer used the tool	About 60–70% of those offered eventually tried it

When D \neq Z for some units, we have non-compliance.

Three Tempting Comparisons, Two of Them Wrong

Comparing by Receipt Breaks Randomization

Comparison	What it compares	Valid?
Intent-to-treat (ITT)	Everyone assigned vs. everyone not assigned, regardless of take-up	Yes: this is the randomized comparison
As-treated	Those who used the treatment vs. those who did not	No: who uses it is a choice
Naive per-protocol	Units that followed their assignment in each arm (takers if offered, non-takers if not)	No: drops a self-selected subset

Developers who choose to use an AI tool may be more motivated, more junior, or working on different tasks. Once we compare by choice, selection bias is back. The as-treated and naive per-protocol comparisons are really observational analyses inside an experiment: they could only be rescued by adjusting for everything that drives take-up (Hernán and Robins 2020).

Compliance Types

Four Kinds of Units

Each unit has two potential receipts: D_i(1) if assigned, D_i(0) if not.

Type	D_i(0)	D_i(1)	Example
Compliers	0	1	Use the tool if and only if offered
Never-takers	0	0	Never use it, even if offered
Always-takers	1	1	Find a way to use it even in the control group
Defiers	1	0	Use it only if not offered

We never observe a unit’s type directly, because we only see one of the two receipts.

One-Sided Non-Compliance

When the Control Group Cannot Get the Treatment

If control units have no access (D_i(0) = 0 for all), there are no always-takers or defiers. Then:

\underbrace{\text{ITT}_Y}_{\text{effect of assignment on } Y} = \underbrace{\text{ITT}_D}_{\text{share of compliers}} \times \underbrace{\text{CACE}}_{\text{effect for compliers}}

\text{CACE} = \frac{\text{ITT}_Y}{\text{ITT}_D} = \frac{E[Y \mid Z=1] - E[Y \mid Z=0]}{E[D \mid Z=1] - E[D \mid Z=0]}

Key assumption, exclusion: assignment affects outcomes only through receipt. An email offering an AI tool must not raise productivity by itself (for example, by signalling that management is watching).

Simulation: As-Treated vs. ITT vs. CACE

library(fixest)
set.seed(2026)
n <- 4000
motivation <- rnorm(n)                                  # unobserved
z <- rbinom(n, 1, 0.5)                                  # random offer
# Only offered developers can adopt; motivated ones adopt more often
d <- z * rbinom(n, 1, plogis(-0.5 + 1.2 * motivation))
y <- 5 + 2 * motivation + 1.5 * d + rnorm(n)            # true effect of use = 1.5

itt_y <- mean(y[z == 1]) - mean(y[z == 0])
itt_d <- mean(d[z == 1]) - mean(d[z == 0])

round(c(take_up       = itt_d,
        as_treated    = mean(y[d == 1]) - mean(y[d == 0]),
        ITT           = itt_y,
        CACE_wald     = itt_y / itt_d,
        CACE_2SLS     = unname(coef(feols(y ~ 1 | d ~ z, data = data.frame(y, d, z)))["fit_d"])), 2)

   take_up as_treated        ITT  CACE_wald  CACE_2SLS 
      0.39       2.87       0.58       1.47       1.47 

• As-treated is badly biased upwards: motivated developers both adopt more and produce more.
• The ITT is unbiased for the effect of the offer, but diluted by non-adopters.
• Dividing by take-up recovers the effect of use for compliers. Two-stage least squares gives the same number; Chapter 6 develops it in general.

Two-Sided Non-Compliance

When Some Controls Get Treated Anyway

If control units can obtain the treatment, always-takers exist. The same ratio still identifies the effect for compliers, now called the local average treatment effect (LATE), under one more assumption:

Monotonicity (no defiers): being offered the treatment never makes anyone less likely to take it.

\text{LATE} = \frac{E[Y \mid Z=1] - E[Y \mid Z=0]}{E[D \mid Z=1] - E[D \mid Z=0]}

• The denominator is now the difference in take-up between arms
• As control units gain access, the difference shrinks, the instrument weakens, and estimates become noisy
• This is an instrumental variables estimator, with random assignment as the instrument (Chapter 6)

Which Effect Does a Decision-Maker Need?

ITT or LATE?

• ITT: what happens if you offer the treatment without forcing it. Relevant for a manager deciding whether to buy licences for everyone (seat-based pricing), or a government deciding whether to send letters
• LATE / CACE: what happens to those who use it because it was offered. Relevant if you pay only for usage, or want to know whether the treatment itself works
• LATE is the effect for compliers only. Never-takers and always-takers may respond differently, so LATE need not equal the ATE

Attrition

Missing Outcomes

When Attrition Breaks Randomization

Attrition: outcomes are missing for some units (survey non-response, people moving, firms closing, employees leaving).

• If attrition is unrelated to treatment and potential outcomes, the remaining sample is still balanced, just smaller
• If treatment affects who stays (e.g., the treated group is easier to contact), the remaining treated and control units are no longer comparable
• Always check: attrition rates by arm, and baseline characteristics of those who leave vs. stay

Responses to Attrition

Prevention First, Then Bounds

Prevention: use administrative outcomes where possible (tax, health, payroll, code repositories); track participants; incentivize responses.

Bounds: when attrition differs by arm, bound the effect instead of assuming it away. Lee bounds trim the arm with less attrition so both arms have the same share of observed outcomes, once from the top and once from the bottom of the outcome distribution (Lee 2009). Assuming treatment affects whether an outcome is observed in only one direction, the effect for units that would be observed in either arm lies between the two.

Reporting: report attrition, and report experiments that failed. Selective reporting of only the experiments that “worked” is attrition at the level of the literature.

Spillovers

When Treatment Leaks Across Units

Interference in Practice

• Learning: a developer using an AI tool shares prompts or code with untreated teammates
• Task reallocation: treated workers finish faster and take over harder tasks from untreated colleagues
• Contamination: control units hear about the treatment and seek it out
• Markets: a subsidy for some buyers raises prices for everyone

Positive spillovers to the control group make the treated–control difference smaller than the true effect; negative spillovers (for example, treated workers taking easy tasks from untreated ones) make it larger. In markets they can change its meaning entirely.

Designing for Spillovers

Options

• Randomize clusters (teams, offices, villages) so spillovers stay within treatment or control (at a cost in power; Chapter 3)
• Separate treated and control units in space or time
• Measure spillovers deliberately: randomize the share of each cluster that is treated, and compare untreated units in highly treated clusters with untreated units in untreated clusters
• Check for contamination: does the control group’s take-up rise over time?

Other Threats Worth Naming

Beyond the Big Three

• Hawthorne effects: being studied changes behaviour
• Demand effects: participants guess the hypothesis and act on it (a concern in lab studies)
• Peeking and early stopping: checking results repeatedly and stopping when significant inflates false positives (common in A/B tests)
• Scale-up: effects in a pilot may not hold when everyone is treated, because markets and organizations adjust

Case Study: AI Experiments in Three Firms

Case Study: Low Take-Up, Contamination, and Attrition

Cui, Demirer, Jaffe, Musolff, Peng & Salz (2026), Management Science (Cui et al. 2026)

Three randomized experiments offering GitHub Copilot to 4,867 developers at Microsoft, Accenture, and a Fortune 100 firm (design details in Chapter 4). The paper is a good guide to analyzing and reporting an imperfect experiment.

The Complications

• Low take-up: in the first two weeks at Microsoft only 8.5% of the treated group signed up; two reminder emails raised it to 42.5%
• Never-takers: across experiments, 30–40% of developers never tried the tool, despite low adoption costs
• Always-takers: a few Microsoft control developers (0.5%) needed access for their work
• Contamination and early end: the Microsoft experiment ended earlier than planned as control developers sought access
• Attrition: a second Accenture experiment was abandoned after a layoff affected 42% of participants, leaving no reliable data on who used the tool
• Noisy outcomes: most developer-weeks have zero completed pull requests, so each experiment has low power

Case Study: How the Authors Responded

Analyzing an Imperfect Experiment

• Instrumental variables: random assignment instruments for adoption, giving a LATE for developers who adopted because they were offered access
• Weighted IV: once control groups gain access, the adoption gap shrinks and the instrument weakens. The authors weight each period by the adoption difference between arms, so periods with more experimental variation count more
• Fixed effects: developer and week fixed effects absorb stable productivity differences and calendar shocks, improving precision
• Pooling: a precision-weighted average across the three experiments: +26.1% completed pull requests (SE 10.3%)
• Reporting the failure: the abandoned Accenture experiment is reported in an appendix; including it, with conservatively imputed adoption, changes the pooled estimate to +21.3%
• ITT vs. LATE: the ITT is much smaller than the LATE because many developers never adopted. Which matters depends on whether the firm pays per licence or per use

Case Study: What to Copy

Practices to Reuse

• Report the ITT and the LATE side by side, and say which one the decision needs (paying per licence vs. per use).
• Down-weight periods with little experimental variation. Once controls gain access, weighting by the adoption gap keeps weak periods from adding noise.
• Report the experiment that failed, and show how including it changes the answer (+26.1% to +21.3%).

Case Study: Discussion Questions

1. Why would comparing developers who used Copilot with those who did not give a misleading answer?
2. Reminder emails raised take-up from 8.5% to 42.5%. How does that change the ITT, and the precision of the LATE?
3. What does the exclusion restriction require here? Could the invitation email itself change productivity?
4. Who are the compliers in this experiment? Would you expect their effect to be larger or smaller than for never-takers?
5. Why is reporting the abandoned experiment important, even though it is noisy?

Case Study: Suggested Answers (1–2)

1. Developers choose whether to use the tool. Those who do may be more motivated, more junior, or working on tasks where it helps, so the comparison mixes the tool’s effect with these differences.
2. With one-sided non-compliance, the ITT equals the take-up gap times the effect for compliers, so higher take-up makes the ITT larger. The LATE becomes more precise, because its standard error is roughly the ITT’s standard error divided by the take-up gap.

Case Study: Suggested Answers (3–5)

3. The invitation must change output only through using Copilot. An email that signals management attention or prompts developers to work harder would violate exclusion and inflate the estimated effect of use.
4. Compliers adopt when offered, often after reminders. Never-takers may expect little benefit, for example because their tasks suit the tool less, so their effect could well be smaller; the data cannot confirm this.
5. Leaving out experiments that went badly biases the evidence, just as attrition biases a single study. Reporting it shows how much the conclusion depends on the experiments that did work (+26.1% vs. +21.3%).

Summary

Summary

• Non-compliance: analyze by assignment (ITT); scale by take-up to get the effect for compliers (CACE/LATE); never compare by receipt
• One-sided non-compliance needs exclusion; two-sided also needs monotonicity, and the estimator becomes IV
• Attrition: prevent it, check it by arm, and bound it when it differs
• Spillovers: randomize at the right level, and measure contamination
• Imperfect experiments can still be informative, if analyzed carefully and reported in full

Next Steps

1. Read: Cui et al. (2026) Sections 3–4
2. Practice: In the simulation, let 10% of the control group adopt (always-takers). Do the Wald and 2SLS estimates still recover 1.5?
3. Next Class: Instrumental variables beyond experiments (Chapter 6)

References

Cui, Zheyuan Kevin, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. 2026. “The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers.” Management Science, ahead of print. https://doi.org/10.1287/mnsc.2025.00535.

Hernán, Miguel A., and James M. Robins. 2020. Causal Inference: What If. Chapman & Hall/CRC. https://miguelhernan.org/whatifbook.

Lee, David S. 2009. “Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects.” Review of Economic Studies 76 (3): 1071–102. https://doi.org/10.1111/j.1467-937X.2009.00536.x.