Experiments in Practice
From the Lab to the Firm
Where This Chapter Fits
Step 1: Can you randomly assign who is treated?
- You arrive here when: You can randomize, and now need to plan and run the experiment.
- This chapter: Where experiments happen, pre-registration, outcomes, mechanisms, and ethics.
- Next: Chapter 5: what to do when the experiment does not go to plan.
See the design flowchart in Chapter 1.
In Practice: Running an Experiment
Practitioner Summary
- Use it when: you have agreement to randomize and must decide where to run it, what to measure, and how to analyze it
- What you need: a primary outcome defined in advance, measured from records or systems rather than surveys where possible
- What must hold: the reported analysis is the one planned; participants and setting resemble the people and decisions you care about
- Checks to run: pre-registration filed before outcomes are seen; a check that the treatment was delivered as intended; quality as well as quantity of output
- In R:
power.t.test()andpower.prop.test()for sample sizes; the same estimators as Chapter 3 - Telling a manager: “We fixed the measure and the analysis before seeing any results, so this finding was not fished for.”
Where Experiments Happen
Lab, Online, Field
A Spectrum of Control and Realism
| Setting | Example | Strength | Weakness |
|---|---|---|---|
| Lab | Participants solve puzzles with or without an AI assistant in a university lab | Tight control; mechanisms can be isolated | Artificial tasks and stakes |
| Online | The same task on a survey platform such as Prolific | Cheap, fast, large samples | Attention and sample quality |
| Field, run by government | IRS letters randomized across 4.5 million tax returns (Chapter 3) | Real decisions, real outcomes | Limited control over what else happens |
| Field, run by firms | Companies randomize access to an AI coding tool | Real work at scale | Firms’ priorities shape the design |
The same question (does AI help people think and work?) can be studied at every point on this spectrum. Results that agree across settings are more convincing than any single study.
Experiments Inside Firms
Randomization as a Business Tool
- Digital firms run thousands of A/B tests a year on prices, interfaces, recommendations, and messages
- Firms also randomize internal changes: tools, training, incentives, schedules
- These experiments are run to make a decision, not to publish a paper:
- the control group is often treated once a decision is made
- treatment may be an offer or encouragement, not a mandate
- outcomes are whatever the firm already measures
These features shape what the experiment can identify (Chapter 5).
Planning an Experiment
Pre-Registration and Pre-Analysis Plans
Record the Plan Before Seeing the Data
A pre-registration records the hypotheses, design, primary outcomes, and planned analysis in a public registry (AEA RCT Registry, OSF, AsPredicted) before the data are analyzed.
Why: with many outcomes, subgroups, controls, and specifications, it is easy to find a significant result by chance and report only that one (the “garden of forking paths”).
What to include: primary and secondary outcomes; the estimator and controls; how to handle attrition and outliers; subgroups; multiple-testing corrections.
What it does not do: forbid exploration. It separates confirmatory results from exploratory ones, and deviations are reported rather than hidden.
Measuring Outcomes
The Outcome Defines What You Learn
- Administrative records (tax filings, death records, claims, code repositories) avoid attrition and self-report bias, but measure only what the system records
- Surveys capture attitudes and well-being, but suffer from non-response and reporting bias
- Behavioral measures (decisions, clicks, completed tasks) are objective, but may be proxies: “pull requests completed” is not the same as “useful software written”
- Quality as well as quantity: if a treatment raises output, check whether it lowers quality
- Decide the primary outcome in advance; everything else is secondary
Testing Mechanisms
Design Arms to Learn Why
- Vary features of the treatment: personalized vs. generic letters; early vs. late mailing (Chapter 3)
- Vary the context: add time pressure, or incentives for accuracy, and see whether the effect changes
- Randomize the treatment’s quality: make an AI assistant right on some questions and wrong on others to see how much people rely on it
- Measure intermediate outcomes (take-up, usage, acceptance of suggestions) to trace the path from treatment to outcome
Between- and Within-Subject Designs
Who Is Compared with Whom?
| Design | Comparison | Pros | Cons |
|---|---|---|---|
| Between-subject | Different people in treatment and control | No carry-over between conditions | Needs larger samples |
| Within-subject | The same person under different conditions (e.g., different trials) | Usually more precise; each person is their own control | Carry-over, learning, and demand effects |
Many modern experiments mix both: randomize access between people, and randomize features within people.
Ethics of Experimentation
Questions to Answer Before Randomizing
- Review and consent: ethics board approval; informed consent where feasible; debriefing after any deception
- Deception: an AI assistant secretly programmed to give wrong answers is deception, justified only if necessary, harmless, and followed by debriefing
- Withholding benefits: is it acceptable to deny the control group something valuable? Easier when supply is limited anyway, or when the benefit is genuinely uncertain
- Experiments on employees and customers: firms have power over the people they experiment on; outcomes such as performance reviews must not be harmed by assignment
- Privacy: linking records requires secure handling and minimal data
Case Study 1: Cognitive Surrender
Case Study 1: Do People Stop Thinking When AI Is Available?
Shaw & Nave (2026), SSRN working paper (Shaw and Nave 2026)
Status: a working paper on SSRN, not yet published in a journal at the time of writing. Included as a supplementary case because its design shows how to isolate a mechanism experimentally.
Question: When people can consult a generative AI assistant while reasoning, do they check its answers, or adopt them with little scrutiny (“cognitive surrender”)?
Task: seven reasoning puzzles adapted from the Cognitive Reflection Test, each with an intuitive wrong answer and a correct answer that requires deliberation.
Scale: three pre-registered experiments; 1,372 participants; 9,593 puzzle attempts; lab and online (Prolific) samples.
Case Study 1: The Design
Two Layers of Randomization
Between subjects (Study 1, N = 359): randomly assigned to Brain-Only (no AI; n = 121) or AI-Assisted (n = 238), oversampling the AI group 2:1.
Within subjects: for AI-Assisted participants, each puzzle was randomly set so the assistant gave either the correct answer or a confident intuitive error, using hidden instructions to the AI. Participants did not know which.
Moderators (Studies 2–3): a 30-second time limit per puzzle; incentives ($0.20 per correct answer plus a lottery) with item-level feedback.
Why this design works: whether to consult the AI is the participant’s choice, so consulters may differ from non-consulters. But the AI’s accuracy is randomized within person, so comparing accurate and faulty trials isolates how much people rely on what the AI says.
Case Study 1: Results
Main Findings (Study 1)
| Condition | Share of puzzles answered correctly |
|---|---|
| Brain-Only | 45.8% |
| AI-Assisted, AI correct | 71.0% |
| AI-Assisted, AI wrong | 31.5% |
- Participants consulted the AI on just over half the puzzles
- When they consulted it, they followed a wrong answer about 80% of the time
- AI access raised confidence by 11.7 points, even though half its answers were wrong
- Time pressure and incentives changed baseline accuracy but did not remove the pattern
- Reliance was stronger for people with more trust in AI and lower need for cognition
Case Study 1: Lessons and Limits
Internal and External Validity
- Internal validity: randomized access and randomized accuracy give clean causal comparisons
- Mechanism by design: the within-person accuracy manipulation isolates reliance on AI from the benefit of having a tool
- External validity: puzzles with known answers differ from real decisions; stakes and repeated use with feedback are missing (the authors list these limitations themselves)
- Ethics: deception (a deliberately faulty AI) required debriefing
- Status: results from an unpublished working paper should be treated as provisional
Case Study 1: What to Copy
Practices to Reuse
- Randomize the quality of the tool as well as access to it. Making the AI right on some puzzles and wrong on others isolates how much people rely on it.
- Keep a no-tool baseline. Only the Brain-Only group shows that a faulty AI leaves people worse than no AI (31.5% vs. 45.8% correct).
- Pre-register, then vary the context. Time pressure and incentives test whether the pattern survives conditions that should weaken it.
Case Study 1: Discussion Questions
- Why compare AI-correct and AI-wrong trials, rather than people who chose to consult the AI with people who did not?
- The Brain-Only group is smaller (2:1 allocation). Why might the researchers accept less precision for that comparison?
- What does a 31.5% accuracy rate with a faulty AI, below the 45.8% Brain-Only rate, tell a firm deploying AI tools?
- Design a field version of this experiment inside a firm. What would you randomize, and what would you measure?
Case Study 1: Suggested Answers (1–3)
- People choose whether to consult the AI, so consulters and non-consulters may differ in ability or motivation. Whether the AI is right is randomized within person, so the accurate–faulty contrast compares like with like.
- The central questions are within the AI group: how accuracy, time pressure, and incentives change reliance. Putting more participants there makes those comparisons more precise; the Brain-Only group is needed only as a baseline.
- A tool that is sometimes wrong can leave people worse off than no tool if they stop checking it. Firms should measure error rates, design checks into the workflow, and train staff to verify outputs, not just provide access.
Case Study 1: Suggested Answers (4)
- For example, randomize which employees get an AI assistant and, within users, randomly seed some suggestions with known errors in a safe test task. Measure accuracy, time, the share of errors caught, and confidence. Ethics review and debriefing are needed.
Case Study 2: Experiments Inside Firms
Case Study 2: Generative AI and Software Developers
Cui, Demirer, Jaffe, Musolff, Peng & Salz (2026), Management Science (Cui et al. 2026)
Question: Does an AI coding assistant (GitHub Copilot) make professional software developers more productive?
The opportunity: three companies randomized access to Copilot “in the ordinary course of business”, to decide whether to adopt it, and shared the data.
Outcomes: from the companies’ version-control systems (GitHub): completed pull requests (units of finished work), commits, builds, and build success.
Case Study 2: Three Experiments, Three Designs
The Same Question, Different Business Constraints
| Microsoft | Accenture | Anonymous Fortune 100 firm | |
|---|---|---|---|
| When | Sept 2022 – Apr 2023 | Jul – Dec 2023 | Sept – Oct 2023 |
| Developers | 1,746 (mainly US) | 320 (Southeast Asia) | 3,054 |
| Randomized | Individuals and teams | Individuals | Teams |
| Design | Treatment vs. control | Treatment vs. control | Staggered: random order of access over six weeks |
| Encouragement | Email, training, manager nudges | None |
Case Study 2: Results and Lessons
Main Findings
- Pooling the three experiments (4,867 developers), using Copilot raises completed pull requests by 26.1% (standard error 10.3%); commits +13.6% and builds +38.4%
- Each experiment on its own is noisy: many developer-weeks have zero pull requests
- Newer developers were more likely to adopt the tool (82% vs. 72% for longer-tenured developers) and gained more
Lessons for Business Experiments
- Firms’ decisions shape the design: encouragement rather than mandates; control groups that get access when the decision is made
- Pooling several imperfect experiments can deliver a precise answer that none provides alone
- Only about 60–70% of developers ever tried the tool, so the effect of access is much smaller than the effect of use. Chapter 5 shows how the authors handle this
Case Study 2: What to Copy
Practices to Reuse
- Turn a business rollout into an experiment. When a permanent control group is unacceptable, randomize the order of access.
- Use outcomes the systems already log. Pull requests, commits, and builds came straight from version control.
- Run several experiments and pool them. Each was noisy; pooled, they give +26.1% (SE 10.3%).
Case Study 2: Discussion Questions
- Why might a firm prefer a staggered rollout (everyone eventually treated, in random order) over a permanent control group?
- Microsoft randomized some developers by team. What spillovers within teams might that design be guarding against?
- “Pull requests completed” is the primary outcome. What could make it a misleading measure of productivity?
- The lab study (Case 1) finds people follow faulty AI answers; the field study finds productivity gains. How can both be true?
Case Study 2: Suggested Answers (1–3)
- Everyone gets the tool within weeks, which is easier to justify to managers and staff, while the random order still creates a valid comparison between early and late groups during the rollout.
- Developers in a team share code, review each other’s work, and divide tasks. A treated developer could pass on prompts or AI-written code, or take on different tasks, so untreated teammates would not be a clean control.
- Pull requests vary in size and difficulty. AI could encourage more, smaller requests, or lower quality that shows up later as bugs or rework. Counts also ignore design, review, and mentoring work.
Case Study 2: Suggested Answers (4)
- They measure different things in different settings. The lab study isolates reliance when the AI is wrong on reasoning puzzles; in real coding, suggestions are often right and errors can be caught by tests and review. Average gains can coexist with over-reliance on bad suggestions.
Summary
Summary
- Experiments range from the lab to the field; each setting trades control for realism
- Firms increasingly run experiments to make decisions, which shapes their designs
- Pre-registration separates confirmatory from exploratory findings
- Outcomes and arms should be chosen to answer the policy question and reveal mechanisms
- Ethics (consent, deception, withholding benefits, power over participants) is part of design, not an afterthought
- Field experiments find productivity gains from generative AI, concentrated among less experienced workers; lab experiments show people over-rely on AI output when it is wrong
Next Steps
References
