Experiments in Practice

From the Lab to the Firm

Instructor

Fei Huang, UNSW Sydney

  • Chapter 3 covered the statistics of experiments. This chapter covers the decisions that come before and around them: where to run the experiment, what to measure, how to commit to an analysis in advance, and how to do it ethically.
  • Two case studies on generative AI: a set of pre-registered lab and online experiments on “cognitive surrender” to AI, and field experiments run inside three large firms.

Where This Chapter Fits

NoteStep 1: Can you randomly assign who is treated?
  • You arrive here when: You can randomize, and now need to plan and run the experiment.
  • This chapter: Where experiments happen, pre-registration, outcomes, mechanisms, and ethics.
  • Next: Chapter 5: what to do when the experiment does not go to plan.

See the design flowchart in Chapter 1.

In Practice: Running an Experiment

TipPractitioner Summary
  • Use it when: you have agreement to randomize and must decide where to run it, what to measure, and how to analyze it
  • What you need: a primary outcome defined in advance, measured from records or systems rather than surveys where possible
  • What must hold: the reported analysis is the one planned; participants and setting resemble the people and decisions you care about
  • Checks to run: pre-registration filed before outcomes are seen; a check that the treatment was delivered as intended; quality as well as quantity of output
  • In R: power.t.test() and power.prop.test() for sample sizes; the same estimators as Chapter 3
  • Telling a manager: “We fixed the measure and the analysis before seeing any results, so this finding was not fished for.”

Where Experiments Happen

Lab, Online, Field

NoteA Spectrum of Control and Realism
Setting Example Strength Weakness
Lab Participants solve puzzles with or without an AI assistant in a university lab Tight control; mechanisms can be isolated Artificial tasks and stakes
Online The same task on a survey platform such as Prolific Cheap, fast, large samples Attention and sample quality
Field, run by government IRS letters randomized across 4.5 million tax returns (Chapter 3) Real decisions, real outcomes Limited control over what else happens
Field, run by firms Companies randomize access to an AI coding tool Real work at scale Firms’ priorities shape the design

The same question (does AI help people think and work?) can be studied at every point on this spectrum. Results that agree across settings are more convincing than any single study.

Experiments Inside Firms

TipRandomization as a Business Tool
  • Digital firms run thousands of A/B tests a year on prices, interfaces, recommendations, and messages
  • Firms also randomize internal changes: tools, training, incentives, schedules
  • These experiments are run to make a decision, not to publish a paper:
    • the control group is often treated once a decision is made
    • treatment may be an offer or encouragement, not a mandate
    • outcomes are whatever the firm already measures

These features shape what the experiment can identify (Chapter 5).

Planning an Experiment

Pre-Registration and Pre-Analysis Plans

NoteRecord the Plan Before Seeing the Data

A pre-registration records the hypotheses, design, primary outcomes, and planned analysis in a public registry (AEA RCT Registry, OSF, AsPredicted) before the data are analyzed.

Why: with many outcomes, subgroups, controls, and specifications, it is easy to find a significant result by chance and report only that one (the “garden of forking paths”).

What to include: primary and secondary outcomes; the estimator and controls; how to handle attrition and outliers; subgroups; multiple-testing corrections.

What it does not do: forbid exploration. It separates confirmatory results from exploratory ones, and deviations are reported rather than hidden.

Measuring Outcomes

WarningThe Outcome Defines What You Learn
  • Administrative records (tax filings, death records, claims, code repositories) avoid attrition and self-report bias, but measure only what the system records
  • Surveys capture attitudes and well-being, but suffer from non-response and reporting bias
  • Behavioral measures (decisions, clicks, completed tasks) are objective, but may be proxies: “pull requests completed” is not the same as “useful software written”
  • Quality as well as quantity: if a treatment raises output, check whether it lowers quality
  • Decide the primary outcome in advance; everything else is secondary

Testing Mechanisms

TipDesign Arms to Learn Why
  • Vary features of the treatment: personalized vs. generic letters; early vs. late mailing (Chapter 3)
  • Vary the context: add time pressure, or incentives for accuracy, and see whether the effect changes
  • Randomize the treatment’s quality: make an AI assistant right on some questions and wrong on others to see how much people rely on it
  • Measure intermediate outcomes (take-up, usage, acceptance of suggestions) to trace the path from treatment to outcome

Between- and Within-Subject Designs

NoteWho Is Compared with Whom?
Design Comparison Pros Cons
Between-subject Different people in treatment and control No carry-over between conditions Needs larger samples
Within-subject The same person under different conditions (e.g., different trials) Usually more precise; each person is their own control Carry-over, learning, and demand effects

Many modern experiments mix both: randomize access between people, and randomize features within people.

Ethics of Experimentation

WarningQuestions to Answer Before Randomizing
  • Review and consent: ethics board approval; informed consent where feasible; debriefing after any deception
  • Deception: an AI assistant secretly programmed to give wrong answers is deception, justified only if necessary, harmless, and followed by debriefing
  • Withholding benefits: is it acceptable to deny the control group something valuable? Easier when supply is limited anyway, or when the benefit is genuinely uncertain
  • Experiments on employees and customers: firms have power over the people they experiment on; outcomes such as performance reviews must not be harmed by assignment
  • Privacy: linking records requires secure handling and minimal data

Case Study 1: Cognitive Surrender

Case Study 1: Do People Stop Thinking When AI Is Available?

NoteShaw & Nave (2026), SSRN working paper (Shaw and Nave 2026)

Status: a working paper on SSRN, not yet published in a journal at the time of writing. Included as a supplementary case because its design shows how to isolate a mechanism experimentally.

Question: When people can consult a generative AI assistant while reasoning, do they check its answers, or adopt them with little scrutiny (“cognitive surrender”)?

Task: seven reasoning puzzles adapted from the Cognitive Reflection Test, each with an intuitive wrong answer and a correct answer that requires deliberation.

Scale: three pre-registered experiments; 1,372 participants; 9,593 puzzle attempts; lab and online (Prolific) samples.

Case Study 1: The Design

TipTwo Layers of Randomization

Between subjects (Study 1, N = 359): randomly assigned to Brain-Only (no AI; n = 121) or AI-Assisted (n = 238), oversampling the AI group 2:1.

Within subjects: for AI-Assisted participants, each puzzle was randomly set so the assistant gave either the correct answer or a confident intuitive error, using hidden instructions to the AI. Participants did not know which.

Moderators (Studies 2–3): a 30-second time limit per puzzle; incentives ($0.20 per correct answer plus a lottery) with item-level feedback.

Why this design works: whether to consult the AI is the participant’s choice, so consulters may differ from non-consulters. But the AI’s accuracy is randomized within person, so comparing accurate and faulty trials isolates how much people rely on what the AI says.

Case Study 1: Results

TipMain Findings (Study 1)
Condition Share of puzzles answered correctly
Brain-Only 45.8%
AI-Assisted, AI correct 71.0%
AI-Assisted, AI wrong 31.5%
  • Participants consulted the AI on just over half the puzzles
  • When they consulted it, they followed a wrong answer about 80% of the time
  • AI access raised confidence by 11.7 points, even though half its answers were wrong
  • Time pressure and incentives changed baseline accuracy but did not remove the pattern
  • Reliance was stronger for people with more trust in AI and lower need for cognition

Case Study 1: Lessons and Limits

WarningInternal and External Validity
  • Internal validity: randomized access and randomized accuracy give clean causal comparisons
  • Mechanism by design: the within-person accuracy manipulation isolates reliance on AI from the benefit of having a tool
  • External validity: puzzles with known answers differ from real decisions; stakes and repeated use with feedback are missing (the authors list these limitations themselves)
  • Ethics: deception (a deliberately faulty AI) required debriefing
  • Status: results from an unpublished working paper should be treated as provisional

Case Study 1: What to Copy

TipPractices to Reuse
  • Randomize the quality of the tool as well as access to it. Making the AI right on some puzzles and wrong on others isolates how much people rely on it.
  • Keep a no-tool baseline. Only the Brain-Only group shows that a faulty AI leaves people worse than no AI (31.5% vs. 45.8% correct).
  • Pre-register, then vary the context. Time pressure and incentives test whether the pattern survives conditions that should weaken it.

Case Study 1: Discussion Questions

  1. Why compare AI-correct and AI-wrong trials, rather than people who chose to consult the AI with people who did not?
  2. The Brain-Only group is smaller (2:1 allocation). Why might the researchers accept less precision for that comparison?
  3. What does a 31.5% accuracy rate with a faulty AI, below the 45.8% Brain-Only rate, tell a firm deploying AI tools?
  4. Design a field version of this experiment inside a firm. What would you randomize, and what would you measure?
  1. People choose whether to consult the AI, so consulters and non-consulters may differ in ability or motivation. Whether the AI is right is randomized within person, so the accurate–faulty contrast compares like with like.
  2. The central questions are within the AI group: how accuracy, time pressure, and incentives change reliance. Putting more participants there makes those comparisons more precise; the Brain-Only group is needed only as a baseline.
  3. A tool that is sometimes wrong can leave people worse off than no tool if they stop checking it. Firms should measure error rates, design checks into the workflow, and train staff to verify outputs, not just provide access.
  4. For example, randomize which employees get an AI assistant and, within users, randomly seed some suggestions with known errors in a safe test task. Measure accuracy, time, the share of errors caught, and confidence. Ethics review and debriefing are needed.

Case Study 2: Experiments Inside Firms

Case Study 2: Generative AI and Software Developers

NoteCui, Demirer, Jaffe, Musolff, Peng & Salz (2026), Management Science (Cui et al. 2026)

Question: Does an AI coding assistant (GitHub Copilot) make professional software developers more productive?

The opportunity: three companies randomized access to Copilot “in the ordinary course of business”, to decide whether to adopt it, and shared the data.

Outcomes: from the companies’ version-control systems (GitHub): completed pull requests (units of finished work), commits, builds, and build success.

Case Study 2: Three Experiments, Three Designs

TipThe Same Question, Different Business Constraints
Microsoft Accenture Anonymous Fortune 100 firm
When Sept 2022 – Apr 2023 Jul – Dec 2023 Sept – Oct 2023
Developers 1,746 (mainly US) 320 (Southeast Asia) 3,054
Randomized Individuals and teams Individuals Teams
Design Treatment vs. control Treatment vs. control Staggered: random order of access over six weeks
Encouragement Email Email, training, manager nudges None
  • The designs differ across firms: Microsoft mixed individual and team randomization; at the anonymous firm every team was invited within about six weeks, and only the order of invitation was randomized.

Case Study 2: Results and Lessons

TipMain Findings
  • Pooling the three experiments (4,867 developers), using Copilot raises completed pull requests by 26.1% (standard error 10.3%); commits +13.6% and builds +38.4%
  • Each experiment on its own is noisy: many developer-weeks have zero pull requests
  • Newer developers were more likely to adopt the tool (82% vs. 72% for longer-tenured developers) and gained more
WarningLessons for Business Experiments
  • Firms’ decisions shape the design: encouragement rather than mandates; control groups that get access when the decision is made
  • Pooling several imperfect experiments can deliver a precise answer that none provides alone
  • Only about 60–70% of developers ever tried the tool, so the effect of access is much smaller than the effect of use. Chapter 5 shows how the authors handle this

Case Study 2: What to Copy

TipPractices to Reuse
  • Turn a business rollout into an experiment. When a permanent control group is unacceptable, randomize the order of access.
  • Use outcomes the systems already log. Pull requests, commits, and builds came straight from version control.
  • Run several experiments and pool them. Each was noisy; pooled, they give +26.1% (SE 10.3%).

Case Study 2: Discussion Questions

  1. Why might a firm prefer a staggered rollout (everyone eventually treated, in random order) over a permanent control group?
  2. Microsoft randomized some developers by team. What spillovers within teams might that design be guarding against?
  3. “Pull requests completed” is the primary outcome. What could make it a misleading measure of productivity?
  4. The lab study (Case 1) finds people follow faulty AI answers; the field study finds productivity gains. How can both be true?
  1. Everyone gets the tool within weeks, which is easier to justify to managers and staff, while the random order still creates a valid comparison between early and late groups during the rollout.
  2. Developers in a team share code, review each other’s work, and divide tasks. A treated developer could pass on prompts or AI-written code, or take on different tasks, so untreated teammates would not be a clean control.
  3. Pull requests vary in size and difficulty. AI could encourage more, smaller requests, or lower quality that shows up later as bugs or rework. Counts also ignore design, review, and mentoring work.
  4. They measure different things in different settings. The lab study isolates reliance when the AI is wrong on reasoning puzzles; in real coding, suggestions are often right and errors can be caught by tests and review. Average gains can coexist with over-reliance on bad suggestions.

Summary

Summary

  • Experiments range from the lab to the field; each setting trades control for realism
  • Firms increasingly run experiments to make decisions, which shapes their designs
  • Pre-registration separates confirmatory from exploratory findings
  • Outcomes and arms should be chosen to answer the policy question and reveal mechanisms
  • Ethics (consent, deception, withholding benefits, power over participants) is part of design, not an afterthought
  • Field experiments find productivity gains from generative AI, concentrated among less experienced workers; lab experiments show people over-rely on AI output when it is wrong

Next Steps

  1. Read: Cui et al. (2026) Sections 1–3; Shaw and Nave (2026) (Study 1)
  2. Practice: Write a one-page pre-analysis plan for an experiment giving an AI assistant to half of a company’s analysts
  3. Next Class: When experiments go wrong: non-compliance, attrition, and spillovers (Chapter 5)

References

Cui, Zheyuan Kevin, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. 2026. “The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers.” Management Science, ahead of print. https://doi.org/10.1287/mnsc.2025.00535.
Shaw, Steven D, and Gideon Nave. 2026. Thinking—Fast, Slow, and Artificial: How AI Is Reshaping Human Reasoning and the Rise of Cognitive Surrender. SSRN Working Paper 6097646. https://doi.org/10.2139/ssrn.6097646.