Adaptive Experimental Design
How precise an A/B test comes out depends on how the treatments are assigned. It depends just as much on who is put into the experiment at all. This topic covers designs that update both as the experiment runs, using the results already in hand, so that a limited sample still gives a precise estimate of the treatment effect. The main result is Active Adaptive Experimental Design for Treatment Effect Estimation with Covariate Choices (Kato, Oga, Komatsubara, and Inokuchi, 2024), accepted as an oral presentation at ICML 2024.
Problems addressed
Designing an experiment means settling a chain of choices: whom to include, which treatment to assign, how interim results feed into the next assignment, and how to estimate the effect from the sample you end up with. Research on adaptive experiments has mostly addressed one of these: the updating of the treatment assignment probabilities. We showed that optimizing the sample selection at the same time, meaning which covariate values the units in the experiment carry, pushes the theoretical bound on the variance of the average treatment effect estimator lower still. We then proposed an experimental design and an estimator that attain that bound.
The setting we have in mind is one where the experiment is expensive, the sample cannot be enlarged, or the schedule cannot be extended, and the conclusion nevertheless has to be sharper.
Core research
The Active Adaptive Experimental Design page sets out the problem, the design we propose, and the theoretical guarantees.
Research thread
- 2024
- Adaptive Experimental Design for Policy Learning. A design aimed at learning a policy, that is, whom to treat, instead of at estimating the effect.
- 2021
- Policy Choice and Best Arm Identification: Asymptotic Analysis of Exploration Sampling. Asymptotic analysis of choosing the best action once the experiment is over.
- 2020
- Efficient Adaptive Experimental Design for Average Treatment Effect Estimation. An early study that updates the treatment assignment sequentially to estimate the average treatment effect efficiently.
Relation to earlier work
The bound on how far the variance of an average treatment effect estimator can be pushed down was derived by Hahn (1998, On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects). Putting that bound to work in experimental design, by optimizing the treatment assignment probabilities adaptively, was established by Hahn, Hirano, and Karlan (2011, Adaptive Experimental Design Using the Propensity Score). Our work generalizes that line by taking the covariate distribution of the experimental units as a design variable alongside the assignment probabilities.
Where it applies
The number of experiments or the budget for testing an advertisement or a policy is often limited, and in medicine or manufacturing a single observation can be expensive. These designs get a firmer conclusion out of the same budget. For inquiries about experimental design or effect measurement, write to mkato-csecon@g.ecc.u-tokyo.ac.jp.