Now that we understand the system we are dealing with, we can ask the question: how can we increase the detectable effect size of our experiments? We are left with a few options:
- Increase the effect size
- Increase the sample size
- Decrease the variance
Increasing the effect size may or may not be possible depending on the effect we are investigating.
Increasing the sample size is often the easiest way to improve the power of a test, however because the detectable effect size scales as, it becomes harder and harder to increase the power of an experiment this way. In reality the sample size is often constrained by cost or time.
This leaves the option of reducing the variance.
Stratified sampling can reduce the variance, however for online experiments we are usually not able to sample from strata ahead of time.
Improving the Sensitivity of Online Controlled Experiments, getting results faster
by Utilizing Pre-Experiment Data
Y ~intercept + beta *trt
Y ~ intercept +beta*trt+some other stuff
assumptions: repeated visitors, if no repeated visitors then use other information for variance reduction (e.g. registration information)