Designing Experiments

Design experiments that give answers you can trust: controls, randomisation, blinding, fair comparisons, ablations, A/B tests, natural experiments and a written plan before you start.

18 video lessons · about 32 min · free to study

1. Watch · module 1Why design matters3 lessons

   • The lady tasting tea2 min
   • Control and comparison2 min
   • Randomisation2 min
   • Practice quiz: Why design matters · practice, optional4 min
2. Watch · module 2Variables and bias3 lessons

   • Variables and confounders2 min
   • Blinding2 min
   • Measuring what you mean2 min
   • Practice quiz: Variables and bias · practice, optional4 min
3. Watch · module 3Fair comparisons in ML3 lessons

   • Fair tuning2 min
   • Seeds and variance2 min
   • Data leakage2 min
   • Practice quiz: Fair comparisons in ML · practice, optional4 min
4. Watch · module 4Controls and checks3 lessons

   • Designing an ablation2 min
   • Positive and negative controls2 min
   • Sanity checks2 min
   • Practice quiz: Controls and checks · practice, optional4 min
5. Watch · module 5Online and field experiments3 lessons

   • A/B tests2 min
   • Peeking and stopping early2 min
   • When you can't randomise2 min
   • Practice quiz: Online and field experiments · practice, optional4 min
6. Watch · module 6Planning3 lessons

   • Pre-registration2 min
   • Writing a test plan2 min
   • Pilot first2 min
   • Practice quiz: Planning · practice, optional4 min
7. Graded · step 1Hands-on project

   Your experiment plan. Checked by a reviewer.

8. Graded · step 2Graded test

   20 questions · 30 minutes · 70% to pass · sign in to take it

9. FinishYour certificate

   For members. Joining is free and takes about 3 minutes.