Randomization
Assign treatment independently of expected outcomes.
Business Domains · Module 17
Experiments & Causal Thinking.
When the business asks whether a change caused an outcome, observational dashboards are often not enough. You need a design that creates a credible counterfactual.
Core idea
Causality asks a counterfactual question: what would have happened to the same population without the intervention?
01
Context
Promo users retain better. Did the promo cause it?
“The users who received the offer retained much better. Why not launch it globally?”
If eligibility targeted high-intent customers, the difference can exist even if the promotion has zero causal effect.
A randomized experiment creates a more credible counterfactual, but only if assignment, exposure, measurement and sample integrity are checked.
02
Experiment design
Create a comparison that can support causality.
Primary metric
Choose the outcome the decision is based on before reading results.
Guardrails
Track harms such as margin, complaints or cancellations.
Integrity
Check sample ratios, exposure, contamination and observation windows.
Rule
Do not upgrade correlation into causation because the business wants a decisive answer.
03
Challenge
Design the promotion experiment.
Exercise · 20 minutes
Create the counterfactual.
- What population is eligible?
- What is the randomization unit?
- What is the primary metric and window?
- Which guardrails protect economics/customer experience?
- How would you detect contamination?
- What would make you stop interpreting the test?
04
AI assist
Use AI as an experiment reviewer.
Experiment design prompt
Review this proposed experiment:
Goal:
estimate the causal effect of a retention promotion.
Population:
eligible active customers.
Need:
treatment/control design, randomization unit, primary metric,
observation window, guardrails, sample-ratio checks,
contamination risks and stopping/invalidity conditions.
Do not invent a required sample size without baseline rates
or a minimum detectable effect.
Separate experiment design from business recommendation.05
Validation
The causal inference checklist.
01
Is treatment assignment independent of expected outcomes?
02
Was the primary metric defined before results?
03
Are treatment/control observation windows equal?
04
Did sample ratios match the assignment plan?
05
Could treatment leak into control?
06
Are guardrail harms included?
PRACTICE THIS NOW ↓
Interactive case · L06
Experiment Integrity Check
Treatment conversion looks better, but the assignment split is wrong. Run integrity checks before reading lift.
06
Worked takeaway
A credible counterfactual changes what you can claim.
Describe associations and confounders honestly.
Use randomized assignment when practical.
Validate assignment and exposure before lift.
Combine causal lift with cost and guardrail effects.
Analyst habit
Whenever someone says “X users perform better,” ask what those users would have done without X.