Distribution
Averages hide spread, skew and outliers. Look at the shape, not just one number.
Analysis · Module 07
Statistics for Analysts.
You do not need to become a statistician. You do need enough statistical judgment to avoid confidently telling the wrong story.
Core idea
Statistics is not about adding complexity. It is about knowing how much confidence a result deserves.
01
Context
A city improved 28%. Should we copy its playbook?
“City C improved cancellation rate by 2.3 percentage points — much more than the other cities. Should we copy what they did?”
The percentage-point improvement is real in the observed data. But City C has only 610 orders while the other cities each have more than 12,000.
The first question is not “is 2.3 bigger than 0.9?” It is how stable is that estimate, and how much business impact does it represent?
02
Core concepts
Six statistical ideas analysts use constantly.
Variance
A metric can move because the underlying process became less stable, even if the average barely changed.
Confidence
Observed differences are estimates. Ask how much uncertainty surrounds them.
Seasonality
Day-of-week, holidays and recurring cycles can create “changes” that are actually normal patterns.
Correlation
Two things moving together does not establish that one caused the other.
Bias
Who is included, excluded or self-selected can matter more than the statistical method.
Rule
Use statistics to calibrate confidence, not to decorate a weak argument with technical language.
03
Distribution
The average can describe nobody.
Delivery time Mean: 31 min
If the distribution has a long right tail, “31 minutes average” can hide a meaningful group of customers waiting 60–90 minutes.
For operational metrics, always ask whether the decision is about the typical experience, the tail, or the share crossing an SLA threshold. Mean, median, percentiles and rates answer different questions.
04
Sample size & uncertainty
A bigger movement is not always stronger evidence.
| City | Orders | Before | After | Observed change |
|---|---|---|---|---|
| A | 12,480 | 8.1% | 7.4% | -0.7 pp |
| B | 12,620 | 8.0% | 7.1% | -0.9 pp |
| C | 610 | 8.2% | 5.9% | -2.3 pp |
Largest observed improvement, but based on only 610 orders.
Smaller movement, but measured across more than 12,000 orders.
Magnitude, uncertainty and absolute impact all matter to the decision.
05
Seasonality
Compare like with like.
Monday vs Sunday
Demand mix, traffic and operations may naturally differ by weekday.
Monday vs recent Mondays
Controls for recurring day-of-week patterns before calling the shift unusual.
Expected range vs actual
Use recent history to understand normal variation, not only the previous period.
06
Correlation
“Users who do X retain better” is not a causal conclusion.
Observed Promo users retain 18% better.
Possible explanation 01 The promotion improves retention.
Possible explanation 02 Already-engaged users are more likely to redeem promotions.
Possible explanation 03 Promo eligibility targets customer segments that naturally retain better.
Language rule
For observational analysis, prefer “associated with,” “correlated with,” or “we observe” unless your design supports a causal claim.
07
Challenge
Which result would you act on?
Exercise · 15 minutes
Interpret before recommending.
- Would you call City C the strongest success? What additional evidence do you need?
- Which is more decision-relevant: percentage-point change or number of cancellations prevented?
- How would you check whether weekday mix explains part of the change?
- If promo users retain better, what confounders would you investigate?
- What result could be statistically convincing but operationally too small to matter?
- How would you communicate uncertainty without becoming vague?
Reveal one defensible interpretation
City CPromising, not proven
The observed improvement is large but the sample is much smaller.
City BSmaller but potentially higher impact
A modest rate improvement across a large order base can prevent more cancellations.
Next stepCheck uncertainty and comparability
Control for weekday mix, customer mix and operational conditions before copying the intervention.
LanguageSeparate observation from cause
Report what changed, how certain you are, and what would establish causality.
08
AI assist
Ask AI to challenge the inference.
Statistical reasoning prompt
I am reviewing a business analysis.
Observed result:
City C cancellation rate improved from 8.2% to 5.9% across 610 orders.
City B improved from 8.0% to 7.1% across 12,620 orders.
Before recommending that we copy City C's intervention:
1. Identify the statistical and business questions I should ask.
2. Explain how sample size affects confidence in the observed change.
3. Suggest checks for seasonality and customer/operational mix.
4. Separate statistical significance from practical business impact.
5. Identify any causal claims that the data does not justify.
6. Do not invent p-values, confidence intervals or missing data. 09
Validation
The statistical judgment checklist.
01
Did I inspect the distribution, not only the average?
02
Is the comparison period affected by seasonality or calendar effects?
03
Is the sample large enough for the conclusion I want to make?
04
Am I confusing a large percentage change with a large business impact?
05
Could selection bias or survivorship bias explain the result?
06
Am I describing correlation as if it were causation?
07
Is the observed difference practically meaningful?
08
What evidence would change my interpretation?
10
Takeaway
Do not confuse precision with truth.
Analyst habit
Whenever a result looks exciting, ask three questions: how large is the effect, how uncertain is the estimate, and what alternative explanation could produce the same pattern?