Technical Interviews
Data Science Case Study Interview: Two Worked Cases
A data science case study interview gives you a product situation, usually a metric that moved or an experiment to design or read, and asks you to reason through it aloud. Interviewers judge the order you work in, whether you check the data before explaining it, and whether you end with a decision.
What is a data science case study interview?
A consulting case asks about markets, profit and strategy, often with market sizing; the case interview preparation guide covers that one. The data science case lives inside a product instead: sign-ups fell, a test came back positive, a feature needs a success metric. This guide answers two of them aloud, a metric drop and an A/B test, and lists the follow-ups that come after each.
The experimentation advice comes from articles by Microsoft’s Experimentation Platform team and from Evan Miller’s writing on A/B testing errors.[1][2] For depth, the standard reference is Kohavi, Tang and Xu’s book on online controlled experiments.[6] No company’s internal rubric is claimed here.
Case 1: “Weekly sign-ups fell 12%. What happened?”
Work in a fixed order: clarify the metric, rule out a data problem, separate internal from external causes, segment until the drop has a location, then form a hypothesis and say what you would do.
The order is this guide’s frame, not a published standard, but each step guards against a specific mistake. Say the order out loud before you start, so the interviewer can follow and redirect you.
- Clarify. What counts as a sign-up, and has that definition changed? Which week against which? Is 12% outside the normal week-to-week range, or is this a normal dip?
- Check the data. Did a logging change, a pipeline delay or a tracking release land this week? Microsoft’s ExP team puts data-quality metrics in the core metric set it recommends for experiments, and the same habit applies to a dashboard number.[3] A surprising number deserves suspicion before explanation; the ExP articles cite Twyman’s law for exactly that.[5]
- Internal or external. Internal: a release, a pricing change, a broken page, a marketing campaign that ended. External: a holiday, a competitor launch, an app store or OS change, an outage at a vendor.
- Segment. Platform, country, new or returning, traffic source, and the step of the sign-up funnel where users stop.
- Hypothesis and decision. Name the most likely cause, the one check that would confirm it, and what you would do today.
Suppose that after clarifying, the interviewer gives you the cut by platform. The figures are invented.
| Platform | Last week | This week | Change |
|---|---|---|---|
| Web | 21,400 | 21,100 | -1% |
| iOS | 9,800 | 9,600 | -2% |
| Android | 8,800 | 4,500 | -49% |
| Total | 40,000 | 35,200 | -12% |
Almost the whole drop is Android: 4,300 of the 4,800 lost sign-ups. That moves the question from “why are fewer people signing up?” to “what broke on Android?”. Ask for the Android funnel next.
| Step | Last week | This week |
|---|---|---|
| Opened sign-up screen | 14,000 | 13,800 |
| Entered phone number | 11,200 | 11,000 |
| Received verification code | 10,900 | 5,300 |
| Completed sign-up | 8,800 | 4,500 |
Saying it out loud
“Users are still reaching the sign-up screen and entering their number at the same rate. The loss is between entering the number and receiving the code: that step went from about 97% to about 48%. So this looks like SMS delivery on Android, not demand. I would check whether an Android release or a change at our SMS provider went out early this week, and look at delivery failures by country and carrier. Today I would raise it with the team that owns verification, because we are losing about 4,300 sign-ups a week, and I would ask whether we can offer email verification as a fallback while it is fixed.”
The answer names the location of the drop with numbers, gives one hypothesis and the check that would confirm it, and ends with an action and its size. It does not list twenty possible causes.
Case 2: “Should we ship the new checkout button?”
Design the test before reading any result: hypothesis, primary and guardrail metrics, randomisation unit, sample size and duration fixed in advance. Then check the test is healthy before you trust its numbers.
Interviewers ask this two ways: design the test, or here are the results, what do you conclude? Cover the design either way, because most wrong conclusions come from a design that was never set.
Design the test
- Hypothesis. “The new button raises completed purchases per user.” Microsoft’s ExP team says a hypothesis should be falsifiable by the metrics you chose.[3]
- Metrics. One primary metric (completed purchases per user), guardrails (page load time, payment errors, refunds) and data-quality checks. The ExP pre-experiment article lists guardrail and data-quality metrics in the core set alongside user satisfaction and engagement metrics.[3]
- Randomisation unit. The user, not the session or page view, so one person sees one version. ExP lists ID stability and network effects among the things to weigh, and warns that cookie-based IDs only suit short tests.[3]
- Sample size and duration, fixed before launch. A power calculation sets the minimum number of units.[3] Evan Miller’s rule of thumb is n = 16σ²/δ², where δ is the smallest effect you want to detect and σ² the variance; for a conversion rate p, σ² = p(1 − p).[1] With a 10% baseline and a smallest effect of half a percentage point, that is 16 × 0.09 / 0.000025, about 57,600 users per variant. Then round up to whole weeks so weekdays and weekends are both in it.
Run it without fooling yourself
Do not stop the first time the result looks significant. Evan Miller’s example shows that checking repeatedly and stopping at the first significant reading can push a nominal 5% false-positive rate to 26.1%.[1] Run to the planned size, or use a sequential design built for early stopping. The exception is harm: ExP recommends alerts on guardrails and automatic shutdown of tests that badly degrade the product.[4]
Check the test is healthy before reading results
First, the split. A sample ratio mismatch means the users in each arm do not match the configured split. Microsoft’s ExP team reports that about 6% of its A/B tests have one, flags it at p < 0.0005, and advises not to trust the results of a test with an SRM until the cause is found.[2] On a 50/50 test with 200,000 users, a 104,000 to 96,000 split is far beyond chance.
Second, the effect over time. ExP notes that a rapid decline of the estimated effect across date segments is a sign of novelty effect.[4] Plot the lift by day. If it starts at 6% and settles at 1%, the honest estimate is closer to 1%.
Decide
If the primary metric is up and a guardrail is down, decide with weights agreed before the test. ExP describes Bing assigning pre-determined weights to ship-decision metrics that do not depend on the test being analysed.[5] And if a minor change shows a huge win, re-run it: “when in doubt, we re-run” is ExP’s own rule.[5]
A recommendation that sounds like a data scientist
“The split is 50.1 to 49.9, so no sample ratio problem. Purchases per user are up 1.8%, and the daily lift has been flat for the last ten days, so I don’t see a novelty effect. Page load and payment errors are unchanged, refunds are within noise. I’d ship it to everyone and keep the refund guardrail on the dashboard for a month, because refunds lag purchases.”
How are data science case answers judged?
On structure, on reading the numbers, on the recommendation and on how clearly you say it. This guide does not claim any employer’s rubric; the table is how the practice round below scores a case.
| Criterion | Weight | What it rewards here |
|---|---|---|
| Framework Application | 25% | An order you say out loud and follow: clarify, check the data, segment, hypothesise, decide. For a test: hypothesis, metrics, unit, sample size, checks. |
| Data Analysis | 25% | Reading the numbers in front of you: which segment carries the drop, which funnel step loses users, whether the split looks right. |
| Strategic Thinking | 25% | A recommendation with its risk, what you would check next, and what result would change your mind. |
| Communication | 25% | Signposting each step, stating assumptions as assumptions, and leading with the conclusion when you sum up. |
The round’s criteria weigh framework application 25%, data analysis 25%, strategic thinking 25% and communication 25%. That matches the advice above: a sharp diagnosis with no recommendation, or a confident recommendation built on a test with a broken split, gives up a whole criterion either way.
What follow-up questions come after a case?
Most follow-ups test one experimentation trap. Know the trap each one is pointing at.
| Follow-up | What a good answer does |
|---|---|
| “The test hit significance on day 3. Can we ship?” | No, not on that alone. Checking repeatedly and stopping at the first significant result inflates false positives. Run to the sample size and duration fixed before launch, unless a guardrail shows real harm. |
| “Control has 52% of users, treatment 48%.” | That is a sample ratio mismatch on a 50/50 split. Stop reading the metrics and find the cause: a redirect, a bot filter, a crash in one arm, a logging difference. |
| “The lift was big in week one and has faded since.” | Plot the effect by day. A steady decline suggests a novelty effect. Report the later, stable effect, and consider whether new and returning users differ. |
| “Conversion is up, but page load time and refunds are worse.” | Decide with the trade-off rule agreed before the test, not a new one invented for this result. If there was none, say what you would need to know to weigh them. |
| “The drop is only on Android, only since Tuesday.” | Look for what changed on Android on Tuesday: an app release, an OS update, an SDK or vendor change. Then find the funnel step where the loss happens. |
| “The result looks too good: plus 20% on a minor feature.” | Be suspicious before excited. Check the data and the trigger, look for a bug that inflates one arm, and re-run if in doubt. |
| “How long would you run the test?” | Long enough to reach the sample size from the power calculation, and at least one full weekly cycle so weekday and weekend users are both in it. |
What mistakes cost candidates in a data science case?
Nearly all of them are about skipping a check because the story was already forming.
- Explaining a drop before asking whether the data or its definition changed.
- Listing every possible cause instead of segmenting until the drop has a location.
- Designing a test with one metric and no guardrails.
- Randomising by session or page view when the change affects a user over several visits.
- Stopping a test the day it turns significant.
- Reading results without checking the split first.
- Reporting the first week’s lift as the long-run effect.
- Ending without a recommendation, or recommending without saying what would change your mind.
How should you prepare in a week?
Learn the checks once, then drill saying them under questioning.
- Day 1: read the ExP pre-, during- and post-experiment articles and Evan Miller’s piece. Write the checks as a one-page list.
- Day 2: compute three sample sizes by hand with the 16σ²/δ² rule, for different baselines and effects, so the numbers feel familiar.
- Day 3: answer five metric-drop prompts aloud (daily active users, sign-ups, orders, session length, retention), timing each to ten minutes.
- Day 4: answer three test-design prompts aloud, then three “here are the results” prompts with a planted SRM or novelty effect.
- Days 5 to 7: full mock cases with someone pushing follow-ups, or the case study round below, and redo the weakest answer each day.
How do you practise a data science case study out loud?
Run a case study round with the role set to Data Scientist. Every question comes with a data table, and the whole round stays on one business situation, which is how a metric-drop case unfolds.
The case study round opens with a situation and a table of figures, then keeps returning to the same product with a new cut each time, for example segments, then a funnel, then costs or options, with numbers that agree across tables. The asks vary the way the cases above do: diagnose, prioritise, size the impact, choose, name the risk. The AI interviewer asks one question at a time and follows up on vague answers, and each answer is scored for relevance, depth and clarity.
The round does not check a sample-size computation for you, so say your numbers and assumptions aloud as you would to an interviewer. You can start a case study round for a data scientist on the free plan. For the rest of the loop, the data scientist mock interview covers modelling and stakeholder questions, and the data analyst mock interview covers SQL and dashboards. For consulting-style cases, use case interview practice.
Frequently asked questions
Short, direct answers to the questions people ask most often about this topic. Each answer stands on its own, so you can jump straight to the one you need.
How is a data science case study different from a consulting case interview?
A consulting case is usually about a business decision such as market entry or profitability, with market sizing. A data science case is about a product metric: why it moved, how you would measure a change, and whether an experiment result can be trusted. The structure habits are similar, the content is not.
How much statistics do I need for a data science case interview?
Enough to design and read an A/B test: hypotheses, significance and power, how sample size depends on the effect you want to detect, why peeking is a problem, and what a sample ratio mismatch is. You rarely need to derive formulas, but you should know which number drives which decision.
Is a product sense interview the same as a case study?
They overlap. Product sense questions ask what to build or how to measure a feature. Case studies usually hand you a metric movement or an experiment to diagnose. Many loops mix both in one conversation, so practise defining a success metric as well as diagnosing a drop.
Will I need SQL in the case study round?
Often not in the same conversation. Many loops test SQL in its own round and keep the case spoken. Say which queries you would run, in plain words, and keep SQL practice for the technical round.
Can I practise a data science case study for free?
Yes. The free plan is 2 interviews a month, free account, no card.
Sources
The studies, statutes and primary pages this guide relies on. Numbers in square brackets in the text link here; where a claim has no number, it is our own guidance from running practice interviews.
- How Not To Run an A/B Test. Evan Miller
- Diagnosing Sample Ratio Mismatch in A/B Testing. Microsoft Research, Experimentation Platform
- Patterns of Trustworthy Experimentation: Pre-Experiment Stage. Microsoft Research, Experimentation Platform, 2020-07-31
- Patterns of Trustworthy Experimentation: During-Experiment Stage. Microsoft Research, Experimentation Platform, 2021-01-25
- Patterns of Trustworthy Experimentation: Post-Experiment Stage. Microsoft Research, Experimentation Platform, 2021-12-24
- Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Kohavi, Tang and Xu, Cambridge University Press, 2020
How answers are scored and which data we publish: data and research · methodology