IntermediateTECHNICAL
Walk me through how you would design and analyze an A/B test for a new homepage layout, including: defining the primary metric, calculating sample size, choosing randomization and experiment duration, and how you’d handle inconclusive or conflicting results.
Data Scientist
General

Sample Answer

I’d start by aligning on the business goal: for a new homepage, I’d usually define primary metric as uplift in qualified sign‑ups or add‑to‑cart rate, not just CTR. Suppose baseline sign‑up rate is 5% and we want to detect a +10% relative lift (to 5.5%) with 80% power and α=0.05. I’d use a power calculator (e.g., statsmodels or an internal tool) to compute required sample size, then convert that to test duration based on daily traffic and expected variance. Randomization would be user‑level, 50/50 split, using our experimentation platform to avoid cross‑device contamination and ensure covariate balance. I’d predefine guardrail metrics (bounce rate, latency). For inconclusive results, I’d check power and variance, run sensitivity analyses, and consider meta‑analysis with prior tests. For conflicting metric movements (e.g., CTR up, sign‑ups flat), I’d do funnel analysis and segment cuts before recommending rollout, iteration, or follow‑up tests.

Keywords

Primary metric tied to business outcome (qualified sign-ups/add-to-cart, not just CTR)Power-based sample size with explicit MDE, α, and power (e.g., statsmodels)User-level randomization, experiment platform, and guardrail metricsHandling inconclusive/conflicting results via power checks, segmentation, and funnel analysis
Related Questions

Based on your hydrology and irrigation engineering background, explain how you would estimate the irrigation water requirement for a kharif crop in a semi-arid region of Gujarat. Walk me through each step: from reference evapotranspiration estimation, crop coefficient selection, effective rainfall calculation, to arriving at canal discharge for a given command area.

IntermediateTECHNICAL

In your civil engineering studies, what specific design coursework or project work did you complete related to irrigation channels or canals (e.g., design of lined/unlined canals, distributaries, minors)? Describe one such design in detail, including how you determined discharge, permissible velocity, section dimensions, and lining choice for Gujarat-type soil and climate conditions.

IntermediateTECHNICAL

Walk me through a recent multi-channel digital marketing campaign you managed end-to-end. How did you set objectives, choose channels, allocate budget, and measure success?

IntermediateBEHAVIORAL

In your resume you note improving or optimizing [a process, KPI, or metric]. What specific baseline metrics did you start from, what steps did you personally take, and how did you verify that the improvement was due to your changes rather than external factors?

IntermediatePROBLEM_SOLVING

On your resume you mention working on a cross-functional project (e.g., involving multiple teams or stakeholders). Describe a situation from that project where priorities conflicted—how did you navigate the trade-offs and what was the final outcome?

IntermediateSITUATIONAL