You ran an A/B experiment for a new AI-driven feature: the primary metric shows a small, non-significant uplift; several secondary metrics show significant improvement; and certain user segments show large positive effects while others show neutral or negative effects. Walk through how you would analyze these results and decide on whether to roll out, iterate, or stop. Include statistical checks, risk assessment, and rollout strategies.

I’d start with statistical triage: verify randomization, check for instrumentation bugs, and recompute effect sizes and 95% CIs. If the primary metric is +0.8% (p=0.12) but CTR secondary +6% (p<0.01) and Segment X shows +15% while Segment Y shows −4%, I’d check power — we may be underpowered for a 1–2% lift. I’d correct for multiple comparisons (Benjamini-Hochberg) and run interaction tests to confirm heterogeneity. Risk-wise I’d estimate revenue and brand impact if negative segments scale. My likely path: don’t full-roll; instead run targeted rollouts for the high-value positive segments (start at 10% ramp to 50%) while iterating on the experience for neutral/negative segments via follow-up experiments. Add monitoring and rollback rules (e.g., revenue drop >2%). If follow-ups show consistent net benefit across cohorts after 4–8 weeks, I’d proceed to broader launch.