A/B Testing &amp; Experimentation Interview Questions for Data Science

Show answer

The control group sees the current experience, and the treatment group sees the change being tested, such as a new checkout button. Users are randomly assigned, so the only systematic difference between groups is the change itself. Any difference in metrics beyond random noise can then be attributed to the treatment. Without a proper control, you cannot separate the effect of your change from seasonality, marketing or other launches.

What is an A/A test and why would you run one?

Show answer

In an A/A test both groups get the identical experience. Since there is no real difference, results should be significant only about 5 percent of the time at alpha 0.05. Running A/A tests validates your experimentation platform: it catches broken randomisation, logging bugs, sample ratio problems and metrics whose variance is wrongly estimated. Teams often run them when launching a new platform or a new metric.

What are primary metrics and guardrail metrics in an experiment?

Show answer

The primary metric is the one the change is meant to improve and that decides success, such as checkout conversion. Guardrail metrics are ones that must not get worse, such as page load time, refund rate, unsubscribes or revenue per user. Defining both before the test prevents cherry-picking afterwards. A change that lifts the primary metric but breaks a guardrail is usually not shipped without further discussion.

Your experiment ended with no statistically significant difference. What do you tell the team?

Show answer

Not significant does not mean no effect. Report the observed effect with its confidence interval, for example minus 0.3 to plus 1.2 percent, and compare it with the MDE the test was powered for. If the interval rules out effects that matter, conclude the change has little impact and decide based on cost or design preference. If the interval is wide, the test was underpowered, and a longer test or variance reduction may help.

A product manager wants to test five different button designs at the same time. How do you set it up?

Show answer

Run an A/B/n test with control plus five variants and equal random allocation. Calculate sample size for the smallest effect worth detecting, remembering that each variant gets only a sixth of traffic, so the test takes longer. Plan the analysis upfront: compare each variant to control and adjust for multiple comparisons. If the team has strong priors, test fewer designs first to save time.

Write Python code for a two-proportion z-test comparing conversion between control and treatment.

Show answer

from statsmodels.stats.proportion import proportions_ztest, proportion_confint conversions = [480, 540] # control, treatment visitors = [10000, 10000] z, p = proportions_ztest(conversions, visitors) lift = conversions[1] / visitors[1] - conversions[0] / visitors[0] ci_t = proportion_confint(conversions[1], visitors[1], method='wilson') print(f'z={z:.2f} p={p:.4f} abs lift={lift:.4f} treatment CI={ci_t}')

Write SQL to compute users, conversions and conversion rate for each experiment variant.

Show answer

SELECT a.variant, COUNT(DISTINCT a.user_id) AS users, COUNT(DISTINCT o.user_id) AS converters, ROUND(COUNT(DISTINCT o.user_id) * 1.0 / COUNT(DISTINCT a.user_id), 4) AS conv_rate FROM assignments a LEFT JOIN orders o ON o.user_id = a.user_id AND o.created_at >= a.assigned_at WHERE a.experiment_id = 'checkout_v2' GROUP BY a.variant;

What is the randomisation unit in an experiment, and why does it matter?

Show answer

The randomisation unit is what you assign to variants: a user, a session, a device, a page view or even a city. Choosing user-level keeps each person's experience consistent and avoids contamination. The analysis unit should match the randomisation unit, because if you randomise by user but analyse by page view, observations are correlated and standard errors come out too small, producing false positives. Use the delta method or bootstrap when they differ.

What is the minimum detectable effect (MDE) in experiment design?

Show answer

The MDE is the smallest true effect your test can reliably detect given the sample size, baseline rate, significance level and power. If baseline conversion is 5 percent and your MDE is 10 percent relative, you can detect a move to 5.5 percent but will likely miss smaller lifts. Choose the MDE from business value: the smallest change worth shipping. A smaller MDE needs a much larger sample, roughly growing with one over MDE squared.

What are the novelty effect and the primacy effect in A/B tests?

Show answer

The novelty effect is when users engage more with a change simply because it is new, so early results overstate the long-term lift. The primacy effect is the opposite: existing users resist change at first, so early results understate it. Both fade with time. To detect them, plot the treatment effect by day and by user tenure, compare new users with existing users, and run the test long enough for behaviour to settle.

What is an Overall Evaluation Criterion (OEC) in experimentation?

Show answer

An OEC is a single agreed metric, or a weighted combination of metrics, that represents long-term business value and is used to judge experiments. Good OECs are sensitive enough to move within a test's duration yet aligned with long-term goals; for example, sessions per user with penalties for churn rather than raw clicks. Having an OEC prevents teams from each optimising their own metric and makes ship decisions consistent across the organisation.

What is Bayesian A/B testing and how are its results reported?

Show answer

Bayesian A/B testing starts with a prior on each variant's metric, updates it with observed data and produces a posterior distribution. Instead of a p-value, you report things like the probability that treatment beats control, say 94 percent, and the expected loss if you choose the wrong variant. Business teams find this easier to interpret. Priors must be chosen sensibly, and continuous monitoring still needs discipline about decision rules.

Your test shows a 2 percent lift in conversion, but revenue per user went down. What do you do?

Show answer

Do not ship on conversion alone. Check whether the change shifted buyers to cheaper items, increased discount usage or pulled in low-value orders. Break down revenue into conversion times average order value, look at item mix, and confirm the revenue drop is statistically meaningful rather than noise. Then go back to the agreed primary metric and guardrails; if revenue is the business goal, it should guide the decision.

A feature won its A/B test with a 4 percent lift, but after full launch the metric barely moved. Why might that happen?

Show answer

Several reasons are common. Winner's curse: significant results from underpowered tests tend to overestimate the true effect. Novelty effect faded after launch. Seasonality or other launches changed the baseline, so a before-after comparison is misleading. The test population may differ from all users, for example only one platform. Use a holdout after launch to measure real impact, and run tests with adequate power and duration.

Your power analysis says you need 200,000 users per group, but the page gets only 5,000 users a week. What are your options?

Show answer

Waiting 80 weeks is unrealistic. Options: test a bolder change so the expected effect is larger; use a more sensitive metric closer to the change, like add-to-cart instead of purchase; apply CUPED to reduce variance; or move the test to a higher-traffic page. If none work, accept a larger MDE explicitly, or decide using qualitative research and a staged rollout with monitoring instead of a formal test.

Halfway through a test you discover a bug that broke the treatment for Android users. What do you do?

Show answer

Stop or pause the affected part, fix the bug and restart the test with fresh data, since mixing broken and fixed periods biases results. Do not simply drop Android users from analysis afterwards, because that can break randomisation balance if the bug affected behaviour or logging. Document the incident, and add pre-launch checks such as running the variant through QA on every platform and monitoring guardrails in the first hours.

Write Python code to calculate the sample size per group needed to detect a lift from 5 percent to 5.5 percent conversion.

Show answer

from statsmodels.stats.power import NormalIndPower from statsmodels.stats.proportion import proportion_effectsize baseline, target = 0.05, 0.055 effect = proportion_effectsize(target, baseline) # Cohen's h n = NormalIndPower().solve_power( effect_size=effect, alpha=0.05, power=0.8, ratio=1.0, alternative='two-sided' ) print(round(n)) # about 31,000 users per group

Write Python code to compare revenue per user between two groups using Welch's t-test.

Show answer

import numpy as np from scipy import stats ctrl = df.loc[df.variant == 'control', 'revenue'].to_numpy() treat = df.loc[df.variant == 'treatment', 'revenue'].to_numpy() t, p = stats.ttest_ind(treat, ctrl, equal_var=False) # Welch diff = treat.mean() - ctrl.mean() se = np.sqrt(treat.var(ddof=1) / len(treat) + ctrl.var(ddof=1) / len(ctrl)) print(f'diff={diff:.2f} 95% CI=({diff - 1.96*se:.2f}, {diff + 1.96*se:.2f}) p={p:.4f}')

Tell me about a time an experiment result changed a product decision.

Show answer

Situation: the team was confident a redesigned home page would increase orders. Task: I owned the experiment design and analysis. Action: I set conversion as primary and session length and refunds as guardrails, ran it for two full weeks and found no lift in orders but a drop for older app versions. Result: we fixed the performance issue, retested, and shipped a version that raised orders by about 1.5 percent.

What is a long-term holdout group and when is it useful?

Show answer

A holdout is a small share of users, say 2 to 5 percent, kept away from a set of launched features for months. Comparing them with everyone else measures the cumulative, long-term impact of many shipped changes, which individual short tests can overstate because of novelty effects or winner's curse. It is useful for recommendation systems, notifications and marketing, but it costs some value from users who miss improvements.

How does a multi-armed bandit differ from a classic A/B test?

Show answer

A classic A/B test splits traffic in fixed proportions until the end, then decides, which is best for learning and clean inference. A multi-armed bandit, such as Thompson sampling, shifts traffic towards better-performing variants as data arrives, reducing the cost of showing a weak variant. Bandits suit short-lived decisions like headlines or promotions. They make unbiased effect estimates and guardrail analysis harder, so they are poor for major product decisions.

What are heterogeneous treatment effects, and what are the risks of slicing results by segment?

Show answer

Heterogeneous treatment effects mean the change helps some segments more than others, for example new users but not repeat users. Finding them can guide targeting. The risk is that slicing by many segments creates false discoveries, and each segment has less power, so differences may be noise. Pre-register key segments, correct for multiple comparisons, and validate any surprising segment effect in a follow-up test before acting on it.

What is interference or a network effect in experiments, and why does it break standard A/B tests?

Show answer

Standard A/B analysis assumes one user's outcome does not depend on another user's assignment, the SUTVA assumption. In social or messaging products, a treated user's new feature affects their untreated friends, so control is contaminated and the effect is biased. Solutions include cluster randomisation, where connected groups of users are assigned together, ego-network designs and comparing results across different cluster sizes to estimate the spillover.

What is CUPED and how does it reduce variance in experiments?

Show answer

CUPED uses a pre-experiment covariate, usually the same metric for each user in the weeks before the test, to remove predictable variation. The adjusted metric is Y minus theta times (X minus the mean of X), where theta is cov(X, Y) divided by var(X). Because randomisation makes X balanced across groups, the estimate stays unbiased while variance can drop by 30 to 50 percent, letting tests reach significance faster with the same traffic.

Why do ratio metrics like click-through rate per user need special variance calculation?

Show answer

A metric like total clicks divided by total page views has a numerator and denominator that both vary by user, and page views within a user are correlated. Treating each page view as independent underestimates variance and inflates false positives. The delta method approximates the variance of a ratio of means using the variances and covariance of the per-user numerator and denominator. Bootstrap by user is an alternative when formulas get complicated.

You want to test a new pricing algorithm in a two-sided marketplace where buyers and sellers interact. Why is user-level randomisation a problem and what would you do?

Show answer

Treated and control buyers compete for the same sellers' supply, so a treatment that makes treated buyers buy more takes inventory from control buyers, exaggerating the measured lift. Use designs that limit interference: switchback tests that alternate the whole market between variants over time slots, or randomise by city or region. Analyse at the switch or cluster level with appropriate variance, and accept that these tests need longer to reach power.

Marketing wants to measure the impact of a TV and outdoor campaign that cannot be randomised at user level. How do you design the measurement?

Show answer

Plan a geo experiment: select comparable cities or regions, randomly assign some to run the campaign and keep others as control, matching them on past sales trends. Measure the difference in sales during and after the campaign, using methods like difference-in-differences or synthetic control to build a counterfactual. Run a pre-period check that test and control regions track each other, and include enough regions for reasonable power.

Write a Python function that deterministically assigns users to experiment variants using hashing.

Show answer

import hashlib def assign(user_id, experiment, variants=('control', 'treatment'), weights=(50, 50)): key = f'{experiment}:{user_id}'.encode() bucket = int(hashlib.sha256(key).hexdigest(), 16) % 100 cumulative = 0 for v, w in zip(variants, weights): cumulative += w if bucket < cumulative: return v # Same user + experiment always gets the same variant; # salting with the experiment name keeps tests independent

Write Python code to apply a CUPED adjustment using each user's pre-experiment metric.

Show answer

import numpy as np # df has: variant, y (metric during test), x (same metric pre-test) x, y = df['x'].to_numpy(), df['y'].to_numpy() theta = np.cov(x, y, ddof=1)[0, 1] / np.var(x, ddof=1) df['y_cuped'] = y - theta * (x - x.mean()) print(df.groupby('variant')[['y', 'y_cuped']].agg(['mean', 'var'])) # Means stay comparable; variance of y_cuped is lower

Tell me about a time you stopped a team from shipping a change based on a flawed experiment.