A/B Testing &amp; Experiments Interview Questions for Data Analytics

Show answer

An A/B test is a controlled experiment where users are split into a control group that sees the current version and a treatment group that sees a change, and you compare a metric such as conversion. Random assignment makes the groups similar in every way except the change, including things you cannot measure. That is why a difference in results can be attributed to the change itself, not to user mix or seasonality.

How do you write a good hypothesis for an experiment?

Show answer

A good hypothesis names the change, the expected effect, the metric and the reason. For example: showing delivery date on the product page will increase add-to-cart rate by at least 2% because uncertainty about delivery stops purchases. It should be written before the test starts, be measurable, and state the null hypothesis of no difference. A vague hypothesis like 'the new page will be better' leads to cherry-picking metrics afterwards.

What are primary metrics and guardrail metrics in an A/B test?

Show answer

The primary metric is the one decision metric the test is designed to move, such as checkout conversion. Guardrail metrics are ones that must not get worse, such as page load time, refund rate, unsubscribe rate or revenue per user. A change that lifts conversion but increases cancellations should not ship. Define both before launch, so the decision rule is clear and nobody picks the favourable metric afterwards.

What are novelty and primacy effects in experiments?

Show answer

A novelty effect is when users engage more with a change simply because it is new, so the early lift fades over time. A primacy effect is the opposite: existing users resist a change at first because they are used to the old design, so results start poor and improve. Both mean short tests mislead. Run tests for at least one or two full weeks and compare new versus returning users.

Write SQL to calculate users, conversions and conversion rate for each test variant.

Show answer

SELECT a.variant, COUNT(DISTINCT a.user_id) AS users, COUNT(DISTINCT o.user_id) AS converters, ROUND(100.0 * COUNT(DISTINCT o.user_id) / COUNT(DISTINCT a.user_id), 2) AS conversion_pct FROM experiment_assignments a LEFT JOIN orders o ON o.user_id = a.user_id AND o.order_ts >= a.assigned_ts WHERE a.experiment_id = 'checkout_v2' GROUP BY a.variant;

When would you use a one-tailed test instead of a two-tailed test?

Show answer

A two-tailed test checks for a difference in either direction and is the safer default for A/B tests. A one-tailed test checks only one direction, such as treatment being better, and needs fewer users. Use it only if you decided beforehand that you would act the same way whether the change has no effect or a negative effect. Otherwise you may miss harm, which is why most teams stay two-tailed.

What is a holdout group and why do teams keep one?

Show answer

A holdout group is a small share of users, often 5-10%, kept out of a feature or of all marketing campaigns for a long time. Comparing them with everyone else shows the cumulative long-term impact, which individual short tests miss, and checks whether many small wins really add up. For example, a notification holdout shows whether push messages raise monthly retention or just shift activity earlier.

What is the difference between an A/B test, an A/B/n test and a multivariate test?

Show answer

An A/B test compares one change with control. An A/B/n test compares several versions of one element, such as three headlines, with control. A multivariate test changes several elements at once, such as headline and button colour, and tests every combination to see each element's effect and their interactions. Multivariate tests need much more traffic because users are spread across many combinations.

Your A/B test ends with no significant difference. How do you report it?

Show answer

A flat result is still useful. Report the observed difference with its confidence interval, for example -0.3% to +0.8%, which shows the range of effects still possible. State whether the test had enough power for the planned MDE. If it did, conclude that any effect is probably small and the team can choose based on cost or simplicity. Share learnings so the same idea is not retested blindly.

What are statistical power and minimum detectable effect (MDE)?

Show answer

Power is the probability that the test detects a real effect of a given size, commonly set at 80%. The minimum detectable effect is the smallest lift you care about and can reliably detect with your sample size. Smaller MDEs, lower baseline rates and higher power all need more users. If you can only get 10,000 users a week, you may only be able to detect large lifts, which should shape what you test.

Write Python code to calculate the sample size per group for a conversion test with a 4% baseline and a 10% relative lift.

Show answer

from statsmodels.stats.power import NormalIndPower from statsmodels.stats.proportion import proportion_effectsize baseline = 0.04 target = baseline * 1.10 # 4.4% effect = proportion_effectsize(target, baseline) n = NormalIndPower().solve_power( effect_size=effect, alpha=0.05, power=0.8, alternative='two-sided' ) print(round(n)) # roughly 39,000 users per group

What is the unit of randomisation in an experiment, and how do you choose it?

Show answer

The unit of randomisation is what gets assigned to control or treatment: a user, a session, a device, a store or a city. Usually it should be the user, so one person always sees the same version. Session-level randomisation can show one user both versions and contaminate results. Your analysis unit should match it; if you randomise by user but analyse per page view, standard errors become too small.

What is an A/A test and why would you run one?

Show answer

An A/A test splits users into two groups that both see the same version. Since nothing differs, you expect no significant difference except about 5% of the time by chance. It checks that randomisation, tracking and the analysis pipeline work correctly, and it reveals sample ratio mismatches or metric variance before real tests depend on the platform. Running many A/A tests also confirms your false positive rate is near 5%.

Write Python code to test whether conversion differs between control and treatment.

Show answer

from statsmodels.stats.proportion import proportions_ztest, proportion_confint conversions = [1180, 1295] # control, treatment visitors = [30000, 30100] z, p = proportions_ztest(conversions, visitors) print(f'z = {z:.2f}, p = {p:.4f}') for c, n in zip(conversions, visitors): low, high = proportion_confint(c, n, method='wilson') print(f'rate {c/n:.3%} 95% CI {low:.3%} to {high:.3%}')

Your test increased conversion rate by 3% but average order value dropped by 5%. What do you recommend?

Show answer

Look at the metric that reflects business value, such as revenue or margin per visitor, which combines both effects. Here revenue per visitor is roughly 1.03 times 0.95, about 2% lower, so the change may hurt. Check whether the drop is significant, which segments drive it, and whether new buyers are making smaller first orders that grow later. Without a strong reason, I would not ship it as is.

On day 4 of a planned 14-day test, the product manager wants to stop because the dashboard shows treatment winning. What do you say?

Show answer

I would explain that early results swing a lot and stopping on a good day inflates false positives, and that four days miss weekly patterns like weekend shopping. I would show how the lift has moved day by day and agree to the planned end date. If speed really matters, we can agree a sequential testing rule for future tests, but changing the rule mid-test after seeing data is not valid.

Tell me about a time an experiment result went against what the team expected.

Show answer

Situation: the team was sure a shorter signup form would raise registrations. Task: I owned the test analysis. Action: registrations rose 6%, but I checked the guardrail and found seven-day activation fell, because we had removed a question that personalised onboarding. I shared both results with the segment breakdown. Result: we kept the short form but moved the question to the first screen after signup, which recovered activation.

Marketing wants to A/B test two email subject lines on a list of 2,000 subscribers. What do you advise?

Show answer

With 1,000 per group and a typical click rate of 3%, only a very large difference, roughly 70% relative or more, would be detectable, so a small real lift would look like noise. I would suggest testing on a larger send, pooling several campaigns, or using open-to-click rate with a bigger effect. If they still run it, report the confidence interval and treat the result as directional.

What is the peeking problem in A/B testing?

Show answer

Peeking means checking the p-value repeatedly during a test and stopping as soon as it drops below 0.05. Each look is another chance for random noise to look significant, so the real false positive rate can rise from 5% to 20% or more. Fix it by fixing the sample size in advance and analysing once, or by using methods built for continuous monitoring, such as sequential testing or always-valid confidence intervals.

Your 50/50 test has 52,400 users in control and 47,600 in treatment. What does this tell you?

Show answer

This is a sample ratio mismatch. A chi-square test on these counts gives a tiny p-value, so the gap is not chance. Something is filtering users unevenly, such as the treatment page crashing on some devices, a redirect losing tracking, bots in one arm or a bug in assignment. The results cannot be trusted until the cause is found. Check splits by browser, platform and day to locate it.

Why is testing many metrics or many variants at once a problem, and how do you handle it?

Show answer

Each comparison at alpha 0.05 has a 5% false positive chance, so checking 20 metrics will probably flag one as significant by luck. This is the multiple comparisons problem. Handle it by choosing one primary metric in advance, treating others as exploratory, and adjusting p-values with Bonferroni or Benjamini-Hochberg when several comparisons matter. With many variants, compare each to control and adjust, or use a larger sample.

Why are A/B tests tricky in two-sided marketplaces such as ride-hailing or food delivery?

Show answer

In marketplaces, groups share supply, so the treatment can affect control. If treatment riders get a discount and book more, fewer drivers remain for control riders, which makes treatment look better than it would at full rollout. This violates the independence assumption. Teams use cluster randomisation by city or region, switchback tests that alternate treatment by time slots, or model the spillover when estimating impact.

An A/B test is flat overall but significant for mobile users in Tier 2 cities. Do you ship it only for that segment?

Show answer

Not straight away. Slicing results into many segments makes a chance finding likely, so this is a hypothesis, not a conclusion. Check whether the segment was planned in advance, how many segments were examined, and whether there is a plausible reason, such as slower networks benefiting from a lighter page. If it still looks promising, run a follow-up test targeted at that segment with enough sample to confirm it.

What is the difference between frequentist and Bayesian A/B testing?

Show answer

Frequentist testing asks how surprising the data would be if there were no difference, and gives p-values and confidence intervals with a fixed sample size. Bayesian testing combines a prior with the data to give a probability that treatment beats control and the expected loss of choosing it. Bayesian results are easier for business users to read, but still depend on priors and still need sensible stopping rules.

What is CUPED and how does it speed up experiments?

Show answer

CUPED is a variance reduction method that uses each user's pre-experiment behaviour, such as spend in the previous four weeks, as a covariate. You adjust the outcome by removing the part explained by past behaviour, which is unrelated to the treatment because assignment was random. The lower variance gives tighter confidence intervals, so you can detect the same effect with fewer users or a shorter test, often 20-50% faster.

How do you measure impact when a proper A/B test is not possible?

Show answer

Use quasi-experimental methods. Difference-in-differences compares the change in a treated group, such as cities that got a new policy, with the change in similar untreated cities over the same period. Synthetic control builds a weighted mix of untreated regions to mimic the treated one. Regression discontinuity uses a cut-off, like a credit score threshold. Always check pre-period trends and state the assumptions clearly.

Write Python code for a simple difference-in-differences estimate of a store pilot.

Show answer

import pandas as pd # df columns: store_type ('pilot' or 'control'), period ('pre' or 'post'), sales avg = df.pivot_table(index='store_type', columns='period', values='sales', aggfunc='mean') pilot_change = avg.loc['pilot', 'post'] - avg.loc['pilot', 'pre'] control_change = avg.loc['control', 'post'] - avg.loc['control', 'pre'] did = pilot_change - control_change print(f'Estimated pilot effect per store: {did:.1f}')

Write Python code to bootstrap a 95% confidence interval for the difference in revenue per user between two groups.