Why Some Data Scientist Candidates Ace Every Round Except the Last

Manager: Tell me about a project you are proud of.

Candidate: I built a churn model with XGBoost. I did
feature engineering on 40 features, tuned it with
Optuna, and got an AUC of 0.87, which beat logistic
regression. I used SHAP for explainability.

Manager: And what happened after that?

Candidate: ...it was a course project, so nothing.

Manager: Tell me about a project you are proud of.

Candidate: I worked on predicting which subscribers
were likely to cancel in the next month, so a retention
team could decide who to call.

The interesting part was not the model. It was deciding
how many people to contact. Calling everyone is too
expensive, and calling too few misses most of the risk.
I set the threshold using the cost of a call and the
value of a saved customer.

It was a portfolio project, so I could not test it live.
If I could, I would hold out a random group that gets no
calls, to measure whether the calls actually reduce churn.

import numpy as np
import pandas as pd
from sklearn.metrics import confusion_matrix

# y_test: 1 = churned, proba: predicted churn probability
# Example assumptions only - replace with the team's real figures
VALUE_SAVED = 2000    # margin kept when a churner is retained
OFFER_COST = 300      # cost of one retention offer
SAVE_RATE = 0.3       # share of contacted churners who stay

rows = []
for t in np.arange(0.1, 0.91, 0.1):
    pred = (proba >= t).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_test, pred, labels=[0, 1]).ravel()
    value = tp * SAVE_RATE * VALUE_SAVED - (tp + fp) * OFFER_COST
    rows.append({"threshold": round(t, 1),
                 "contacted": tp + fp,
                 "churners_caught": tp,
                 "net_value": value})

result = pd.DataFrame(rows).sort_values("net_value", ascending=False)
print(result.head())

"The model gives each customer a churn score.
 We still need to choose how many people to contact.

 I treated it as a cost question. Each offer costs money.
 Only some churners stay after an offer, and contacting
 someone who would have stayed anyway is wasted.

 So for each threshold I calculated the expected value:
 money saved from retained churners minus the cost of
 all offers sent.

 The default threshold of 0.5 is rarely the best one.
 With these example numbers the best threshold was lower,
 because a saved customer is worth more than an offer.
 If offers became more expensive, it would move up."

import pandas as pd
from statsmodels.stats.proportion import proportions_ztest

# flagged: customers above the threshold, randomly assigned
# group: 'treated' (got the offer) or 'holdout' (did not)
# churned: 1 if the customer left within 30 days
summary = flagged.groupby("group")["churned"].agg(["sum", "count"])
print(summary.assign(churn_rate=summary["sum"] / summary["count"]))

stat, p_value = proportions_ztest(
    count=summary.loc[["treated", "holdout"], "sum"].values,
    nobs=summary.loc[["treated", "holdout"], "count"].values,
)
print(f"z = {stat:.2f}, p = {p_value:.4f}")

# Say aloud: "The model chooses who is at risk. The holdout
# tells us whether the offer changes their behaviour."

Q: When would you advise against building a model?

A: "Three situations come to mind.

 First, when a simple rule already captures most of the
 value, for example 'customers whose card payment failed
 twice'. A rule is easier to explain and maintain.

 Second, when nobody can act on the prediction. A churn
 score is useless if there is no retention team or offer.

 Third, when there is too little labelled history to
 learn from. Then I would start by collecting better data
 and use simple analysis in the meantime."

PROJECT: ______________________________

Business problem in one line: ____________________
Who acts on the output: _________________________
What action they take: __________________________
Cost of a false positive: _______________________
Cost of a false negative: _______________________
How the threshold was chosen: ___________________
How impact was or would be measured: ____________
What I would do differently: ____________________

PRACTISE ALOUD, NO JARGON ALLOWED

1. Why did this project matter to the business?
2. Who used your work, and what did they do with it?
3. How would you know the model made a difference?
4. Tell me about a time you pushed back on a request.
5. When is a model the wrong answer?
6. How do you explain uncertainty to a leader?
7. What would you do in your first month here?
8. What kind of problems do you want to work on?

REPLACE THE TERM WITH THE PLAIN VERSION

AUC 0.87        -> ranks risky customers well
Precision 0.6   -> 6 of 10 people we contact are real risks
Recall 0.7      -> we catch 7 of 10 customers who would leave
Overfitting     -> works on old data, not on new customers
Feature import. -> the main signals the model relies on
p-value 0.03    -> unlikely to be just random noise
Confidence int. -> the range the true effect probably sits in

(Ratios here illustrate the translation, not real results.)