Why Most Data Scientist Candidates Can't Defend Their Own Numbers

Interviewer: Your resume says you improved accuracy by 12%.
Improved from what?

Candidate: From the earlier model.

Interviewer: What was its accuracy?

Candidate: Around 80, I think. Or maybe that was F1.

Interviewer: Was the 12% measured on the test set?

Candidate: It was the best score I got while tuning.

Interviewer: Your resume says you improved recall.

Candidate: Yes. Recall on the positive class went from
0.61 with the logistic regression baseline to 0.68 with
the gradient boosting model, at the same precision of 0.50.

That is seven percentage points. I measured both on the
same held-out test set, which was the final two months
of data, and I only scored that set once.

The test set was small, so I bootstrapped an interval.
The new model's recall was roughly 0.64 to 0.72, so the
improvement is real but I would not claim more precision
than that.

import numpy as np
from sklearn.metrics import roc_auc_score

def bootstrap_ci(y_true, y_score, metric=roc_auc_score,
                 n_boot=2000, alpha=0.05, seed=42):
    rng = np.random.default_rng(seed)
    y_true = np.asarray(y_true)
    y_score = np.asarray(y_score)
    n = len(y_true)
    stats = []
    for _ in range(n_boot):
        idx = rng.integers(0, n, n)
        if len(np.unique(y_true[idx])) < 2:   # AUC needs both classes
            continue
        stats.append(metric(y_true[idx], y_score[idx]))
    lo, hi = np.percentile(stats, [100 * alpha / 2, 100 * (1 - alpha / 2)])
    return metric(y_true, y_score), lo, hi

auc, lo, hi = bootstrap_ci(y_test, proba)
print(f"AUC {auc:.3f} (95% CI {lo:.3f} to {hi:.3f})")

from statsmodels.stats.proportion import proportion_confint

# Example: 46 conversions out of 520 users who saw the new page
conversions, users = 46, 520
rate = conversions / users
low, high = proportion_confint(conversions, users,
                               alpha=0.05, method="wilson")
print(f"rate {rate:.3f}, 95% CI {low:.3f} to {high:.3f}")

# Spoken: "The observed rate is about 8.8 percent, but with
# this many users the plausible range is roughly 6.7 to 11.6
# percent. I would not make a big decision on this alone."

import numpy as np
from statsmodels.stats.contingency_tables import mcnemar

# pred_a, pred_b: class predictions from two models on the same test set
correct_a = (pred_a == y_test)
correct_b = (pred_b == y_test)

table = [
    [np.sum(correct_a & correct_b),  np.sum(correct_a & ~correct_b)],
    [np.sum(~correct_a & correct_b), np.sum(~correct_a & ~correct_b)],
]
result = mcnemar(table, exact=False, correction=True)
print(table)
print(f"statistic={result.statistic:.2f}, p-value={result.pvalue:.4f}")

# Only the off-diagonal cells matter: rows where one model
# is right and the other is wrong.

Q: Is model B really better than model A?

A: "Both were scored on the same test set, so I used a
 paired comparison rather than comparing two separate
 accuracy numbers.

 McNemar's test looks only at rows where the models
 disagree. If B fixes many more of A's mistakes than it
 introduces, the test shows it.

 In my project the p-value was below 0.05, so I treated
 the improvement as real. I also checked that it held
 within the bootstrap intervals and across folds.

 If it had not, I would have kept model A, because it was
 simpler."

CLAIM ON RESUME: _______________________________

Metric and exact value: _______________________
Dataset and split: ____________________________
Number of rows / positives: ___________________
Measured once at the end? Y / N
Baseline or previous value: ___________________
Absolute change: ______  Relative change: ______
95% interval (bootstrap or formula): __________
What it means in plain words: _________________

BEFORE -> AFTER (illustrative wording)

"Improved accuracy by 12%"
 -> "Raised recall from 0.61 to 0.68 at fixed precision
    versus a logistic regression baseline"

"Built a highly accurate model"
 -> "Gradient boosting model, AUC 0.78 on a later
    time period, baseline 0.70"

"Reduced costs significantly"
 -> "Simulated a lower contact cost under stated
    assumptions; real impact not yet measured"

Use your own true figures, never invented ones.

ASK YOURSELF ALOUD FOR EACH NUMBER

1. Where exactly did this number come from?
2. Training, validation or test data?
3. How many examples, and how many positives?
4. What was the baseline value?
5. Is the change in points or percent?
6. What is a plausible range for the true value?
7. Would it survive a different random seed?
8. What decision does this number support?