Why ML Engineer Candidates Can't Defend Their Own Numbers

BORROWED NUMBER — READ AS AN INTERVIEWER:

Candidate: "I improved model accuracy to 94%."

Interviewer: "94% on what?"

Candidate: "On the test set."

Interviewer: "Which split? Random or
time-based?"

Candidate: "Random, I think."

Interviewer: "What was the baseline?"

Candidate: "The previous model was around
85%, I believe."

Interviewer: "What happened in production?"

Candidate: "I'm not sure — the data team
tracked it."

What the interviewer thinks at each answer:
- "94% on what?" → vague. ❌
- "Random split" → for a churn model, that
  means customer leakage. ❌
- "Around 85%, I believe" → you don't know
  your own baseline. ❌
- "The data team tracked it" → you didn't
  own the outcome. ❌
- "I can't defend any of these numbers." ❌

Verdict: the number is now a liability. The
candidate would have been stronger saying
"we got 0.78 AUC against a 0.71 baseline on
a time-based split, but production degraded
to 0.72 within 6 weeks because of drift."

Why it collapsed:
❌ Borrowed from a dashboard, not measured
❌ No split strategy
❌ No baseline you can name
❌ No production outcome
❌ Every follow-up exposes a gap

NO BASELINE — READ AS AN INTERVIEWER:

Candidate: "We got an AUC of 0.87."

Interviewer: "0.87 compared to what?"

Candidate: "Compared to... the old model?"

Interviewer: "What was the old model's AUC?"

Candidate: "I think it was lower. Maybe 0.8?"

Interviewer: "Was there a rule-based baseline?"

Candidate: "I don't think we built one."

Interviewer: "How did you decide 0.87 was
good enough to ship?"

Candidate: "The team decided it was good."

What the interviewer thinks:
- "0.87 with no baseline is a number with
  no meaning." ❌
- "A rule-based baseline would have cost a
  day and told you if the model was worth
  the complexity." ❌
- "'The team decided' means you didn't
  own the decision." ❌
- "I can't tell if this model is good or
  just better than nothing." ❌

Verdict: no baseline, no decision, no
defense.

What it signals:
❌ You didn't measure against anything
❌ You didn't own the ship decision
❌ You can't say why 0.87 mattered
❌ You optimized a metric, not a problem

Why it collapses:
❌ No baseline = no meaning
❌ No decision threshold = no ownership
❌ No business outcome = no impact
❌ The interviewer has nothing to compare

FULL DEFENSE — A NUMBER THAT SURVIVES:

Interviewer: "I see '0.78 AUC' on your
resume. Walk me through it."

WHAT YOU SAY:
"Sure. The baseline was 0.71 AUC from the
previous model, and 0.64 from a rule-based
baseline we built in a day to check whether
the model was worth it.

The split was time-based: 18 months of
training data, the last 3 months as the test
set. I chose time-based because a random
split puts the same customer in train and
test — for churn, that's leakage.

I also checked for target leakage. The
original feature set included
account_closed_date, which is only set after
the customer churns. Dropping it dropped the
offline AUC from 0.92 to 0.78 — which told
me the 0.92 was leakage, not skill.

In production, the AUC settled at 0.73, below
the 0.78 offline. The cause was drift on
monthly_spend after a pricing change. We
added a PSI monitor with an alert at 0.2, and
retrained quarterly.

So the number I'd defend is 0.73 in
production against 0.71 baseline — a small
lift, but a real one, and we know why it's
small."

WHAT IT PROVES:
✅ Baseline: 0.71 previous, 0.64 rule-based
✅ Split: time-based, 18/3 months
✅ Leakage: found and dropped a leaking
  feature, which changed the number
✅ Production: 0.73, with a cause (drift)
  and a fix (PSI monitor)
✅ Honesty: "a small lift, but a real one"

Why it survives follow-ups:
- "Why time-based?" → leakage
- "Why 0.78 vs. 0.92?" → leakage feature
- "Why did production drop?" → drift
- "What did you do?" → PSI monitor, retrain
Every answer is already in the story.

WHEN IT FAILED — A NUMBER THAT STILL SURVIVES:

Interviewer: "You claim a 12% latency
improvement. Walk me through it."

WHAT YOU SAY:
"Sure. p95 latency went from 340ms to 300ms
after we added batch inference and a feature
cache. That's a 12% cut, but I want to be
honest: it's smaller than the 40% we hoped
for.

The baseline was the previous model serving
path: p95 340ms under our production load of
about 1,200 requests/sec. We measured with
the same load test before and after.

The split for evaluation was the same: a
held-out 5% traffic slice for a week, not a
synthetic benchmark.

What didn't work: we tried quantization
first, and it gave a 6% gain but degraded
recall on the tail class by 3 points, so we
rolled it back.

What we shipped: batch inference for the
batch-scoring path (no latency win) and a
feature cache for the real-time path (the
12% win). The 40% target assumed both would
help real-time, and they didn't.

If I were doing it again, I'd measure the
feature cache first, before spending two
weeks on quantization."

WHAT IT PROVES:
✅ Baseline: 340ms p95 at 1,200 req/sec
✅ Split: held-out 5% traffic for a week
✅ What didn't work: quantization, rolled
  back for a real reason
✅ Honest gap: 12%, not 40%
✅ Reflection: what you'd do differently

Why it survives:
- "Why only 12%?" → quantization didn't
  help real-time
- "Why roll back quantization?" → recall
  drop on the tail class
- "What would you change?" → measure the
  cache first
A failed number, defended well, is stronger
than a big number you can't explain.

COST — DEFENDING A NUMBER:

Interviewer: "You say you cut inference cost
by 60%. Defend it."

WHAT YOU SAY:
"Sure. Baseline GPU spend was $18K/month
across three inference clusters. After the
changes, it was $7K/month — that's a 61%
cut.

The changes were three: first, batching
requests at the gateway, which raised GPU
utilization from 22% to 61%. Second,
right-sizing: we moved from 4×A100 to
6×A10G, which cost less per hour and matched
our throughput. Third, a model swap: we
moved the tail-class model to a distilled
version that was 4x cheaper and only lost
2 points of recall on that class.

The trade-off was real: the distilled model
is 2 points worse on the tail class, and we
accepted that because the tail class is 3%
of traffic and the recall drop was inside
our SLA.

We measured with the same load test before
and after — 1,200 req/sec, p95 under 310ms
both times. So the cost cut didn't come
from degrading latency.

What I'd watch: the distilled model needs
monthly retraining, or it drifts faster than
the full model. We added that to the
pipeline."

WHAT IT PROVES:
✅ Baseline: $18K/month, 3 clusters
✅ Changes: batching, right-sizing, model
  swap — named
✅ Utilization: 22% → 61%, a measured
  improvement
✅ Trade-off: 2-point recall drop on a 3%
  class, inside SLA
✅ Measurement: same load test, p95 held
✅ Risk: distilled model drifts faster,
  retraining added

Why it survives:
- "Where did the savings come from?" →
  three named changes
- "What did you lose?" → 2-point recall
  on a 3% class
- "How do you know latency held?" → same
  load test
- "What's the risk?" → faster drift,
  monthly retrain

MODEL SWAP — DEFENDING A NUMBER:

Interviewer: "You replaced a model and
improved NDCG by 8%. Walk me through it."

WHAT YOU SAY:
"Sure. Baseline NDCG@10 was 0.42 from the
previous two-tower model. The new model got
0.45 on the same evaluation set — that's a
roughly 7–8% relative lift.

The evaluation set was the last 4 weeks of
user interactions, not a random split,
because a random split leaks future
interactions into training. We also
deduplicated users across train and test.

The lift wasn't uniform. It came mostly from
cold-start users, where the new model uses
content features. For warm users, the lift
was under 1% — basically noise.

So the honest claim is '7% overall, driven
by cold-start users, flat for warm users.'
Not '8% for everyone.'

In A/B, the online lift was 5%, not 7%. The
gap was a filter that the offline eval didn't
model. We shipped the model and added the
filter to the offline eval so the next
comparison is fair.

What I'd do differently: include the filter
in the offline eval from day one, so we
don't ship on an inflated number."

WHAT IT PROVES:
✅ Baseline: 0.42 NDCG@10
✅ Split: last 4 weeks, deduplicated users
✅ Slice analysis: cold-start lift, flat
  warm users
✅ Online vs. offline gap: 7% → 5%, with a
  named cause
✅ Reflection: add the filter to offline eval

Why it survives:
- "Why time-based split?" → future
  interaction leakage
- "Is the lift uniform?" → no, cold-start
  only
- "Why 5% online, not 7%?" → the filter gap
- "What changed?" → added the filter to
  the offline eval

ML Number Audit Checklist:

FOR EVERY NUMBER ON YOUR RESUME:
☐ Baseline: what did you compare against?
  (previous model, rule-based, random)
☐ Split: time-based? customer-disjoint?
  random? Why that choice?
☐ Leakage: which feature could inflate
  the number, and did you drop it?
☐ Offline vs. online: what was the
  production result?
☐ Slice: where is the lift real, and
  where is it flat?
☐ Trade-off: what did you give up?
  (recall, latency, cost, complexity)
☐ Failure: what didn't work, and why
  did you roll it back?
☐ Reflection: what would you do
  differently?

RED FLAGS IN YOUR OWN NUMBERS:
❌ You can't name the baseline
❌ You can't name the split
❌ You can't name the production result
❌ You borrowed the number from a
  dashboard
❌ The number is bigger than the baseline
  by a suspicious amount
❌ You can't explain the lift

THE TWO-QUESTION TEST:
1. "Compared to what?" → you answer
2. "What happened in production?" → you
  answer

If both answers are ready, the number
is defensible. If not, shrink the number
until they are.

30-DAY NUMBER DEFENSE PLAN:

Week 1 — Audit
☐ List every number on your resume
☐ For each, write the baseline
☐ For each, write the split
☐ For each, write the production result
☐ Mark the ones you can't answer

Week 2 — Shrink
☐ For each undefendable number, shrink
  it to something you can defend
☐ "94% accuracy" → "0.73 production AUC
  against a 0.71 baseline"
☐ "40% latency cut" → "12% p95 cut at
  1,200 req/sec"
☐ Remove numbers you can't trace

Week 3 — Rehearse
☐ For each number, record a 60-second
  defense: baseline → split → leakage →
  production → trade-off
☐ Listen back for vague language
  ("around," "I think," "the team")
☐ Re-record until every number is
  specific

Week 4 — Stress test
☐ Have a peer ask "compared to what?"
  and "what happened in production?"
  after every number
☐ If you pause, that's the number to
  shrink or remove
☐ Re-run the stress test until no pauses

Signs it's working:
✅ You lead with the baseline, not the
  number
✅ You say "0.73 in production" not
  "0.92 offline"
✅ You name the leakage feature you
  dropped
✅ You name what didn't work
✅ Interviewers stop pushing after the
  second question