Why I Always Ask About Failed Models, Not Just Wins

STRONG ANSWER — FIRST 90 SECONDS:

Interviewer: "Tell me about a model that
failed."

Candidate: "Sure. The churn model collapsed
in production, and it took a month to
notice.

The impact: the model ranked the top 200
customers for the retention call list. In
the first month, the team called 200
customers and saved 12 — versus 18 for a
random list. The model was worse than
random.

The root cause: the validation used random
K-fold on time-series data. The ticket
feature — 'support tickets in the last 30
days' — included tickets filed after the
churn date. So the model learned from the
future, and the validation score was 0.85.
In production, where the future wasn't
available, the score was 0.58.

The fix: I moved to time-based validation
with a 30-day gap, and I removed the ticket
feature. The honest score was 0.62 — lower
than the inflated 0.85, but real.

The prevention: I added a leak check that
fails the pipeline if any feature has a
timestamp after the prediction cutoff. It
caught two more leaky features in the next
model.

The lesson: a validation score that's too
good is a warning, not a win. I now
compare the validation score to a
production shadow score before launch.

I can show you the leak check and the
shadow score."

What the interviewer thinks in 90 seconds:
- "They named the model and the impact." ✅
- "They named the root cause: leakage." ✅
- "They named the fix and the prevention." ✅
- "They named the lesson in a reusable
  rule." ✅
- "They understand their own model." ✅

Signal: strong. The rest of the interview
confirms it.

WEAK ANSWER — FIRST 90 SECONDS:

Interviewer: "Tell me about a model that
failed."

Candidate: "Yeah, there was a churn model
that didn't work out. The accuracy was fine
in validation but it didn't work in
production. We ended up shipping the
rule-based version instead. It was a while
ago so I don't remember the details."

What the interviewer thinks in 90 seconds:
- "No model named." ❌
- "No impact — was anyone using it?" ❌
- "No root cause — 'didn't work out' is
  not a cause." ❌
- "No fix, no prevention." ❌
- "They didn't understand the model well
  enough to remember it." ❌
- "I'll ask a follow-up, but I already
  doubt the wins." ⚠️

What happens next:
- The interviewer asks "What was the
  validation scheme?"
- The candidate says "I think it was
  cross-validation."
- The interviewer asks "Was the data
  time-ordered?"
- The candidate pauses.

Signal: weak. The candidate's wins now look
less impressive, because they can't explain
a failure.

Why it failed:
❌ No specific model
❌ No impact named
❌ No root cause
❌ No fix or prevention
❌ No lesson
❌ The first 90 seconds signaled "I trained
  models," not "I own models"

ROOT CAUSE ANSWER — WHAT DECIDES:

Interviewer: "How did you find the root
cause?"

WHAT THE CANDIDATE SAYS:
"I found it by comparing the validation
score to the production score, then working
backwards.

Step one: I checked the production score
against a random baseline. The model's top
200 had a 6% save rate; a random 200 had a
9% save rate. That's worse than random —
which almost always means leakage in
validation or a distribution shift.

Step two: I checked for distribution
shift. I compared the feature distributions
between validation and production — tenure,
spend, tickets. Tenure and spend matched.
Tickets didn't: validation had a much higher
ticket count for churners.

Step three: I tested the timestamp. The
ticket feature was 'tickets in the last 30
days,' and for churned customers, some
tickets were filed after the churn date —
when they called to cancel. So the feature
encoded the outcome.

The root cause was my validation scheme:
random K-fold put the post-churn tickets in
the training set, and the model learned
'churners have tickets.' That's true, but
it's not predictive — it's the result, not
the cause.

The fix: time-based validation with a
30-day gap, and the ticket feature removed.
The honest score was 0.62 — 23 points lower
than the inflated 0.85, but real.

The trade-off: I lost the ticket feature,
which was the strongest predictor on paper.
I accepted that because a leaky feature isn't
a predictor — it's a time machine."

What the interviewer thinks:
- "They compared to a random baseline." ✅
- "They used distributions to narrow the
  cause." ✅
- "They tested the timestamp." ✅
- "The root cause is a validation error." ✅
- "The cost of the fix is named." ✅

Signal: confirmed judgment.

Why it decides:
✅ Baseline comparison
✅ Distribution check
✅ Timestamp test
✅ Root cause is honest
✅ The cost of removal is named

PREVENTION ANSWER — WHAT DECIDES:

Interviewer: "How did you make sure it
didn't happen again?"

WHAT THE CANDIDATE SAYS:
"Three layers, and each catches a different
failure.

Layer one: a leak check. Every feature has a
declared 'as-of' timestamp, and the pipeline
fails if any feature's as-of is after the
prediction cutoff. That catches the exact
mistake I made — a feature that encodes the
future.

Layer two: time-based validation by
default. The validation split is time-ordered
with a gap that matches the prediction
horizon. Random K-fold is only allowed if
the data has no time dimension, and that
requires a written justification. That
catches the scheme I got wrong.

Layer three: a shadow score. Before launch,
the model scores live traffic for two weeks
without acting on it. If the shadow score
diverges from validation by more than 10
points, the launch is paused. That catches
distribution shift and any remaining leaks
that the timestamp check missed.

The cost: the shadow period delays launch
by two weeks, and the leak check adds about
2 minutes to the pipeline. I chose that over
a faster launch because a model that's worse
than random for a month costs 200 calls a
week — the shadow period is cheap insurance.

The result: the leak check has caught two
more leaky features, and the shadow score
caught one drift issue before launch. Neither
reached production."

What the interviewer thinks:
- "Three layers, each with a purpose." ✅
- "The checks target different failures." ✅
- "The leak check targets the exact
  mistake." ✅
- "The cost is quantified: 2 weeks, 2
  minutes." ✅
- "The checks have evidence." ✅

Signal: confirmed judgment.

Why it decides:
✅ Three layers, not one
✅ Each targets a failure class
✅ The exact mistake is covered
✅ The cost is quantified
✅ The checks have real evidence

DRIFT STORY — STRONG VS. WEAK SIGNAL:

Scenario: "Tell me about a model that
decayed over time."

WEAK SIGNAL:
"The model got old and we stopped using it.
I think the data changed."
What the interviewer thinks:
- "No root cause — what changed?"
- "No prevention — would it happen again?"
- "They waited for the model to decay
  instead of monitoring it."
- Signal: weak.

STRONG SIGNAL:
"The churn model decayed after a pricing
change in Q3. Here's what happened.

The timeline: the model launched in Q1 with
a 0.62 validation score. By Q3, the shadow
score had dropped to 0.51. The retention
team noticed the call list was worse than
random before I did.

The root cause: we changed the pricing
model — from monthly to annual with a
discount. The 'spend in the last 30 days'
feature was now measuring something
different: annual customers had a spike at
renewal and near-zero in between. The model
read the near-zero as 'about to churn,' but
it was actually 'already paid for a year.'

The fix: I added a feature for plan type and
retrained. The score recovered to 0.59 —
lower than 0.62, because the pricing change
genuinely changed churn behavior, but real.

The prevention: I added a drift monitor that
runs weekly on the feature distributions. If
any feature's population stability index
crosses 0.2, it alerts. The pricing change
would have triggered it in week one.

The cost: the drift monitor adds about 10
minutes a week. I chose that over a monthly
review because a quarter of bad calls is
expensive — 200 calls a week for 13 weeks.

The lesson: a model is a snapshot of a
process, and the process changes. I now
treat a pricing or product change as a
trigger to re-check the model, not a
coincidence to ignore."

What the interviewer thinks:
- "The timeline is specific." ✅
- "The root cause is a pricing change." ✅
- "The fix is specific and honest about the
  score." ✅
- "The prevention is a drift monitor." ✅
- "The lesson is a reusable rule." ✅
- Signal: confirmed judgment.

Why it decides:
✅ Timeline is specific
✅ Root cause is a business change
✅ Fix is honest about the new score
✅ Prevention targets the cause
✅ Lesson is reusable

INTERVIEW FOLLOW-UP — STRONG VS. WEAK SIGNAL:

Scenario: "How did you know the retrained
model was better?"

WEAK SIGNAL:
"I retrained it and the score looked good."
What the interviewer thinks:
- "Good compared to what?"
- "No validation."
- Signal: weak.

STRONG SIGNAL:
"Three checks, and I can show the numbers.

Check one: I compared the retrained model
to the old model on the same time-based
validation window. The new model scored 0.59
versus 0.51 for the old one — an 8-point
improvement, but still below the original
0.62. That's honest: the pricing change
changed churn behavior, and the original
score isn't achievable anymore.

Check two: I ran both models in shadow mode
for two weeks. The new model's top 200 had
a 9% save rate; the old model's had 6%. The
shadow confirmed the validation improvement
translated to production.

Check three: I back-tested the drift
monitor against the Q3 data. It would have
fired in week one of the pricing change —
the population stability index for the spend
feature crossed 0.2 in the first week. So
the monitor would have caught this before
the retention team noticed.

The cost: the shadow period delayed the
launch by two weeks. I chose that because
8 points of validation improvement doesn't
always translate to production — the shadow
period is the check that matters.

The trade-off: the new model is more complex
— it has the plan-type feature, which needs
a join to the billing table. That join adds
about 2 seconds to the scoring pipeline. I
accepted it because the plan-type feature is
the one that solves the root cause."

What the interviewer thinks:
- "Three checks, each with a purpose." ✅
- "The comparison is honest about the
  lower score." ✅
- "The shadow period is real evidence." ✅
- "The back-test proves the monitor would
  work." ✅
- "The trade-off is named with a cost." ✅
- Signal: confirmed judgment.

Why it decides:
✅ Comparison against the old model
✅ Honest about the lower score
✅ Shadow period is real evidence
✅ Back-test of the monitor
✅ Trade-off with a cost

Failed-Model Story Checklist:

FIRST 90 SECONDS:
☐ Name the model and what decision it
  supported
☐ Name the impact: calls wasted, revenue
  lost, customers misclassified
☐ Name the root cause, not the blame
☐ Avoid "it didn't work out" — name the
  actual cause

WHAT DECIDES:
☐ Root cause is a validation error, a leak,
  a drift, or a definition error
☐ The fix addresses the root cause, not a
  symptom
☐ The prevention targets the failure class
☐ Name how you knew the fix worked:
  comparison, shadow, back-test
☐ Name the cost of the fix or the
  prevention

WHAT CONFIRMS:
☐ A reusable lesson or rule
☐ Honest about who found it
☐ The follow-up "how did you know?" is
  answered with evidence
☐ "Looks right" is rejected as evidence
☐ The prevention has caught real issues
  since

RED FLAGS:
❌ "I don't remember the details"
❌ No model named
❌ No impact
❌ "It didn't work out" as the cause
❌ No fix, no prevention
❌ "I retrained it and it looked good"
❌ Blame on the data or the stakeholder
❌ No lesson

Why this works:
- The first 90 seconds set the signal
- The rest of the story confirms or
  overturns it
- Impact, root cause, and prevention are
  the three things that can't be faked

30-DAY PRACTICE PLAN (FOR CANDIDATES):

Week 1 — Recall the failures
☐ List 5 models that failed in the last
  2 years
☐ For each, write: the model, the
  decision, the impact
☐ Pick the 3 with the clearest impact
☐ Record yourself telling each in
  90 seconds
☐ Cut every "I don't remember" and
  "it didn't work out"

Week 2 — Root causes
☐ For each story, write the root cause
  in one sentence
☐ Classify it: validation error, leak,
  drift, or definition error
☐ Write how you found it: baseline,
  distribution, timestamp, shadow
☐ Write the fix and why it addresses
  the cause

Week 3 — Prevention and evidence
☐ For each story, write the prevention
  and what it targets
☐ Write how you knew the fix worked
☐ Write one reusable lesson or rule
☐ Add the cost of the fix or prevention

Week 4 — Mock interviews
☐ Run 2 mock interviews with a peer
☐ Ask them to challenge: "How did you
  know?" and "What was the root cause?"
☐ Re-run with stronger evidence
☐ Compare the two recordings

Signs it's working:
✅ You can name the impact without
  pausing
✅ You name a validation error or a leak,
  not "it didn't work out"
✅ You name how you found it
✅ You name how you knew the fix worked
✅ You name a reusable lesson
✅ The interviewer asks "what else?"
  instead of "what was the model?"