Interview Insights · ML Engineer
The One Question That Separates Real ML Engineers From Pretenders
Quick summary — Real vs pretend ML
Asking what you do when a deployed model gets worse separates people who have owned models from people who have only trained them. Pretenders jump straight to 'retrain the model'. Real ML engineers start by confirming the drop and scoping it, then check the data pipeline, because broken inputs are a very common cause. Only after that do they look for data drift or concept drift, and they decide between fixing the pipeline, retraining or rolling back based on evidence. This order cannot be faked, because it comes from having debugged something.
In this guide you will learn:
- Why this question is so hard to bluff
- The triage order real ML engineers follow
- Pipeline checks in pandas you can show in an interview
- How to tell data drift from concept drift with simple tests
- When to fix, retrain or roll back and how to explain the choice
SECTION 01Why this one question is so hard to fake
Most interview questions have a textbook answer. 'What is regularisation?' 'Explain the bias-variance trade-off.' Anyone with a few weeks of preparation can recite those. The degraded-model question is different because it has no single answer. It asks for a process, and a process only comes from having been stuck with a real problem.
The giveaway is the first move. A typical weak answer starts with 'I would retrain the model on new data.' That sounds proactive, but retraining on broken data can make things worse, and it hides the cause. A strong answer starts with 'First I would confirm the drop is real and find out where it is happening.'
| Answer pattern | Example first sentence | What it suggests | How I follow up |
|---|---|---|---|
| Retrain immediately | I would retrain on the latest data | Has not debugged a production model | What if the new data is broken? |
| Swap the model | Maybe try a more powerful model | Treats every problem as a modelling problem | How do you know the model is the cause? |
| Vocabulary only | This is concept drift | Knows terms, not the process | How would you prove that? |
| Owner's process | First I would confirm the drop and check the inputs | Has thought about real systems | Great, what do you check in the inputs? |
Q: Your fraud model's recall dropped last week.
What do you do?
A: I would retrain the model with the latest data, maybe
try XGBoost instead of random forest, and tune the
hyperparameters again. I could also add more features.
(Every action changes the model. None of them finds out
why it got worse.)Q: Your fraud model's recall dropped last week.
What do you do?
A: First I would confirm it. Are labels for last week
complete yet, or are some frauds not reported? Is the drop
across all merchants or one segment?
Then I would check the input pipeline: schema changes,
missing values, row counts, a feature suddenly constant.
If the pipeline is fine, I would compare feature
distributions with training data, and check whether fraud
patterns themselves changed. Only then decide: fix the
pipeline, retrain, or roll back to the previous model.The owner answer contains no advanced theory. It is just the right order of questions. That is what makes it so convincing, and so hard to produce from memorised notes.
SECTION 02Check the pipeline before you blame the model
In many real incidents, the model did not get worse at all. The data feeding it did. An upstream team renamed a column, a join started dropping rows, a currency changed, or a feature that used to be filled is now mostly missing. The model receives garbage and produces garbage, faithfully.
That is why a real ML engineer checks the pipeline first. The checks are simple and fast: compare the schema, row counts, missing rates and basic ranges of today's input with a reference sample from training. The pandas script below is the kind of thing you can describe in an interview, and run in a real job.
| Check | What can break | Simple signal | Typical cause |
|---|---|---|---|
| Schema | Column renamed, dropped or new type | Columns differ from reference | Upstream table change |
| Row volume | Many fewer or more rows | Count far from usual | Failed join or duplicated load |
| Missing rate | A feature becomes mostly empty | Null share jumps | Source system stopped sending it |
| Value range | Units or encoding changed | Mean or max far from reference | Rupees became paise, new category codes |
import pandas as pd
def pipeline_report(ref: pd.DataFrame, cur: pd.DataFrame) -> list:
issues = []
missing_cols = set(ref.columns) - set(cur.columns)
new_cols = set(cur.columns) - set(ref.columns)
if missing_cols:
issues.append(f"missing columns: {sorted(missing_cols)}")
if new_cols:
issues.append(f"new columns: {sorted(new_cols)}")
for col in ref.columns.intersection(cur.columns):
r_null, c_null = ref[col].isna().mean(), cur[col].isna().mean()
if c_null - r_null > 0.10:
issues.append(f"{col}: null rate {r_null:.0%} -> {c_null:.0%}")
if pd.api.types.is_numeric_dtype(ref[col]) and pd.api.types.is_numeric_dtype(cur[col]):
r_med, c_med = ref[col].median(), cur[col].median()
if r_med and abs(c_med / r_med - 1) > 0.5:
issues.append(f"{col}: median {r_med:.1f} -> {c_med:.1f}")
return issues
ref = pd.DataFrame({"amount": [500, 800, 1200, 300], "device": ["a", "b", "a", "c"]})
cur = pd.DataFrame({"amount": [50000, 80000, None, 30000], "device_type": ["a", "b", "a", "c"]})
for line in pipeline_report(ref, cur):
print(line)missing columns: ['device']
new columns: ['device_type']
amount: null rate 0% -> 25%
amount: median 650.0 -> 50000.0
How I would explain this in the interview:
"The device column was renamed upstream, so the model is
receiving no device feature at all. The amount median jumped
about a hundred times, which looks like a unit change, maybe
rupees to paise. Both are pipeline bugs. Retraining on this
data would teach the model the bug. I would fix the inputs
first and re-score last week."The thresholds in the script, such as a ten-point rise in null rate, are illustrative. In a real system you would agree on them with the team and tune them so they catch real problems without constant false alarms.
SECTION 03Data drift, concept drift and the final decision
If the pipeline is healthy, the next question is whether the world changed. There are two kinds of change, and telling them apart is what makes an answer convincing. Data drift means the inputs look different: new kinds of customers, a new region, a seasonal shift. The relationship between inputs and outcome may still hold. Concept drift means the relationship itself changed: fraudsters adopted a new pattern, so the same inputs now mean something different.
The tests are different too. Data drift can be checked without labels, by comparing feature distributions. Concept drift needs labels, because you have to see that the model's errors grew even on familiar-looking inputs. The decision follows from the diagnosis.
| Diagnosis | How you detect it | Usual action | What to say |
|---|---|---|---|
| Pipeline bug | Schema, null or range checks fail | Fix the inputs, re-score, no retraining | The model was fine, the data was not |
| Data drift only | Feature distributions differ, error by segment stable | Monitor, possibly retrain to cover new segments | Inputs moved, relationship holds |
| Concept drift | Errors grow even on similar inputs, needs labels | Retrain on recent labelled data, review features | The pattern itself changed |
| Bad new release | Drop starts exactly at a deployment | Roll back to the previous model, then investigate | Safest move first, root cause after |
import numpy as np
from scipy.stats import ks_2samp
rng = np.random.default_rng(3)
# Data drift check: no labels needed
train_amount = rng.lognormal(mean=6.5, sigma=0.6, size=5000)
week_amount = rng.lognormal(mean=6.9, sigma=0.6, size=1500)
stat, p = ks_2samp(train_amount, week_amount)
print(f"KS statistic={stat:.3f}, p-value={p:.2e}")
# Concept drift check: needs labels.
# Compare recall on recent weeks for the SAME segment.
weekly_recall = {"w1": 0.78, "w2": 0.77, "w3": 0.76, "w4": 0.61}
baseline = np.mean([weekly_recall[w] for w in ("w1", "w2", "w3")])
for week, r in weekly_recall.items():
flag = " <- investigate" if baseline - r > 0.05 else ""
print(week, r, flag)"Here is how I would decide.
If the drop starts exactly when we deployed a new version,
I roll back first. Users get the old, working model while
I investigate.
If pipeline checks fail, I fix the data and re-score. No
retraining needed.
If inputs drifted but errors per segment are stable, I keep
monitoring and plan a retrain that includes the new segment.
If errors grew even on familiar inputs, that is concept
drift. I retrain on recent labelled data, check whether old
features still make sense, and compare the new model against
the current one on the latest weeks before switching."The weekly recall numbers in the script are sample values to show the logic. A p-value from a KS test on large samples will often be tiny even for small shifts, so look at the size of the statistic and at whether the model's errors actually changed, not only at significance.
If you can walk through confirm, pipeline, drift and decide in about three minutes, you have answered the question that separates owners from pretenders.
SECTION 04Rehearse the degraded-model answer
Use these three tabs to prepare and practise. In Uncodemy mock interviews we often ask this question about the candidate's own project, so prepare it for yours, not only for a generic fraud model.
MODEL GOT WORSE: TRIAGE ORDER
1. CONFIRM
- Are labels complete for the period?
- Which segment, region, device, product?
- Did anything deploy at that time?
2. PIPELINE
- Schema, row counts, null rates, value ranges
- Upstream changes announced?
3. DRIFT
- Inputs vs training (no labels needed)
- Errors on similar inputs (labels needed)
4. DECIDE
- Deployment caused it -> roll back
- Data bug -> fix and re-score
- Data drift -> monitor, plan retrain
- Concept drift -> retrain, compare, switchMY PROJECT: _______________________
If this model got worse next month, the most likely causes:
1. Pipeline: __________________________________
2. Data drift: ________________________________
3. Concept drift: _____________________________
Features I would check first and why: ___________
How long until I have labels: __________________
What I would roll back to: _____________________
Who I would tell, and when: ____________________Ask a friend to play interviewer and push on each step:
- 'Labels take a month to arrive. Now what?'
- 'The pipeline checks all pass. Next?'
- 'Drift is only in one city. What do you do?'
- 'Retraining made it worse. Why might that be?'
- 'How would you stop this happening again?'
For the last one, talk about automated input checks,
alerts on prediction share, and a scheduled review.SECTION 05Habits that make your answer sound lived-in
Real experience shows in small details. These habits help your answer carry that weight even if your experience is from projects.
- Start with scope, not solutions — Ask which metric, which segment and since when before proposing anything. It is the habit of every experienced engineer.
- Say the cheap check first — Mention schema, null and volume checks before drift statistics. Cheap and likely beats clever and rare.
- Name the label delay — Point out that concept drift needs labels and say how long they take to arrive in your example. Few candidates mention it.
- Include communication — Say who you would inform and when. Owning a model includes keeping the team updated during an incident.
- End with prevention — Finish by naming one automated check you would add so the same issue is caught earlier next time.
SECTION 06Interview Q&A — Real vs pretend ML
Q1Your model's accuracy dropped in production. What do you check first?
I first confirm the drop is real, checking whether labels for the period are complete and which segments are affected, and whether anything was deployed at that time. Then I check the input pipeline for schema changes, missing values, row counts and value ranges before I consider drift or retraining.
Q2What is the difference between data drift and concept drift?
Data drift means the distribution of inputs changed, for example more users from a new region, while the input-output relationship may still hold. Concept drift means the relationship itself changed, so the same inputs now lead to different outcomes. Data drift can be checked without labels; concept drift needs labels.
Q3When would you roll back a model instead of retraining it?
When the drop lines up with a deployment of a new model or new feature code. Rolling back restores a known good state quickly for users, and then I can investigate the new version calmly without pressure.
Q4How would you detect drift without labels?
I would compare the distribution of key input features and of the model's predictions against a training reference, using simple statistics like a KS test or a stability index, and track the share of positive predictions over time. Large changes trigger a closer look.
Q5Retraining on new data made the model worse. Why might that happen?
The new data might contain a pipeline bug, such as a unit change or missing feature, so the model learned the bug. Labels might be incomplete for recent data, or the new window might be too short or unrepresentative. I would check data quality and label completeness before retraining again.
SECTION 07Test yourself — Real vs pretend ML quiz
Five questions. No sign-up.
0 / 5Pick an answer to see why it is right or wrong.
SECTION 08Frequently asked questions
What is the hardest question in an ML engineer interview?
Process questions with no textbook answer, such as 'your model got worse in production, what do you do?', are among the hardest because they test real debugging thinking rather than memorised definitions.
How do you debug a machine learning model in production?
Confirm and scope the drop, check the data pipeline for schema, volume, nulls and ranges, test for data drift and concept drift, then decide whether to fix inputs, retrain or roll back.
What is model drift in machine learning?
Model drift is a general term for a model's performance changing over time. It is usually caused by data drift, where inputs change, or concept drift, where the relationship between inputs and outcomes changes.
How often should a machine learning model be retrained?
It depends on how fast the data and patterns change. Many teams retrain on a schedule and also when monitoring shows drift or a drop in metrics. Always compare the new model with the current one before switching.
Can a fresher answer production ML questions without job experience?
Yes. Explain the triage order clearly and use your own project as the example. Breaking your project on purpose and catching the problem with simple checks gives you a real story to tell.
Is there a course that covers ML Engineer interview answers?
Yes. Uncodemy's Machine Learning Training Course in Noida (classroom and online) includes live projects, one-to-one mock interviews and resume reviews, so you practise ML Engineer interview answers with trainer feedback before the real interview. You can book a free demo class from this page to see how the sessions work.
SECTION 09Related reads
Classroom & online · Noida
Think like an ML engineer who owns the model
Our Machine Learning Training Course covers deployment, data checks, drift and retraining decisions, with mock interviews built around real production scenarios.
Talk to a counsellor for current fees- Production ML scenarios
- Data quality and drift labs
- Mock interviews with feedback
- Weekday & weekend batches