Model Evaluation and Tuning Interview Questions for Machine Learning
25 Model Evaluation and Tuning questions with answers for machine learning roles — 5 easy, 11 medium, 9 hard.
About Model Evaluation and Tuning
A model is only as good as its evaluation. This topic covers k-fold, stratified, group and time series cross-validation, nested CV, learning curves, grid search, random search and Optuna, log loss and calibration, bootstrap confidence intervals, fair model comparison, fairness checks and the gap between offline metrics and online A/B test results.
Sheet progressStart practicing
3 QUESTIONS FREESign up to unlock the full question bank and track your progress.Unlock every question and answer, then continue practising company-wise and package-wise.
✓ 3 questions free • Unlock the rest in seconds
01
What is k-fold cross-validation and why is it better than a single split?
k-fold cross-validation splits training data into k folds, trains on k-1 folds and validates on the remaining one, repeating k times so every row is validated once. Averaging the k scores gives a more stable estimate than one random split and shows variance across folds. Five or ten folds are common. Use StratifiedKFold for classification and GroupKFold when rows from one user must stay together.
02
What is a holdout set and why should it be touched only once?
A holdout or test set is kept aside for the final evaluation after all model and hyperparameter choices are done. If you check it repeatedly and adjust the model, you are indirectly tuning on it, so its score becomes optimistic. Use cross-validation or a separate validation set for decisions, and report the holdout score once as the realistic estimate.
03
Run stratified 5-fold cross-validation with multiple metrics.
from sklearn.model_selection import StratifiedKFold, cross_validate
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
res = cross_validate(pipe, X, y, cv=cv,
scoring=['roc_auc', 'average_precision', 'f1'], n_jobs=-1)
for k in ['test_roc_auc', 'test_average_precision', 'test_f1']:
print(k, res[k].mean().round(3), res[k].std().round(3))
I use their own example. For fraud: 'If we flag more transactions, we catch more fraud, which is recall, but more genuine customers get blocked, which hurts precision. Here is a table of what each threshold means per day: frauds caught, customers disturbed, reviews needed.' Then we choose the threshold together based on cost. Managers relate much better to counts than to percentages.
06
When should you use TimeSeriesSplit or GroupKFold instead of normal k-fold?
Use TimeSeriesSplit when data is ordered in time, so each validation fold comes after its training data and no future information leaks back. Use GroupKFold when multiple rows belong to the same entity, such as several visits per patient, so the same entity never appears in both train and validation. Normal k-fold in these cases gives over-optimistic scores.
07
Compare grid search, random search and Bayesian optimisation.
Grid search tries every combination of listed values; it is thorough but expensive and wastes trials on unimportant parameters. Random search samples combinations and usually finds good settings faster in high-dimensional spaces. Bayesian optimisation, as in Optuna or Hyperopt, builds a model of past trials to pick promising settings next, using fewer trials. Successive halving methods add early pruning of weak trials.
A learning curve plots training and validation scores against training set size; if both converge at a low score the model underfits, and if a gap remains, more data may help. A validation curve plots scores against one hyperparameter, such as max_depth, showing where overfitting starts. Both help decide whether to collect data, add features or change complexity.
Log loss, or cross-entropy, is the negative average log of the probability assigned to the true class. It heavily penalises confident wrong predictions, such as predicting 0.99 for an event that does not happen. It evaluates probability quality, not just ranking, so it suits problems where probabilities drive decisions. Lower is better, and a constant base-rate prediction is the baseline to beat.
10
What is the difference between offline evaluation and online A/B testing of a model?
Offline evaluation measures metrics like AUC on historical holdout data; it is fast and cheap but cannot capture user behaviour changes or feedback loops. Online A/B testing exposes a share of real traffic to the new model and measures business outcomes like conversion or loss rate. Offline gains do not always translate online, so important models are validated with A/B tests or shadow deployments.
11
A new model has slightly better AUC but worse precision in the top 1,000 cases the team acts on. Which do you pick?
I would pick the model that does better where decisions are actually made. If the team only reviews the top 1,000 cases, precision at 1,000 or recall at that cut-off matters more than global AUC. I would make that the primary metric in future comparisons, and confirm the result holds across several time periods before switching.
12
Grid search with 5 hyperparameters is taking two days. How do you speed it up?
Switch to random search or Optuna with pruning, which find good settings with far fewer trials. Narrow ranges using a quick coarse search, tune the most important parameters first, use early stopping for boosting, run trials in parallel with n_jobs, and tune on a stratified sample of the data before a final refit on the full set. HalvingRandomSearchCV is another quick option.
13
Your manager asks for a single number showing the model's business value. What do you give?
I would convert model outcomes into money or time using a cost matrix agreed with the business, for example savings from caught fraud minus the cost of reviewing false alerts. I would compare this against the current process or a rule-based baseline on holdout data and show a range rather than one point. A short note on assumptions keeps the number honest.
from sklearn.model_selection import LearningCurveDisplay
import numpy as np
LearningCurveDisplay.from_estimator(model, X, y, cv=5,
train_sizes=np.linspace(0.1, 1.0, 6), scoring='roc_auc', n_jobs=-1)
plt.show()
A persistent gap between curves suggests more data or regularisation; two low curves suggest a richer model or better features.
15
Check calibration with a reliability curve and Brier score.
from sklearn.calibration import CalibrationDisplay
from sklearn.metrics import brier_score_loss
proba = clf.predict_proba(X_test)[:, 1]
print('Brier', brier_score_loss(y_test, proba))
CalibrationDisplay.from_predictions(y_test, proba, n_bins=10)
plt.show()
Points on the diagonal mean predicted probabilities match observed frequencies.
16
Tell me about a time you discovered your own mistake after sharing results.
Situation: after presenting a 12 percent uplift from a pricing model, I found my evaluation used a random split on time-series data. Task: correct it quickly. Action: I re-ran with a time-based split, which showed 7 percent, and informed my manager the same day with the corrected analysis. Result: the team appreciated the honesty, the rollout plan used the right number, and I added split checks to our review template.
17
What is nested cross-validation and when do you need it?
Nested CV has an inner loop that tunes hyperparameters and an outer loop that evaluates the whole tuning process on unseen folds. It gives an unbiased performance estimate when you tune heavily on small data, because a normal CV score used both to tune and to report is optimistic. It is expensive, so it is mainly used for small datasets or research reporting.
Use the same data splits and folds, the same preprocessing inside pipelines and the same metric tied to the business goal. Look at the mean and standard deviation across folds, not one number, and use a paired test like a corrected t-test or bootstrap on the fold differences. Also compare cost factors: latency, memory, interpretability and maintenance.
19
What fairness checks would you run on a model that affects people?
Compare metrics across groups such as gender, age band or region: approval rate (demographic parity), true positive rate (equal opportunity) and false positive rate. Check calibration per group too. Remove or audit proxy features like pin code that may stand in for protected attributes. Document findings in a model card, and involve legal or compliance teams, especially for lending and hiring use cases.
20
Your cross-validation score is 0.85 but the test score is 0.70. What could explain it?
Possible causes: leakage inside CV, such as preprocessing or feature selection fit outside folds; repeated tuning that overfit the CV folds; a test set from a different time period or population; or duplicate entities across folds but not test. I would check pipeline structure, switch to time-based or group splits that match how test data differs, and compare feature distributions between the two sets.
21
An A/B test of a new recommendation model shows no significant uplift despite better offline metrics. What next?
I would check the test first: enough sample size and duration, correct randomisation, and no bugs in serving such as stale features. Then investigate whether offline metrics really reflect user value, for example optimising clicks rather than purchases, or popularity bias. Segment-level analysis may show gains in some groups. I would revise the offline metric to better match the online goal.
I explain the risk clearly: inflated numbers lead to bad decisions and lose trust when production results arrive. I present the honest metric with confidence intervals, and then offer a concrete plan to improve it, such as new data sources. In one case I showed that a 'better' number came from leakage, and leadership agreed to report the lower but reliable figure.
25
Explain the bias in AUC when classes are extremely imbalanced, with a numeric intuition.
With 100 fraud and 100,000 genuine cases, a false positive rate of 1 percent looks tiny on the ROC curve, but it means 1,000 false alerts versus at most 100 true ones, so precision is below 10 percent. ROC-AUC can be 0.97 while the model is impractical. Precision-recall curves expose this because precision directly depends on the count of false positives.
Frequently asked questions
How important is Model Evaluation and Tuning in Machine Learning interviews?
A model is only as good as its evaluation. This topic covers k-fold, stratified, group and time series cross-validation, nested CV, learning curves, grid search, random search and Optuna, log loss and calibration, bootstrap confidence intervals, fair model comparison, fairness checks and the gap between offline metrics and online A/B test results.
How should I use the Model Evaluation and Tuning sheet?
Go top to bottom — easy to hard. Answer before opening each answer, tick what you solved and save misses to Last-Minute Revision.