Machine Learning Engineers build, train and ship models that run inside real products, such as fraud checks, recommendations and pricing. They write clean Python, design leak-proof pipelines, tune models like LightGBM, and work with backend and data teams to serve predictions through APIs or batch jobs. They own model quality after release too, monitoring drift and latency and retraining when needed.
Python and scikit-learn Gradient boosting tuning FastAPI model serving Docker and MLflow Model monitoring
Solved for this roleStart practicing
3 QUESTIONS FREESign up to unlock the full question bank and track your progress.Unlock every question and answer, then continue practising company-wise and package-wise.
Start with the right metrics, such as PR-AUC, recall at fixed precision or F1, not accuracy. Then use class_weight='balanced', resampling like random undersampling or SMOTE applied only on training folds, and threshold tuning based on business cost. Boosting models with scale_pos_weight also help. Stratified splits keep class ratios consistent across folds.
03
Your random forest model file is 2 GB and too slow for real-time scoring. What can you do?
Reduce size by limiting max_depth and min_samples_leaf, using fewer trees since gains flatten after a few hundred, and dropping weak features. Alternatively switch to a gradient boosting model like LightGBM, which usually gives similar accuracy with far smaller, faster models. Converting to ONNX or using a compiled predictor can also speed up inference. Measure accuracy and latency after each change.
Use statsmodels on the numeric feature matrix with a constant added.
import pandas as pd
from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools import add_constant
Xc = add_constant(X_num)
vif = pd.Series([variance_inflation_factor(Xc.values, i)
for i in range(Xc.shape[1])], index=Xc.columns)
print(vif.drop('const'))
Values above 5 to 10 indicate strong multicollinearity.
05
Your cross-validation score is 0.85 but the test score is 0.70. What could explain it?
Possible causes: leakage inside CV, such as preprocessing or feature selection fit outside folds; repeated tuning that overfit the CV folds; a test set from a different time period or population; or duplicate entities across folds but not test. I would check pipeline structure, switch to time-based or group splits that match how test data differs, and compare feature distributions between the two sets.
06
What are shadow deployment and canary release for models?
In shadow deployment the new model receives a copy of live traffic and its predictions are logged but not used, so you can compare it with the current model safely. In a canary release a small share of real traffic, such as 5 percent, is served by the new model, and the share grows if metrics stay healthy. Both reduce the risk of a bad model affecting all users.
07
Which hyperparameters matter most when tuning a gradient boosting model?
The key ones are learning_rate and number of trees, tuned together with early stopping; tree complexity through max_depth or num_leaves and min_child_samples; and randomness through subsample and colsample_bytree. Regularisation terms like lambda and alpha help on noisy data. A common approach is to fix a small learning rate such as 0.05, use early stopping for trees, then tune complexity.
08
Random forest and LightGBM give the same AUC. Which would you deploy?
I would compare beyond AUC: model size, inference latency, training time, calibration and stability across time periods. LightGBM is usually smaller and faster to score, while random forest needs less tuning and is less prone to overfitting. If both are equal, I would choose the one that is cheaper to run and easier for the team to maintain and retrain.
09
How do you decide when a model is good enough to stop tuning?
I agree on a target metric and business threshold before starting, such as recall of 70 percent at 90 percent precision. I stop tuning when gains fall below the noise between CV folds or when further work costs more than the expected value. In one project I stopped at a 0.3 percent gain and spent the time on monitoring instead, which proved more useful.
10
When should you use TimeSeriesSplit or GroupKFold instead of normal k-fold?
Use TimeSeriesSplit when data is ordered in time, so each validation fold comes after its training data and no future information leaks back. Use GroupKFold when multiple rows belong to the same entity, such as several visits per patient, so the same entity never appears in both train and validation. Normal k-fold in these cases gives over-optimistic scores.
11
XGBoost validation loss starts rising after 300 rounds while training loss keeps falling. What do you do?
That is overfitting beyond about 300 rounds. I would use early stopping with a patience such as 50 rounds on a validation set so training stops at the best iteration, then lower the learning rate, reduce max_depth, add subsample and colsample_bytree below 1, and increase min_child_weight or lambda. The final model is refit using the best iteration count found.
Batch scoring runs on a schedule, for example nightly churn scores for all customers written to a table; it is simple, cheap and tolerant of slow models. Real-time inference returns predictions on request, such as fraud checks during payment, and needs low latency, high availability, scaling and fresh features. Choose real-time only when the decision must happen at the moment of the event.
13
Labels for your loan model arrive only after 6 months. How do you monitor it meanwhile?
I would rely on proxy signals: input data drift with PSI, prediction score distribution, approval rates by segment, and early indicators like first missed payment at 30 or 60 days. Once enough early outcomes arrive, I would check performance on these. I would also set up a delayed evaluation job that computes full metrics as labels mature and compares them with the validation baseline.
14
Check calibration with a reliability curve and Brier score.
from sklearn.calibration import CalibrationDisplay
from sklearn.metrics import brier_score_loss
proba = clf.predict_proba(X_test)[:, 1]
print('Brier', brier_score_loss(y_test, proba))
CalibrationDisplay.from_predictions(y_test, proba, n_bins=10)
plt.show()
Points on the diagonal mean predicted probabilities match observed frequencies.
from sklearn.model_selection import LearningCurveDisplay
import numpy as np
LearningCurveDisplay.from_estimator(model, X, y, cv=5,
train_sizes=np.linspace(0.1, 1.0, 6), scoring='roc_auc', n_jobs=-1)
plt.show()
A persistent gap between curves suggests more data or regularisation; two low curves suggest a richer model or better features.
I would first check data pipelines for broken features, nulls or schema changes, then compare current feature distributions with training data using PSI to find drift. I would check whether the label definition or business process changed, and analyse errors by segment. If drift is real, retrain on recent data, possibly with time-weighted samples, and set up automatic drift alerts and scheduled retraining.
19
A new model has slightly better AUC but worse precision in the top 1,000 cases the team acts on. Which do you pick?
I would pick the model that does better where decisions are actually made. If the team only reviews the top 1,000 cases, precision at 1,000 or recall at that cut-off matters more than global AUC. I would make that the primary metric in future comparisons, and confirm the result holds across several time periods before switching.
20
An A/B test of a new recommendation model shows no significant uplift despite better offline metrics. What next?
I would check the test first: enough sample size and duration, correct randomisation, and no bugs in serving such as stale features. Then investigate whether offline metrics really reflect user value, for example optimising clicks rather than purchases, or popularity bias. Segment-level analysis may show gains in some groups. I would revise the offline metric to better match the online goal.
21
Tell me about a time you took a model from notebook to production.
Situation: our demand model ran only in a notebook on my laptop. Task: make it run daily for 300 stores. Action: I refactored it into modules with a scikit-learn pipeline, added tests and input checks, containerised it with Docker and scheduled it with Airflow writing forecasts to the warehouse. Result: forecasts became available by 6 a.m. daily without manual work, and failures triggered alerts instead of silent gaps.
A feature store is a system that defines, computes and serves features consistently for both training and real-time inference, with an offline store for historical data and an online store for low-latency lookup. It prevents training-serving skew and lets teams reuse features. It is worth it when many models share features or need real-time features; small teams can start with well-managed tables.
23
What is boosting and how is it different from bagging?
Boosting trains models sequentially, each one focusing on the errors of the combined previous models, which mainly reduces bias. Bagging trains models independently in parallel and averages them, which mainly reduces variance. Boosting uses shallow trees and a learning rate, and can overfit if run too long, so early stopping is important. Examples are AdaBoost, Gradient Boosting, XGBoost, LightGBM and CatBoost.
Gradient boosting starts with a simple prediction, like the mean, then repeatedly fits a small tree to the negative gradient of the loss, which for MSE is just the residuals. Each tree's output is scaled by a learning rate and added to the ensemble. A lower learning rate with more trees usually generalises better. Any differentiable loss can be used, which makes it flexible.
25
Your model must run on a mobile app with no internet. What do you consider?
Model size, memory and latency on low-end phones become key. I would choose a compact model, reduce features, and convert it to a mobile format such as ONNX, TensorFlow Lite or Core ML, applying quantisation if needed. I would make sure all preprocessing is replicated on-device, test accuracy after conversion, and plan how model updates will ship with app releases.
26
When would you use AdaBoost and how does it re-weight samples?
AdaBoost trains weak learners, usually decision stumps, in sequence. After each round it increases the weights of misclassified samples so the next learner focuses on them, and gives each learner a vote weight based on its accuracy. It works well on clean data but is sensitive to noisy labels and outliers, which get ever larger weights. Gradient boosting is a generalisation that has largely replaced it.
from sklearn.inspection import permutation_importance
import pandas as pd
r = permutation_importance(model, X_val, y_val, n_repeats=10,
scoring='roc_auc', random_state=42, n_jobs=-1)
imp = pd.Series(r.importances_mean, index=X_val.columns)
print(imp.sort_values(ascending=False))
Near-zero or negative values mean the feature adds little on unseen data.
29
Your team retrains models manually in notebooks every month. How would you improve it?
I would convert the notebook into a versioned training pipeline with separate steps for data extraction, validation, feature building, training and evaluation, scheduled with Airflow or a cloud pipeline. Each run would log to MLflow, compare against the production model on a fixed holdout, and register the new model only if it passes thresholds. Promotion would stay a reviewed step, at least initially.
30
Your manager asks for a single number showing the model's business value. What do you give?
I would convert model outcomes into money or time using a cost matrix agreed with the business, for example savings from caught fraud minus the cost of reviewing false alerts. I would compare this against the current process or a rule-based baseline on holdout data and show a range rather than one point. A short note on assumptions keeps the number honest.
A model registry is a central store of model versions with metadata: training data version, parameters, metrics, owner and stage such as staging or production. It makes promotions and rollbacks controlled and auditable, and lets serving systems load a model by name and version. MLflow Model Registry, SageMaker Model Registry and Vertex AI Model Registry are common examples.
import lightgbm as lgb
m = lgb.LGBMClassifier(n_estimators=5000, learning_rate=0.05,
num_leaves=31, subsample=0.8, subsample_freq=1, colsample_bytree=0.8)
m.fit(X_train, y_train, eval_set=[(X_val, y_val)],
eval_metric='auc', callbacks=[lgb.early_stopping(100)])
print(m.best_iteration_)
The large n_estimators is only an upper limit; early stopping picks the real number.
33
What should you monitor for a model in production?
Monitor service health such as latency, error rate and throughput; input data quality such as missing values, schema changes and out-of-range values; feature and prediction distribution drift; and real performance once labels arrive. Also track business KPIs the model influences. Set alerts with thresholds and owners, and keep a dashboard so issues are found before users report them.
Data drift, or covariate shift, means the distribution of input features changes, such as average order value rising after a pricing change. Concept drift means the relationship between features and target changes, such as fraud patterns evolving. Data drift can be detected without labels using PSI or KS tests; concept drift usually needs labels and shows up as falling performance. Both trigger investigation and retraining.
35
What is the difference between offline evaluation and online A/B testing of a model?
Offline evaluation measures metrics like AUC on historical holdout data; it is fast and cheap but cannot capture user behaviour changes or feedback loops. Online A/B testing exposes a share of real traffic to the new model and measures business outcomes like conversion or loss rate. Offline gains do not always translate online, so important models are validated with A/B tests or shadow deployments.
36
Explain the bias in AUC when classes are extremely imbalanced, with a numeric intuition.
With 100 fraud and 100,000 genuine cases, a false positive rate of 1 percent looks tiny on the ROC curve, but it means 1,000 false alerts versus at most 100 true ones, so precision is below 10 percent. ROC-AUC can be 0.97 while the model is impractical. Precision-recall curves expose this because precision directly depends on the count of false positives.
37
Describe a time you found a serious data quality issue late in a project.
Situation: two days before a demo, I noticed sales for one region were doubled. Task: fix it without missing the deadline. Action: I traced it to a join that duplicated rows on a non-unique key, fixed the join, added row-count assertions to the pipeline and retrained. I informed my lead immediately with the impact. Result: the demo went ahead with correct numbers, and the assertions caught a similar issue later.
38
What makes XGBoost, LightGBM and CatBoost different?
XGBoost adds regularisation to the objective, uses second-order gradients and handles missing values natively. LightGBM grows trees leaf-wise and uses histogram binning, making it very fast on large data, though it can overfit small data. CatBoost handles categorical features with ordered target statistics and uses symmetric trees, giving strong defaults with less tuning. All three support GPU training.
39
Describe a project where you improved a model's performance significantly.
Situation: a product-return prediction model had 0.68 AUC with logistic regression. Task: improve it for a warehouse pilot. Action: I engineered features like past return rate per customer and category, moved to LightGBM with early stopping, and tuned it with Optuna on time-based folds. Result: AUC rose to 0.81, and the pilot reduced avoidable returns in two categories. I documented each step's gain.
40
A new model version performs badly right after release. How do you handle it?
Roll back immediately to the previous version from the model registry, which should be a one-step action. Then investigate: compare the input features served with those used in validation, check for training-serving skew, version mismatches or a wrong artifact. Add the failure case to automated pre-release checks, and use canary or shadow releases next time so issues affect only a small share of traffic.
Frequently asked questions
What does a Machine Learning Engineer do?
Machine Learning Engineers build, train and ship models that run inside real products, such as fraud checks, recommendations and pricing. They write clean Python, design leak-proof pipelines, tune models like LightGBM, and work with backend and data teams to serve predictions through APIs or batch jobs. They own model quality after release too, monitoring drift and latency and retraining when needed.
Which skills are tested in a Machine Learning Engineer interview?
Expect questions on Python and scikit-learn, Gradient boosting tuning, FastAPI model serving, Docker and MLflow, Model monitoring. Most rounds also ask you to explain a project and solve one practical problem live.
How do I prepare for a Machine Learning Engineer interview?
Revise Decision Trees and Ensembles, Model Evaluation and Tuning, MLOps and Deployment first using the topic sheets, practise the questions on this page aloud, then do one full mock interview to practise timing.