MLOps Engineer Interview Questions &amp; Answers

Show answer

Data drift, or covariate shift, means the distribution of input features changes, such as average order value rising after a pricing change. Concept drift means the relationship between features and target changes, such as fraud patterns evolving. Data drift can be detected without labels using PSI or KS tests; concept drift usually needs labels and shows up as falling performance. Both trigger investigation and retraining.

Compare batch scoring and real-time inference.

Show answer

Batch scoring runs on a schedule, for example nightly churn scores for all customers written to a table; it is simple, cheap and tolerant of slow models. Real-time inference returns predictions on request, such as fraud checks during payment, and needs low latency, high availability, scaling and fresh features. Choose real-time only when the decision must happen at the moment of the event.

How do you approach a time series forecasting problem differently from normal regression?

Regression ModelsMedium

Show answer

Time series data is ordered and autocorrelated, so random splits leak future information. I use time-based splits or TimeSeriesSplit, create lag and rolling-window features from past values only, and handle trend and seasonality. Baselines like last value or seasonal naive come first. Models range from ARIMA and Prophet to gradient boosting on lag features.

How do you compare two models fairly?

Show answer

Use the same data splits and folds, the same preprocessing inside pipelines and the same metric tied to the business goal. Look at the mean and standard deviation across folds, not one number, and use a paired test like a corrected t-test or bootstrap on the fold differences. Also compare cost factors: latency, memory, interpretability and maintenance.

What is probability calibration and why does it matter?

Classification ModelsHard

Show answer

A calibrated model's predicted probabilities match real frequencies: of cases scored 0.8, about 80 percent should be positive. It matters when scores drive decisions like credit limits or expected loss. Random forests and SVMs are often poorly calibrated. Fix it with CalibratedClassifierCV using Platt scaling (sigmoid) or isotonic regression, and check with a reliability curve or Brier score.

What is a feature store and when is it needed?

Show answer

A feature store is a system that defines, computes and serves features consistently for both training and real-time inference, with an offline store for historical data and an online store for low-latency lookup. It prevents training-serving skew and lets teams reuse features. It is worth it when many models share features or need real-time features; small teams can start with well-managed tables.

What is the difference between offline evaluation and online A/B testing of a model?

Clustering and Dimensionality ReductionHard

Show answer

Offline evaluation measures metrics like AUC on historical holdout data; it is fast and cheap but cannot capture user behaviour changes or feedback loops. Online A/B testing exposes a share of real traffic to the new model and measures business outcomes like conversion or loss rate. Offline gains do not always translate online, so important models are validated with A/B tests or shadow deployments.

You need to detect unusual transactions but have no fraud labels. What do you do?

Show answer

I would start with unsupervised anomaly detection: Isolation Forest or Local Outlier Factor on features like amount relative to the customer's usual spend, time of day, new device and merchant risk. I would set the threshold based on what the review team can handle, collect investigator feedback as labels, and move to a supervised model once enough labelled cases exist.

Your gradient boosting model performs worse on recent months than older ones. What do you check?

Decision Trees and EnsemblesHard

Show answer

I would check for drift between periods in feature distributions and target rate, and confirm that the evaluation uses a time-based split. If behaviour has changed, I would train on a more recent window or weight recent data more, add features that capture the change, and set up a regular retraining schedule. Comparing SHAP importance across periods shows which relationships have shifted.

When should you use TimeSeriesSplit or GroupKFold instead of normal k-fold?

Show answer

Use TimeSeriesSplit when data is ordered in time, so each validation fold comes after its training data and no future information leaks back. Use GroupKFold when multiple rows belong to the same entity, such as several visits per patient, so the same entity never appears in both train and validation. Normal k-fold in these cases gives over-optimistic scores.

How do you handle pressure to report better model numbers than the data supports?

Show answer

I explain the risk clearly: inflated numbers lead to bad decisions and lose trust when production results arrive. I present the honest metric with confidence intervals, and then offer a concrete plan to improve it, such as new data sources. In one case I showed that a 'better' number came from leakage, and leadership agreed to report the lower but reliable figure.

Get a bootstrap confidence interval for a test metric.

Show answer

import numpy as np from sklearn.metrics import roc_auc_score rng = np.random.default_rng(0); scores = [] yt, pp = np.array(y_test), np.array(proba) for _ in range(1000): i = rng.integers(0, len(yt), len(yt)) if len(np.unique(yt[i])) == 2: scores.append(roc_auc_score(yt[i], pp[i])) print(np.percentile(scores, [2.5, 97.5]))

Your manager asks for a single number showing the model's business value. What do you give?

Show answer

I would convert model outcomes into money or time using a cost matrix agreed with the business, for example savings from caught fraud minus the cost of reviewing false alerts. I would compare this against the current process or a rule-based baseline on holdout data and show a range rather than one point. A short note on assumptions keeps the number honest.

A new model version performs badly right after release. How do you handle it?

Show answer

Roll back immediately to the previous version from the model registry, which should be a one-step action. Then investigate: compare the input features served with those used in validation, check for training-serving skew, version mismatches or a wrong artifact. Add the failure case to automated pre-release checks, and use canary or shadow releases next time so issues affect only a small share of traffic.

Your cross-validation score is 0.85 but the test score is 0.70. What could explain it?

Show answer

Possible causes: leakage inside CV, such as preprocessing or feature selection fit outside folds; repeated tuning that overfit the CV folds; a test set from a different time period or population; or duplicate entities across folds but not test. I would check pipeline structure, switch to time-based or group splits that match how test data differs, and compare feature distributions between the two sets.

Print a confusion matrix and classification report in scikit-learn.

Classification ModelsEasy

Show answer

from sklearn.metrics import confusion_matrix, classification_report pred = clf.predict(X_test) print(confusion_matrix(y_test, pred)) print(classification_report(y_test, pred, digits=3)) The report shows precision, recall, F1 and support per class plus macro and weighted averages. ConfusionMatrixDisplay.from_estimator draws it as a plot.

What fairness checks would you run on a model that affects people?

Show answer

Compare metrics across groups such as gender, age band or region: approval rate (demographic parity), true positive rate (equal opportunity) and false positive rate. Check calibration per group too. Remove or audit proxy features like pin code that may stand in for protected attributes. Document findings in a model card, and involve legal or compliance teams, especially for lending and hiring use cases.

Plot a learning curve in scikit-learn.

Show answer

from sklearn.model_selection import LearningCurveDisplay import numpy as np LearningCurveDisplay.from_estimator(model, X, y, cv=5, train_sizes=np.linspace(0.1, 1.0, 6), scoring='roc_auc', n_jobs=-1) plt.show() A persistent gap between curves suggests more data or regularisation; two low curves suggest a richer model or better features.

Log a training run to MLflow.

Show answer

import mlflow, mlflow.sklearn mlflow.set_experiment('churn') with mlflow.start_run(): pipe.fit(X_train, y_train) auc = roc_auc_score(y_val, pipe.predict_proba(X_val)[:, 1]) mlflow.log_params({'C': 1.0, 'model': 'logreg'}) mlflow.log_metric('val_auc', auc) mlflow.sklearn.log_model(pipe, 'model', registered_model_name='churn-model')

A model's accuracy dropped 10 percent over three months in production. How do you investigate?

Show answer

I would first check data pipelines for broken features, nulls or schema changes, then compare current feature distributions with training data using PSI to find drift. I would check whether the label definition or business process changed, and analyse errors by segment. If drift is real, retrain on recent data, possibly with time-weighted samples, and set up automatic drift alerts and scheduled retraining.

What is training-serving skew and how do you prevent it?

Show answer

Training-serving skew is when features at prediction time are computed differently from features used in training, causing silent accuracy loss. Causes include separate code paths, different data sources, or time windows that use future data in training. Prevent it by sharing one feature-computation code path or a feature store, logging served features and comparing their distributions with training data.

Tell me about a time you took a model from notebook to production.

Show answer

Situation: our demand model ran only in a notebook on my laptop. Task: make it run daily for 300 stores. Action: I refactored it into modules with a scikit-learn pipeline, added tests and input checks, containerised it with Docker and scheduled it with Airflow writing forecasts to the warehouse. Result: forecasts became available by 6 a.m. daily without manual work, and failures triggered alerts instead of silent gaps.

Write a simple Dockerfile for an ML API.

Show answer

FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY app.py model_v1.joblib ./ EXPOSE 8000 CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"] The exec-form CMD avoids a shell wrapper. Pin versions in requirements.txt so training and serving match.

What is stacking and when is it worth it?

Decision Trees and EnsemblesHard

Show answer

Stacking trains several diverse base models and then a meta-model on their out-of-fold predictions to combine them. It often gives a small accuracy gain, common in competitions. In production it adds complexity, latency and maintenance, so it is worth it only when the gain has clear business value. scikit-learn provides StackingClassifier and StackingRegressor.

An A/B test of a new recommendation model shows no significant uplift despite better offline metrics. What next?

Show answer

I would check the test first: enough sample size and duration, correct randomisation, and no bugs in serving such as stale features. Then investigate whether offline metrics really reflect user value, for example optimising clicks rather than purchases, or popularity bias. Segment-level analysis may show gains in some groups. I would revise the offline metric to better match the online goal.

A new model has slightly better AUC but worse precision in the top 1,000 cases the team acts on. Which do you pick?

Show answer

I would pick the model that does better where decisions are actually made. If the team only reviews the top 1,000 cases, precision at 1,000 or recall at that cut-off matters more than global AUC. I would make that the primary metric in future comparisons, and confirm the result holds across several time periods before switching.

Labels for your loan model arrive only after 6 months. How do you monitor it meanwhile?

Show answer

I would rely on proxy signals: input data drift with PSI, prediction score distribution, approval rates by segment, and early indicators like first missed payment at 30 or 60 days. Once enough early outcomes arrive, I would check performance on these. I would also set up a delayed evaluation job that computes full metrics as labels mature and compares them with the validation baseline.

What is the difference between a generative and a discriminative model?

ML FundamentalsHard

Show answer

A discriminative model learns the decision boundary or P(y|x) directly, such as logistic regression or SVM. A generative model learns how data is produced, P(x|y) and P(y), and can generate new samples, such as Naive Bayes or Gaussian mixture models. Discriminative models usually give better accuracy with enough data, while generative models handle missing data and small samples better.

Your model must run on a mobile app with no internet. What do you consider?

Show answer

Model size, memory and latency on low-end phones become key. I would choose a compact model, reduce features, and convert it to a mobile format such as ONNX, TensorFlow Lite or Core ML, applying quantisation if needed. I would make sure all preprocessing is replicated on-device, test accuracy after conversion, and plan how model updates will ship with app releases.

Check calibration with a reliability curve and Brier score.

Show answer

from sklearn.calibration import CalibrationDisplay from sklearn.metrics import brier_score_loss proba = clf.predict_proba(X_test)[:, 1] print('Brier', brier_score_loss(y_test, proba)) CalibrationDisplay.from_predictions(y_test, proba, n_bins=10) plt.show() Points on the diagonal mean predicted probabilities match observed frequencies.

What is nested cross-validation and when do you need it?

Show answer

Nested CV has an inner loop that tunes hyperparameters and an outer loop that evaluates the whole tuning process on unseen folds. It gives an unbiased performance estimate when you tune heavily on small data, because a normal CV score used both to tune and to report is optimistic. It is expensive, so it is mainly used for small datasets or research reporting.

Grid search with 5 hyperparameters is taking two days. How do you speed it up?

Show answer

Switch to random search or Optuna with pruning, which find good settings with far fewer trials. Narrow ranges using a quick coarse search, tune the most important parameters first, use early stopping for boosting, run trials in parallel with n_jobs, and tune on a stratified sample of the data before a final refit on the full set. HalvingRandomSearchCV is another quick option.

What are shadow deployment and canary release for models?

Show answer

In shadow deployment the new model receives a copy of live traffic and its predictions are logged but not used, so you can compare it with the current model safely. In a canary release a small share of real traffic, such as 5 percent, is served by the new model, and the share grows if metrics stay healthy. Both reduce the risk of a bad model affecting all users.

What should you monitor for a model in production?

Show answer

Monitor service health such as latency, error rate and throughput; input data quality such as missing values, schema changes and out-of-range values; feature and prediction distribution drift; and real performance once labels arrive. Also track business KPIs the model influences. Set alerts with thresholds and owners, and keep a dashboard so issues are found before users report them.

Tune LightGBM with Optuna.

Show answer

import optuna from sklearn.model_selection import cross_val_score def objective(trial): p = {'num_leaves': trial.suggest_int('num_leaves', 15, 255), 'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.2, log=True), 'min_child_samples': trial.suggest_int('min_child_samples', 10, 200)} m = lgb.LGBMClassifier(n_estimators=500, **p) return cross_val_score(m, X, y, cv=5, scoring='roc_auc').mean() study = optuna.create_study(direction='maximize') study.optimize(objective, n_trials=50)

Write a unit test for a model prediction function.

Show answer

import joblib, pandas as pd def test_predict_shape_and_range(): model = joblib.load('model_v1.joblib') df = pd.DataFrame([{'age': 30, 'income': 50000.0, 'city': 'Pune'}, {'age': 45, 'income': 0.0, 'city': 'UnknownCity'}]) p = model.predict_proba(df)[:, 1] assert p.shape == (2,) assert ((p >= 0) & (p <= 1)).all() Run with pytest in CI so broken artifacts never reach production.

Calculate the Population Stability Index for a feature.

Show answer

import numpy as np def psi(expected, actual, bins=10): cuts = np.quantile(expected, np.linspace(0, 1, bins + 1)) cuts[0], cuts[-1] = -np.inf, np.inf e = np.histogram(expected, cuts)[0] / len(expected) + 1e-6 a = np.histogram(actual, cuts)[0] / len(actual) + 1e-6 return float(np.sum((a - e) * np.log(a / e))) Rule of thumb: below 0.1 stable, 0.1 to 0.25 moderate, above 0.25 significant drift.

How do you select the top k features using mutual information in scikit-learn?

Data Preprocessing and Feature EngineeringMedium

Show answer

SelectKBest with mutual_info_classif captures non-linear relationships, unlike plain correlation. from sklearn.feature_selection import SelectKBest, mutual_info_classif sel = SelectKBest(score_func=mutual_info_classif, k=20) X_new = sel.fit_transform(X_train, y_train) print(X_train.columns[sel.get_support()]) Put the selector inside a pipeline so selection is redone in each CV fold.

Your real-time model API takes 400 ms but the requirement is under 50 ms. What do you do?

Show answer

Profile first to see whether time goes to feature lookup, preprocessing or the model. Then cache or precompute features, load the model once at startup, reduce trees or features, use a faster library like LightGBM or ONNX Runtime, and batch requests if possible. Run several workers and keep the service close to its data store. Re-measure p95 and p99 latency, not just the average.

What are learning curves and validation curves?