Hard Machine Learning Interview Questions

Show answer

The L1 constraint region is a diamond with corners on the axes, so the optimal point often lands on a corner where some coefficients are exactly zero. The L2 region is a circle with no corners, so coefficients shrink smoothly but stay non-zero. In gradient terms, L1 applies a constant push toward zero regardless of size, while L2's push becomes tiny as weights get small.

When would you use Huber or quantile regression instead of OLS?

Show answer

Huber regression uses squared loss for small errors and absolute loss for large ones, so it resists outliers while staying smooth. Quantile regression predicts a chosen percentile, such as the 90th percentile of delivery time, which is useful for setting safe promises or prediction intervals. Gradient boosting libraries support both losses for non-linear versions.

Lasso dropped a feature the business considers very important. How do you handle it?

Show answer

Lasso often keeps only one of a group of correlated features, so the dropped feature may be redundant rather than useless. I would check correlations, try Elastic Net which keeps correlated groups, and compare cross-validated performance with the feature forced in. I would explain the trade-off clearly; if the accuracy difference is small, keeping the feature for trust and interpretability is reasonable.

You need prediction intervals, not just point estimates, for delivery time. How?

Show answer

Options are quantile regression, training separate models for the 10th and 90th percentiles with GradientBoostingRegressor(loss='quantile') or LightGBM; conformal prediction, which wraps any model and gives intervals with guaranteed coverage; or bootstrapping. I would check calibration: about 80 percent of actual values should fall within an 80 percent interval on a holdout set.

How do you create lag and rolling features for a sales forecasting model in pandas?

Show answer

Sort by item and date, then shift so features use only past values. df = df.sort_values(['item', 'date']) g = df.groupby('item')['sales'] df['lag_7'] = g.shift(7) df['roll_mean_28'] = g.transform(lambda s: s.shift(1).rolling(28).mean()) The shift before rolling prevents the current day's sales from leaking into its own feature.

What is the kernel trick?

Show answer

The kernel trick computes dot products of data in a high-dimensional feature space without explicitly transforming the data, using a kernel function such as RBF, exp(-gamma times squared distance). This lets linear algorithms like SVM learn non-linear boundaries efficiently. In RBF SVMs, larger gamma makes each point's influence more local, which can lead to overfitting.

What is probability calibration and why does it matter?

Show answer

A calibrated model's predicted probabilities match real frequencies: of cases scored 0.8, about 80 percent should be positive. It matters when scores drive decisions like credit limits or expected loss. Random forests and SVMs are often poorly calibrated. Fix it with CalibratedClassifierCV using Platt scaling (sigmoid) or isotonic regression, and check with a reliability curve or Brier score.

Your classifier works well overall but poorly for one customer segment. What do you do?

Show answer

I would measure metrics per segment to confirm, then check if that segment is under-represented or has different feature distributions. Fixes include collecting more data, sample weighting, segment-specific features, or a separate model if the behaviour is truly different. If the segment relates to a protected attribute, I would also review fairness metrics and involve the compliance team.

A credit model's probability of 0.3 is used to set interest rates, but defaults in that band are 12 percent. What is wrong?

Show answer

The model is poorly calibrated: it ranks customers but its probabilities are not reliable. Since pricing depends on actual probabilities, I would calibrate it with isotonic regression or Platt scaling on a held-out set and validate with a reliability curve and Brier score. I would also check whether the default rate has drifted since training, which would require retraining.

How do you apply SMOTE correctly without leakage?

Show answer

Use an imbalanced-learn pipeline so oversampling happens only on training folds. from imblearn.pipeline import Pipeline from imblearn.over_sampling import SMOTE from sklearn.model_selection import cross_val_score pipe = Pipeline([('pre', pre), ('smote', SMOTE(random_state=42)), ('clf', LogisticRegression(max_iter=1000))]) print(cross_val_score(pipe, X, y, cv=5, scoring='average_precision').mean()) Applying SMOTE before splitting puts synthetic copies of test points into training.

Which hyperparameters matter most when tuning a gradient boosting model?

Show answer

The key ones are learning_rate and number of trees, tuned together with early stopping; tree complexity through max_depth or num_leaves and min_child_samples; and randomness through subsample and colsample_bytree. Regularisation terms like lambda and alpha help on noisy data. A common approach is to fix a small learning rate such as 0.05, use early stopping for trees, then tune complexity.

Why can impurity-based feature importance be misleading?

Show answer

Impurity-based importance is computed on training data and favours high-cardinality or continuous features, which offer more split points, even if they are noise. It also splits credit between correlated features. Permutation importance on validation data, which measures the score drop when a feature is shuffled, is more reliable, and SHAP values give consistent per-prediction attributions.

What is stacking and when is it worth it?

Show answer

Stacking trains several diverse base models and then a meta-model on their out-of-fold predictions to combine them. It often gives a small accuracy gain, common in competitions. In production it adds complexity, latency and maintenance, so it is worth it only when the gain has clear business value. scikit-learn provides StackingClassifier and StackingRegressor.

Business users do not trust a gradient boosting model because it is a black box. How do you help?

Show answer

I would use SHAP to show global drivers with a summary plot and explain individual decisions with waterfall plots, such as why one applicant got a high risk score. I would also check that the top drivers make business sense, add monotonic constraints where logic requires it, and share a few real case examples. If trust is still an issue, a scorecard model can serve as a transparent benchmark.

Your tree model predicts higher risk for customers with higher income, which is illogical. What do you do?

Show answer

First check whether data supports it, perhaps due to confounding or a small segment. If the relationship should be one-directional for business or regulatory reasons, I would apply monotonic constraints, available in XGBoost, LightGBM and HistGradientBoosting, so risk can only decrease with income. Then compare validation metrics; usually the loss is small and the model becomes more trustworthy.

Explain a single prediction with SHAP for a tree model.

Show answer

import shap explainer = shap.TreeExplainer(model) sv = explainer(X_val) shap.plots.waterfall(sv[0]) shap.plots.beeswarm(sv) The waterfall shows how each feature pushes one prediction up or down from the base value; the beeswarm shows global importance and direction. TreeExplainer is fast and exact for tree ensembles.

What is the difference between PCA and t-SNE or UMAP?

Show answer

PCA is a linear method that preserves global variance, is fast, deterministic and can transform new data, so it is used for preprocessing. t-SNE and UMAP are non-linear methods that preserve local neighbourhoods, producing good 2D visualisations of clusters. t-SNE distances between clusters are not meaningful. UMAP is faster, keeps more global structure and can transform new points.

What is a Gaussian Mixture Model and how is it different from k-means?

Show answer

A GMM assumes data comes from a mix of Gaussian distributions and fits their means, covariances and weights with the Expectation-Maximisation algorithm. Unlike k-means' hard assignments, GMM gives soft probabilities of belonging to each cluster and can model elliptical clusters of different sizes. You can choose the number of components using BIC or AIC.

How do you evaluate clustering when there are no labels?

Show answer

Use internal metrics: silhouette score, Davies-Bouldin index (lower is better) and Calinski-Harabasz index (higher is better). Check stability by re-clustering on bootstrap samples. Most important is external usefulness: profile each cluster with key statistics and check whether business teams find the segments distinct and actionable, for example different response to a campaign.

You need to detect unusual transactions but have no fraud labels. What do you do?

Show answer

I would start with unsupervised anomaly detection: Isolation Forest or Local Outlier Factor on features like amount relative to the customer's usual spend, time of day, new device and merchant risk. I would set the threshold based on what the review team can handle, collect investigator feedback as labels, and move to a supervised model once enough labelled cases exist.

The segments you built six months ago no longer match customer behaviour. What do you do?

Show answer

I would measure drift by comparing feature distributions and segment sizes over time. Then refit clustering on recent data and map new clusters to old ones by centroid similarity so business names stay consistent. I would set up a quarterly refresh with stability checks and alert when segment sizes shift sharply, and explain any renamed or merged segments to stakeholders.

Run DBSCAN and count clusters and noise points.

Data Preprocessing and Feature EngineeringHard

Show answer

from sklearn.cluster import DBSCAN import numpy as np db = DBSCAN(eps=0.5, min_samples=10).fit(Xs) labels = db.labels_ n_clusters = len(set(labels)) - (1 if -1 in labels else 0) print(n_clusters, 'clusters,', np.sum(labels == -1), 'noise') Pick eps from the knee of a sorted k-nearest-neighbour distance plot.

Your preprocessing works in notebooks but gives different results in production. Why?

Show answer

Common causes are preprocessing written separately in two places, statistics like means recomputed on production data instead of using training values, different library versions, or different column order and data types. The fix is to save one fitted Pipeline with preprocessing and model together, pin versions, validate input schema, and test that the saved pipeline gives identical outputs on sample rows.

What is nested cross-validation and when do you need it?

Show answer

Nested CV has an inner loop that tunes hyperparameters and an outer loop that evaluates the whole tuning process on unseen folds. It gives an unbiased performance estimate when you tune heavily on small data, because a normal CV score used both to tune and to report is optimistic. It is expensive, so it is mainly used for small datasets or research reporting.

How do you compare two models fairly?

Show answer

Use the same data splits and folds, the same preprocessing inside pipelines and the same metric tied to the business goal. Look at the mean and standard deviation across folds, not one number, and use a paired test like a corrected t-test or bootstrap on the fold differences. Also compare cost factors: latency, memory, interpretability and maintenance.

What fairness checks would you run on a model that affects people?

Show answer

Compare metrics across groups such as gender, age band or region: approval rate (demographic parity), true positive rate (equal opportunity) and false positive rate. Check calibration per group too. Remove or audit proxy features like pin code that may stand in for protected attributes. Document findings in a model card, and involve legal or compliance teams, especially for lending and hiring use cases.

Your cross-validation score is 0.85 but the test score is 0.70. What could explain it?

Show answer

Possible causes: leakage inside CV, such as preprocessing or feature selection fit outside folds; repeated tuning that overfit the CV folds; a test set from a different time period or population; or duplicate entities across folds but not test. I would check pipeline structure, switch to time-based or group splits that match how test data differs, and compare feature distributions between the two sets.

An A/B test of a new recommendation model shows no significant uplift despite better offline metrics. What next?

Show answer

I would check the test first: enough sample size and duration, correct randomisation, and no bugs in serving such as stale features. Then investigate whether offline metrics really reflect user value, for example optimising clicks rather than purchases, or popularity bias. Segment-level analysis may show gains in some groups. I would revise the offline metric to better match the online goal.

Tune LightGBM with Optuna.

Show answer

import optuna from sklearn.model_selection import cross_val_score def objective(trial): p = {'num_leaves': trial.suggest_int('num_leaves', 15, 255), 'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.2, log=True), 'min_child_samples': trial.suggest_int('min_child_samples', 10, 200)} m = lgb.LGBMClassifier(n_estimators=500, **p) return cross_val_score(m, X, y, cv=5, scoring='roc_auc').mean() study = optuna.create_study(direction='maximize') study.optimize(objective, n_trials=50)

Get a bootstrap confidence interval for a test metric.

Show answer

import numpy as np from sklearn.metrics import roc_auc_score rng = np.random.default_rng(0); scores = [] yt, pp = np.array(y_test), np.array(proba) for _ in range(1000): i = rng.integers(0, len(yt), len(yt)) if len(np.unique(yt[i])) == 2: scores.append(roc_auc_score(yt[i], pp[i])) print(np.percentile(scores, [2.5, 97.5]))

How do you handle pressure to report better model numbers than the data supports?

Show answer

I explain the risk clearly: inflated numbers lead to bad decisions and lose trust when production results arrive. I present the honest metric with confidence intervals, and then offer a concrete plan to improve it, such as new data sources. In one case I showed that a 'better' number came from leakage, and leadership agreed to report the lower but reliable figure.

Explain the bias in AUC when classes are extremely imbalanced, with a numeric intuition.

Show answer

With 100 fraud and 100,000 genuine cases, a false positive rate of 1 percent looks tiny on the ROC curve, but it means 1,000 false alerts versus at most 100 true ones, so precision is below 10 percent. ROC-AUC can be 0.97 while the model is impractical. Precision-recall curves expose this because precision directly depends on the count of false positives.

What is training-serving skew and how do you prevent it?

Show answer

Training-serving skew is when features at prediction time are computed differently from features used in training, causing silent accuracy loss. Causes include separate code paths, different data sources, or time windows that use future data in training. Prevent it by sharing one feature-computation code path or a feature store, logging served features and comparing their distributions with training data.

Your real-time model API takes 400 ms but the requirement is under 50 ms. What do you do?

Show answer

Profile first to see whether time goes to feature lookup, preprocessing or the model. Then cache or precompute features, load the model once at startup, reduce trees or features, use a faster library like LightGBM or ONNX Runtime, and batch requests if possible. Run several workers and keep the service close to its data store. Re-measure p95 and p99 latency, not just the average.

Labels for your loan model arrive only after 6 months. How do you monitor it meanwhile?

Show answer

I would rely on proxy signals: input data drift with PSI, prediction score distribution, approval rates by segment, and early indicators like first missed payment at 30 or 60 days. Once enough early outcomes arrive, I would check performance on these. I would also set up a delayed evaluation job that computes full metrics as labels mature and compares them with the validation baseline.

Your model must run on a mobile app with no internet. What do you consider?

Show answer

Model size, memory and latency on low-end phones become key. I would choose a compact model, reduce features, and convert it to a mobile format such as ONNX, TensorFlow Lite or Core ML, applying quantisation if needed. I would make sure all preprocessing is replicated on-device, test accuracy after conversion, and plan how model updates will ship with app releases.

Calculate the Population Stability Index for a feature.

Show answer

import numpy as np def psi(expected, actual, bins=10): cuts = np.quantile(expected, np.linspace(0, 1, bins + 1)) cuts[0], cuts[-1] = -np.inf, np.inf e = np.histogram(expected, cuts)[0] / len(expected) + 1e-6 a = np.histogram(actual, cuts)[0] / len(actual) + 1e-6 return float(np.sum((a - e) * np.log(a / e))) Rule of thumb: below 0.1 stable, 0.1 to 0.25 moderate, above 0.25 significant drift.

What is a feature store and when is it needed?

Show answer

A feature store is a system that defines, computes and serves features consistently for both training and real-time inference, with an offline store for historical data and an online store for low-latency lookup. It prevents training-serving skew and lets teams reuse features. It is worth it when many models share features or need real-time features; small teams can start with well-managed tables.

Your gradient boosting model performs worse on recent months than older ones. What do you check?

Show answer

I would check for drift between periods in feature distributions and target rate, and confirm that the evaluation uses a time-based split. If behaviour has changed, I would train on a more recent window or weight recent data more, add features that capture the change, and set up a regular retraining schedule. Comparing SHAP importance across periods shows which relationships have shifted.

What is Local Outlier Factor and how does it differ from Isolation Forest?