Decision Trees and Ensembles Interview Questions for Machine Learning

Show answer

At each node the tree tries candidate features and thresholds and picks the split that most reduces impurity in the child nodes. For classification, impurity is measured by Gini or entropy; for regression, by variance or MSE. The process repeats recursively until a stopping rule such as max_depth, min_samples_leaf or pure leaves is reached. This greedy approach is fast but not globally optimal.

What is the difference between Gini impurity and entropy?

Show answer

Gini impurity is 1 - sum of p squared and measures the chance of misclassifying a random sample. Entropy is - sum of p log p and comes from information theory; the reduction is called information gain. Both usually produce very similar trees. Gini is slightly faster because it avoids logarithms, which is why it is the scikit-learn default.

Why do single decision trees overfit and how do you control it?

Show answer

An unrestricted tree keeps splitting until leaves are pure, memorising noise. Control it with pre-pruning hyperparameters like max_depth, min_samples_split, min_samples_leaf and max_leaf_nodes, or with post-pruning through cost-complexity pruning using ccp_alpha. In practice ensembles such as random forests or boosting are used to reduce the variance of single trees.

Explain bagging and how a random forest uses it.

Show answer

Bagging trains many models on bootstrap samples, random samples with replacement, and averages their predictions to reduce variance. A random forest adds feature randomness: each split considers only a random subset of features, controlled by max_features. This de-correlates the trees, so averaging helps more. Out-of-bag samples, about 37 percent left out per tree, give a free validation estimate.

Train a random forest and show feature importances in scikit-learn.

Show answer

from sklearn.ensemble import RandomForestClassifier import pandas as pd rf = RandomForestClassifier(n_estimators=300, min_samples_leaf=2, n_jobs=-1, random_state=42, oob_score=True) rf.fit(X_train, y_train) print('OOB', rf.oob_score_) imp = pd.Series(rf.feature_importances_, index=X_train.columns) print(imp.sort_values(ascending=False).head(10))

Visualise a small decision tree and its rules.

Show answer

from sklearn.tree import DecisionTreeClassifier, export_text, plot_tree import matplotlib.pyplot as plt dt = DecisionTreeClassifier(max_depth=3, random_state=0).fit(X_train, y_train) print(export_text(dt, feature_names=list(X_train.columns))) plot_tree(dt, feature_names=X_train.columns, filled=True) plt.show() Shallow trees are a good way to explain rules to business teams.

What is boosting and how is it different from bagging?

Show answer

Boosting trains models sequentially, each one focusing on the errors of the combined previous models, which mainly reduces bias. Bagging trains models independently in parallel and averages them, which mainly reduces variance. Boosting uses shallow trees and a learning rate, and can overfit if run too long, so early stopping is important. Examples are AdaBoost, Gradient Boosting, XGBoost, LightGBM and CatBoost.

How does gradient boosting work?

Show answer

Gradient boosting starts with a simple prediction, like the mean, then repeatedly fits a small tree to the negative gradient of the loss, which for MSE is just the residuals. Each tree's output is scaled by a learning rate and added to the ensemble. A lower learning rate with more trees usually generalises better. Any differentiable loss can be used, which makes it flexible.

What makes XGBoost, LightGBM and CatBoost different?

Show answer

XGBoost adds regularisation to the objective, uses second-order gradients and handles missing values natively. LightGBM grows trees leaf-wise and uses histogram binning, making it very fast on large data, though it can overfit small data. CatBoost handles categorical features with ordered target statistics and uses symmetric trees, giving strong defaults with less tuning. All three support GPU training.

Your random forest model file is 2 GB and too slow for real-time scoring. What can you do?

Show answer

Reduce size by limiting max_depth and min_samples_leaf, using fewer trees since gains flatten after a few hundred, and dropping weak features. Alternatively switch to a gradient boosting model like LightGBM, which usually gives similar accuracy with far smaller, faster models. Converting to ONNX or using a compiled predictor can also speed up inference. Measure accuracy and latency after each change.

XGBoost validation loss starts rising after 300 rounds while training loss keeps falling. What do you do?

Show answer

That is overfitting beyond about 300 rounds. I would use early stopping with a patience such as 50 rounds on a validation set so training stops at the best iteration, then lower the learning rate, reduce max_depth, add subsample and colsample_bytree below 1, and increase min_child_weight or lambda. The final model is refit using the best iteration count found.

Random forest and LightGBM give the same AUC. Which would you deploy?

Show answer

I would compare beyond AUC: model size, inference latency, training time, calibration and stability across time periods. LightGBM is usually smaller and faster to score, while random forest needs less tuning and is less prone to overfitting. If both are equal, I would choose the one that is cheaper to run and easier for the team to maintain and retrain.

Train LightGBM with early stopping.

Show answer

import lightgbm as lgb m = lgb.LGBMClassifier(n_estimators=5000, learning_rate=0.05, num_leaves=31, subsample=0.8, subsample_freq=1, colsample_bytree=0.8) m.fit(X_train, y_train, eval_set=[(X_val, y_val)], eval_metric='auc', callbacks=[lgb.early_stopping(100)]) print(m.best_iteration_) The large n_estimators is only an upper limit; early stopping picks the real number.

Compute permutation importance on validation data.

Show answer

from sklearn.inspection import permutation_importance import pandas as pd r = permutation_importance(model, X_val, y_val, n_repeats=10, scoring='roc_auc', random_state=42, n_jobs=-1) imp = pd.Series(r.importances_mean, index=X_val.columns) print(imp.sort_values(ascending=False)) Near-zero or negative values mean the feature adds little on unseen data.

Describe a project where you improved a model's performance significantly.

Show answer

Situation: a product-return prediction model had 0.68 AUC with logistic regression. Task: improve it for a warehouse pilot. Action: I engineered features like past return rate per customer and category, moved to LightGBM with early stopping, and tuned it with Optuna on time-based folds. Result: AUC rose to 0.81, and the pilot reduced avoidable returns in two categories. I documented each step's gain.

How do you decide when a model is good enough to stop tuning?

Show answer

I agree on a target metric and business threshold before starting, such as recall of 70 percent at 90 percent precision. I stop tuning when gains fall below the noise between CV folds or when further work costs more than the expected value. In one project I stopped at a 0.3 percent gain and spent the time on monitoring instead, which proved more useful.

When would you use AdaBoost and how does it re-weight samples?

Show answer

AdaBoost trains weak learners, usually decision stumps, in sequence. After each round it increases the weights of misclassified samples so the next learner focuses on them, and gives each learner a vote weight based on its accuracy. It works well on clean data but is sensitive to noisy labels and outliers, which get ever larger weights. Gradient boosting is a generalisation that has largely replaced it.

Use HistGradientBoosting with native categorical support in scikit-learn.

Show answer

from sklearn.ensemble import HistGradientBoostingClassifier clf = HistGradientBoostingClassifier( categorical_features='from_dtype', max_iter=500, learning_rate=0.05, early_stopping=True, random_state=42) X_train[cat_cols] = X_train[cat_cols].astype('category') clf.fit(X_train, y_train) It handles missing values natively and is much faster than GradientBoostingClassifier on large data.

Which hyperparameters matter most when tuning a gradient boosting model?

Show answer

The key ones are learning_rate and number of trees, tuned together with early stopping; tree complexity through max_depth or num_leaves and min_child_samples; and randomness through subsample and colsample_bytree. Regularisation terms like lambda and alpha help on noisy data. A common approach is to fix a small learning rate such as 0.05, use early stopping for trees, then tune complexity.

Why can impurity-based feature importance be misleading?

Show answer

Impurity-based importance is computed on training data and favours high-cardinality or continuous features, which offer more split points, even if they are noise. It also splits credit between correlated features. Permutation importance on validation data, which measures the score drop when a feature is shuffled, is more reliable, and SHAP values give consistent per-prediction attributions.

What is stacking and when is it worth it?

Show answer

Stacking trains several diverse base models and then a meta-model on their out-of-fold predictions to combine them. It often gives a small accuracy gain, common in competitions. In production it adds complexity, latency and maintenance, so it is worth it only when the gain has clear business value. scikit-learn provides StackingClassifier and StackingRegressor.

Business users do not trust a gradient boosting model because it is a black box. How do you help?

Show answer

I would use SHAP to show global drivers with a summary plot and explain individual decisions with waterfall plots, such as why one applicant got a high risk score. I would also check that the top drivers make business sense, add monotonic constraints where logic requires it, and share a few real case examples. If trust is still an issue, a scorecard model can serve as a transparent benchmark.

Your tree model predicts higher risk for customers with higher income, which is illogical. What do you do?

Show answer

First check whether data supports it, perhaps due to confounding or a small segment. If the relationship should be one-directional for business or regulatory reasons, I would apply monotonic constraints, available in XGBoost, LightGBM and HistGradientBoosting, so risk can only decrease with income. Then compare validation metrics; usually the loss is small and the model becomes more trustworthy.

Explain a single prediction with SHAP for a tree model.

Show answer

import shap explainer = shap.TreeExplainer(model) sv = explainer(X_val) shap.plots.waterfall(sv[0]) shap.plots.beeswarm(sv) The waterfall shows how each feature pushes one prediction up or down from the base value; the beeswarm shows global importance and direction. TreeExplainer is fast and exact for tree ensembles.

Your gradient boosting model performs worse on recent months than older ones. What do you check?