Stacking

Stacking combines predictions from several different models by feeding those predictions into a final "meta-model," which learns how to best weigh and combine them.

How Stacking Works

1. Train several different base models (e.g. Logistic Regression, SVM, KNN)
2. Use their predictions on held-out data as new input features
3. Train a meta-model on those predictions to produce the final output

Implementing in Python

from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier

stack = StackingClassifier(
    estimators=[
        ("svc", SVC(probability=True)),
        ("knn", KNeighborsClassifier())
    ],
    final_estimator=LogisticRegression()
)

stack.fit(X_train, y_train)
print(stack.predict(X_test))

Stacking vs Bagging and Boosting

While bagging and boosting typically combine many versions of the same algorithm, stacking is designed to combine genuinely different algorithms, letting the meta-model learn which base model to trust in which situations.

Coming Up Next

Next, you'll look at Voting and Averaging, the simplest ways to combine ensemble predictions.