Homogeneous and Heterogeneous Ensembles

Ensembles can be built in two broad ways, depending on whether the combined models are of the same type or different types - known as Homogeneous and Heterogeneous Ensembles.

Homogeneous Ensembles

# Same algorithm, trained multiple times
# (e.g. many Decision Trees)

Examples: Random Forest (Bagging), Gradient Boosting, AdaBoost

These ensembles typically vary the training data (via bagging) or focus on correcting previous errors (via boosting), while keeping the underlying algorithm the same across all models.

Heterogeneous Ensembles

# Different algorithms, trained on the same data

Examples: Stacking a Logistic Regression, a Decision Tree,
          and a KNN model together

These ensembles combine models with different strengths and weaknesses, hoping that their errors don't overlap - so what one model gets wrong, another might get right.

Implementing a Heterogeneous Ensemble in Python

from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC

stack = StackingClassifier(estimators=[
    ("dt", DecisionTreeClassifier()),
    ("svc", SVC(probability=True))
], final_estimator=LogisticRegression())

stack.fit(X_train, y_train)
print(stack.predict(X_test))

Coming Up Next

Next, you'll learn about Bagging, one of the foundational ensemble techniques.