Ensemble Learning: Bagging, Boosting, Voting and Stacking Compared

You have already used two ensembles in this part without calling them that: the random forest and the boosted trees. An ensemble is simply a group of models that, together, is more accurate and more stable than any member alone. This session steps back to compare the families side by side, adds the third one, stacking, and then does something textbooks often skip: it checks whether ensembles actually beat the best single model on a real dataset.

You will also see the one condition that decides whether combining models helps at all. It builds on the forest (Session 12), the booster (Session 13) and the SVM (Session 14).

Why a crowd beats an individual (and when it does not)

Imagine three friends who each answer a yes/no quiz question correctly 70 percent of the time, independently. The chance that at least two of the three are right is higher than 70 percent. Add more independent voters and the majority becomes nearly always right. This is an old result, known since the 1780s as Condorcet's jury theorem.

import numpy as np
from math import comb

def majority_accuracy(n_models, p):
    """Chance that more than half of n independent models are right, each right with probability p."""
    return sum(comb(n_models, k) * p**k * (1 - p)**(n_models - k) for k in range(n_models // 2 + 1, n_models + 1))

print("each model is right 70% of the time, and their mistakes are independent:")
for n in (1, 3, 11, 51, 201):
    print(f"  {n:3d} models voting -> {majority_accuracy(n, 0.7):.3f}")

# Now the realistic case: the models make the SAME mistakes much of the time.
rng = np.random.default_rng(0)
n_test, n_models = 20000, 11
shared = rng.random(n_test) < 0.7                    # a "hard question" most models miss together
print("\nsame 11 models, but errors are partly shared:")
for shared_share in (0.0, 0.5, 0.9):
    correct = np.zeros((n_models, n_test), bool)
    for m in range(n_models):
        own = rng.random(n_test) < 0.7
        use_shared = rng.random(n_test) < shared_share
        correct[m] = np.where(use_shared, shared, own)
    vote = correct.sum(axis=0) > n_models // 2
    print(f"  {shared_share:.0%} of mistakes shared -> single model {correct[0].mean():.3f}, majority vote {vote.mean():.3f}")

each model is right 70% of the time, and their mistakes are independent:
    1 models voting -> 0.700
    3 models voting -> 0.784
   11 models voting -> 0.922
   51 models voting -> 0.999
  201 models voting -> 1.000

same 11 models, but errors are partly shared:
  0% of mistakes shared -> single model 0.701, majority vote 0.922
  50% of mistakes shared -> single model 0.694, majority vote 0.739
  90% of mistakes shared -> single model 0.698, majority vote 0.698

Look at the first block. Each model is right 70 percent of the time. Three voting give 0.784, eleven give 0.922 and fifty-one give 0.999. The catch is in the second block. When the models share a large fraction of their mistakes, the benefit disappears. At 50 percent shared mistakes, the majority vote is only 0.739, and at 90 percent shared it is 0.698, exactly the same as a single model.

That is the single most important fact about ensembles: they only help when the members make different mistakes. It is why a forest randomises its trees and why stacking mixes different kinds of models. Eleven copies of the same model add nothing.

The three families

Bagging (Session 12) trains many models of the same type independently on random samples of the data, then votes or averages. It reduces variance. Example: random forest.

Boosting (Session 13) trains weak models one after another, each correcting the last. It reduces bias, but it can overfit without regularisation. Examples: AdaBoost, gradient boosting, XGBoost, LightGBM.

Stacking (stacked generalisation) combines the predictions of different model types (say a forest, an SVM and a booster) using another model, the meta-learner, which learns how best to combine them.

Voting: the simplest blend

A voting ensemble just combines the predictions of several models you already have.

• Hard voting: each model casts one vote for a class and the majority wins.
• Soft voting: each model gives class probabilities, and you average them. It lets a confident model outweigh several unsure ones, and it usually does a little better when the models can produce good probabilities.

Stacking: a model that learns to combine models

Stacking trains a second-level model on the first level's predictions. There is a trap, and it is the same leakage trap as ever. If you train the base models on all the training data and then train the meta-learner on their predictions for that same data, the base models have seen those answers and look better than they are, so the meta-learner learns to over-trust them. The cure is to feed the meta-learner out-of-fold predictions, made by models that did not train on that row. That is what scikit-learn's cv=5 inside StackingClassifier does for you.

Does it actually help? An honest benchmark

Time to test the theory on the breast-cancer data with three genuinely different models: a random forest, a scaled SVM and an XGBoost model, then voting and stacking on top. The scores are averaged over 10 train/test splits (5 folds, twice).

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier, StackingClassifier, VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RepeatedStratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from xgboost import XGBClassifier

X, y = load_breast_cancer(return_X_y=True)
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=2, random_state=0)

base = [("rf",  RandomForestClassifier(n_estimators=200, random_state=0, n_jobs=-1)),
        ("svm", make_pipeline(StandardScaler(), SVC(probability=True, random_state=0))),
        ("xgb", XGBClassifier(n_estimators=200, learning_rate=0.05, max_depth=3, eval_metric="logloss", random_state=0))]

models = dict(base)
models["hard vote"]  = VotingClassifier(base, voting="hard")
models["soft vote"]  = VotingClassifier(base, voting="soft")
models["stacking"]   = StackingClassifier(base, final_estimator=LogisticRegression(), cv=5)

rows = []
for name, m in models.items():
    s = cross_val_score(m, X, y, cv=cv)
    rows.append((name, s.mean(), s.std()))
print(pd.DataFrame(rows, columns=["model", "accuracy", "std"]).round(3).to_string(index=False))

    model  accuracy   std
       rf     0.965 0.014
      svm     0.974 0.013
      xgb     0.972 0.012
hard vote     0.974 0.012
soft vote     0.975 0.011
 stacking     0.972 0.011

Look at the table without wishful thinking. The best single model, the SVM, scores 0.974. Hard voting also scores 0.974, soft voting 0.975 and stacking 0.972. The ensembles are basically level with the best member, not better in any way that matters, since the spread across splits is about 0.012. The random forest (0.965) is the weakest of the group, and the ensembles do beat it, but only because it was the weakest.

Why? The breast-cancer data is easy and the models are already strong, so there is little room left to improve, and the three models make many of the same mistakes on the same hard cases, the condition from the first section. This is not a failure of ensembling, it is a lesson about when to use it.

from sklearn.ensemble import VotingClassifier, StackingClassifier, RandomForestClassifier
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from xgboost import XGBClassifier

base = [("rf", RandomForestClassifier(n_estimators=200)),
        ("svm", SVC(probability=True)),
        ("xgb", XGBClassifier(n_estimators=200, eval_metric="logloss"))]

vote = VotingClassifier(base, voting="soft").fit(X_train, y_train)
stack = StackingClassifier(base, final_estimator=LogisticRegression(), cv=5).fit(X_train, y_train)
print(vote.score(X_test, y_test), stack.score(X_test, y_test))

The wall of the classical models

Take a step back to see where Part III has left us. Linear models, trees, forests, boosting, SVMs and their ensembles are superb on tabular data, where each column is a meaningful feature that a person has already prepared. But all of them depend on hand-crafted features. Give them raw pixels, raw audio or raw sentences and they struggle: they have no idea that neighbouring pixels belong together or that word order changes meaning.

That is the wall that Part IV takes down, by using networks that learn their own features from raw data and whose structure matches the data: convolutions for images, recurrence for sequences.

Part III complete: what you can do now

You can build a readable tree and explain why it overfits, turn it into a forest or a boosted model, draw a maximum-margin boundary and bend it with a kernel, and combine models while knowing the one condition under which combining works. Most importantly, you can compare them honestly. On many real business problems, a boosted tree model with sensible tuning is still the model to beat.

Try it yourself

1. In why_ensembles_work.py, change the single-model accuracy from 0.7 to 0.55. How many voters do you need to reach 0.9?
2. Add a plain LogisticRegression (with scaling) as a fourth base model in voting_stacking.py. Does the stack improve?
3. Make a stack from three random forests with different seeds. Does it help? Why not?
4. Try your own benchmark on the diabetes data with VotingRegressor and StackingRegressor.

What comes next

Session 16 opens Part IV, deep learning for images and sequences. We start with the convolutional neural network, the model that finally lets a machine learn to see.

Frequently asked questions