Ensemble Learning
Definition
Ensemble learning combines multiple models to produce one stronger prediction.
Instead of relying on one model:
\[\hat f_1(x)\]an ensemble combines many models:
\[\hat f_1(x), \hat f_2(x), \ldots, \hat f_B(x)\]Main Idea
Different models make different errors.
If their errors are not perfectly identical, averaging them can reduce variance and improve prediction stability.
For regression, an ensemble often predicts:
\[\hat f(x) = \frac{1}{B}\sum_{b=1}^{B}\hat f_b(x)\]Common Ensemble Methods
- Bagging
- Random Forests
- boosting
Why It Helps
A single tree can be unstable.
Averaging many trees usually produces a smoother and more reliable prediction.
This is the main idea behind random forests.
Retail Example
For basket-size prediction, one tree may overreact to unusual large orders.
An ensemble averages many trees, reducing the effect of any single unstable split.
Strengths
- Often improves predictive accuracy.
- Reduces variance.
- Works well with nonlinear data.
- Useful for complex datasets.
Weaknesses
- Less interpretable than a single model.
- More computationally expensive.
- Can be harder to explain in a report.
Exercises
- Why can averaging models reduce variance?
- Why is a random forest usually more stable than one decision tree?
- Give one disadvantage of ensemble methods.