Splines
Definition
Splines are flexible regression functions made by joining simple polynomial pieces together.
They are used to model nonlinear relationships while keeping the curve smooth.
Main Idea
Instead of fitting one polynomial over the whole input range, splines fit different polynomial pieces in different intervals.
The points where the pieces meet are called knots.
Knots
A knot is a value of $x$ where the spline is allowed to change shape.
If the knots are:
\[\xi_1, \xi_2, \ldots, \xi_K\]then the spline can behave differently between these knot points.
Why Not Just Use One Big Polynomial
A high-degree polynomial can behave badly near the edges and can oscillate too much.
Splines solve this by using low-degree polynomial pieces instead of one large global polynomial.
Cubic Splines
A common choice is a cubic spline.
Each piece is a cubic polynomial, but the pieces are joined so that the overall curve is smooth.
The curve is usually continuous, with continuous first and second derivatives.
Regression Use
In regression, splines estimate a flexible conditional mean function:
\[\hat m(x) \approx E[Y \mid X=x]\]They are more flexible than linear regression but often smoother than tree-based methods.
Retail Example
Suppose:
- $X$ = current basket size
- $Y$ = final basket size
A spline can model a relationship where final basket size increases quickly at first, then slows down, then behaves differently for unusually large baskets.
Strengths
- Flexible but still smooth.
- More interpretable than many black-box models.
- Good for one-dimensional or low-dimensional nonlinear regression.
- Avoids some problems of high-degree polynomials.
Weaknesses
- Requires choosing knots or smoothness level.
- Less natural for many categorical variables.
- Can overfit if too flexible.
- Does not automatically capture complex interactions unless extended.
Relation to Local Smoothing
Both splines and local smoothing estimate flexible curves.
Splines build a global smooth function from polynomial pieces.
Local smoothing estimates the curve using nearby observations around each point.
Exercises
- What is a knot in a spline?
- Why can splines behave better than high-degree polynomial regression?
- In basket-size prediction, why might a spline be useful for current basket size?