Parametric Regression
Definition
Parametric regression is a regression approach where the relationship between input variables and the target variable is assumed to have a fixed mathematical form with a finite number of parameters.
The general idea is:
\[Y = f(X; \theta) + \varepsilon\]where:
- $Y$ is the target variable.
- $X$ is the input variable or feature vector.
- $f(X; \theta)$ is a chosen model form.
- $\theta$ is a finite set of parameters to estimate.
- $\varepsilon$ is the error term.
Main Idea
In parametric regression, we choose the shape of the model before fitting it.
For example, in linear regression we assume:
\[Y \approx \beta_0 + \beta_1 X\]The data is then used to estimate the unknown parameters $\beta_0$ and $\beta_1$.
Why It Is Called Parametric
It is called parametric because the whole model is controlled by a limited number of parameters.
For example:
\[\hat y = \beta_0 + \beta_1 x\]has only two parameters:
- $\beta_0$
- $\beta_1$
Once those are known, the entire prediction rule is known.
Common Examples
- Linear Regression
- Polynomial Regression
- Logistic Regression
- Poisson Regression
- Exponential Regression
Strengths
- Easy to interpret.
- Needs less data than many flexible methods.
- Fast to train.
- Gives explicit formulas.
- Useful when the assumed model form is approximately correct.
Weaknesses
- Can be too rigid.
- Performs badly if the true relationship has a different shape.
- May hide important nonlinear structure.
- Can underfit complex data.
Relation to Conditional Mean
Regression often tries to estimate the conditional mean:
\[E[Y \mid X = x]\]A parametric model assumes this conditional mean has a specific form, such as:
\[E[Y \mid X = x] = \beta_0 + \beta_1x\]Example: Basket Size Prediction
Suppose we want to predict the final basket size $Y$ from the observed partial basket size $X$.
A simple parametric model could be:
\[\hat Y = \beta_0 + \beta_1X\]This assumes that every additional observed item increases the final basket size by a roughly constant amount.
Comparison With Nonparametric Regression
| Approach | Model shape | Flexibility | Example |
|---|---|---|---|
| Parametric | Chosen in advance | Lower | Linear regression |
| Nonparametric | Learned more directly from data | Higher | k-NN regression |
When To Use
Use parametric regression when:
- You want interpretability.
- You have limited data.
- You believe the relationship has a simple form.
- You need a clean mathematical model.
Exercises
- Explain why $\hat y = \beta_0 + \beta_1x$ is a parametric model.
- Give one example where a linear model would underfit the data.
- In basket-size prediction, what does $\beta_1$ mean if $X$ is the current number of observed items?