Linear Regression for Experimental Research
Simple linear model
The population model is
\[Y_i=\beta_0+\beta_1X_i+\varepsilon_i.\]Ordinary least squares chooses the coefficients that minimize the sum of squared residuals:
\[\min_{\beta_0,\beta_1} \sum_i(Y_i-\beta_0-\beta_1X_i)^2.\]The estimated slope and intercept are
\[\hat\beta_1= \frac{\sum_i(X_i-\bar X)(Y_i-\bar Y)} {\sum_i(X_i-\bar X)^2},\] \[\hat\beta_0=\bar Y-\hat\beta_1\bar X.\]The fitted value and residual are
\[\hat Y_i=\hat\beta_0+\hat\beta_1X_i, \qquad e_i=Y_i-\hat Y_i.\]With an intercept and one slope, the residual variance estimate is
\[\hat\sigma^2=\frac{\sum_i e_i^2}{n-2}.\]Coefficient inference
To test $H_0:\beta_j=\beta_{j,0}$,
\[t=\frac{\hat\beta_j-\beta_{j,0}} {SE(\hat\beta_j)}.\]Under the classical linear model, use the residual degrees of freedom. Robust or design-based standard errors may be appropriate when the classical variance assumptions do not hold.
Coefficient of determination
With
\[SSE=\sum_i(Y_i-\hat Y_i)^2, \qquad SST=\sum_i(Y_i-\bar Y)^2,\]the coefficient of determination is
\[R^2=1-\frac{SSE}{SST}.\]$R^2$ measures in-sample fit. A high $R^2$ does not establish causality, and a randomized treatment effect can be credible even when $R^2$ is low.
Multiple regression in matrix form
The classical linear regression model is
\[Y=X\beta+\varepsilon.\]When $X^\top X$ is invertible, the OLS estimator is
\[\boxed{\hat\beta=(X^\top X)^{-1}X^\top Y}.\]Under homoskedastic uncorrelated errors with variance $\sigma^2$,
\[\operatorname{Var}(\hat\beta\mid X) = \sigma^2(X^\top X)^{-1}.\]For the common null $H_0:\beta_j=0$,
\[t_j=\frac{\hat\beta_j}{SE(\hat\beta_j)}.\]The matrix variance formula is model-based. Randomized experiments can also justify inference from the assignment mechanism; see Regression Adjustment in Experiments.