Linear Regression

A linear regression predicts an outcome, y, as a function of observable predictors, X, but the function need not be linear in the explanatory variables, because elements of X can include squares or logs of variables, or any other transformation. The key factor is that the regression model is linear in b, the set of parameters to be estimated.

If we write y = Xb+e with e a random error that has mean zero, then the conditional mean of y is Xb. If we assume that e is independently and identically distributed and uncorrelated with X (e is exogenous), then ordinary least squares is the best (least-variance) linear unbiased estimator of b, by virtue of the Gauss-Markov theorem.

When e is not independently and identically distributed, we need a weighted-least squares estimator or a robust inference method. When e is correlated with X (e is endogenous), then ordinary least squares suffers from bias and we need to use a quasi-experimental method to estimate b.