Introduction to Multiple Linear Regression

\(y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \ldots + \beta_k x_{ik} + \epsilon_i\)
Assumptions:
- \(\epsilon_i \sim N(0, \sigma^2)\), \(Cov(\epsilon_i, \epsilon_j) = 0\)
- Strict: \(\epsilon_i \overset{i.i.d.}{\sim} N(0, \sigma^2)\)

Matrix form

In matrix notation we write:
- \[\begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix} = \begin{bmatrix} 1 & x_{11} & x_{12} & \ldots & x_{1k} \\ 1 & x_{21} & x_{22} & \ldots & x_{2k} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n1} & x_{n2} & \ldots & x_{nk} \end{bmatrix} \cdot \begin{bmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_k \end{bmatrix} + \begin{bmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_n \end{bmatrix}\]
Or in short (with dimension): \(\mathbf{y}_{n \times 1} = \mathbf{X}_{n \times (k+1)}\mathbf{\beta}_{(k+1) \times 1} + \mathbf{\epsilon}_{n \times 1}\)
alternatively, just: \(\mathbf{y} = \mathbf{X}\mathbf{\beta} + \mathbf{\epsilon}\)

Some properties

\(E(\mathbf{\epsilon}) = \mathbf{0}\)
\(Var(\mathbf{\epsilon}) = \sigma^2\mathbf{I}\)
Under standard and strict assumptions the random vector \(\mathbf{y}\) has expectation \(E(\mathbf{y}) = \mathbf{X}\mathbf{\beta}\) and variance \(Var(\mathbf{y}) = \sigma^2\mathbf{I}\)

Quick review of matrix algebra

Let \(\mathbf{A}\) be a \(k \times k\) matrix. The inverse of \(\mathbf{A}\), denoted by \(\mathbf{A}^{-1}\), is another \(k \times k\) matrix such that \(\mathbf{A}^{-1}\mathbf{A} = \mathbf{A}\mathbf{A}^{-1} = \mathbf{I}\)
Let \(\mathbf{A}\) be a \(n \times k\) matrix. The transpose of \(\mathbf{A}\), denoted by \(\mathbf{A}^T\), is a \(k \times n\) matrix, whose columns are the rows of \(\mathbf{A}\), vice versa.
If \(\mathbf{A}\) is a \(n \times m\) matrix and \(\mathbf{B}\) is a \(m \times p\) matrix, then \((\mathbf{A}\mathbf{B})^T\) = \(\mathbf{B}^T\mathbf{A}^T\)

Quick review of matrix algebra (cont.)

Let \(\mathbf{A}\) be a \(k \times k\) matrix
- \(\mathbf{A}\) is symmetric if \(\mathbf{A} = \mathbf{A}^T\)
- \(\mathbf{A}\) is idempotent if \(\mathbf{A}^2 = \mathbf{A}\)
- \(\mathbf{A}\) is orthonormal if \(\mathbf{A}^T\mathbf{A} = \mathbf{I}\)
Let \(\mathbf{y}\) be a \(k \times 1\) vector. The expression: \[\mathbf{y}^T\mathbf{A}\mathbf{y} = \sum_{i=1}^k\sum_{j=1}^k \mathbf{y}_i \cdot \mathbf{A[ij]} \cdot \mathbf{y}_j\] is called a quadratic form. In this case, \(\mathbf{A}\) is the matrix of the quadratic form.

Matrix derivatives

Let \(\mathbf{A}\) be a \(k \times k\) matrix, \(\mathbf{a}\) be a \(k \times 1\) vector, and \(\mathbf{y}\) be a \(k \times 1\) vector of variables.

if z = \(\mathbf{a}^T\mathbf{y}\), then \(\frac{\partial z}{\partial \mathbf{y}} = \mathbf{a}\)
if z = \(\mathbf{y}^T\mathbf{y}\), then \(\frac{\partial z}{\partial \mathbf{y}} = 2\mathbf{y}\)
if z = \(\mathbf{a}^T\mathbf{A}\mathbf{y}\), then \(\frac{\partial z}{\partial \mathbf{y}} = \mathbf{A}^T\mathbf{a}\)

Matrix derivatives (cont.)

Let \(\mathbf{A}\) be a \(k \times k\) matrix, \(\mathbf{a}\) be a \(k \times 1\) vector, and \(\mathbf{y}\) be a \(k \times 1\) vector of variables.

if z = \(\mathbf{y}^T\mathbf{A}\mathbf{y}\), then \(\frac{\partial z}{\partial \mathbf{y}} = \mathbf{Ay} + \mathbf{A}^T\mathbf{y}\)
if z = \(\mathbf{y}^T\mathbf{A}\mathbf{y}\) with \(\mathbf{A}\) symmetric, then \(\frac{\partial z}{\partial \mathbf{y}} = 2\mathbf{Ay}\)

Least Squares Estimation

\(Q = \sum_{j=1}^n [y_i - (\beta_0 + \beta_1 x_{i1} + \ldots + \beta_k x_{ik})]^2\)
in matrix form \(Q = (\mathbf{y} - \mathbf{X}\mathbf{\beta})^T(\mathbf{y} - \mathbf{X}\mathbf{\beta})\)
minimize SEE with the score equation \(\frac{\partial Q}{\partial \mathbf{\beta}} = 0\) this yields \[\hat{\mathbf{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}\]

Estimating \(\sigma^2\)

\(MSE = \frac{SSE}{n-k-1}\)
\(SSE = (\mathbf{y} - \mathbf{X}\mathbf{\hat{\beta}})^T(\mathbf{y} - \mathbf{X}\mathbf{\hat{\beta}})\)

Fitted values and residuals

\(\hat{\mathbf{y}} = \mathbf{X}\mathbf{\hat{\beta}}\)
\(\hat{\mathbf{e}} = \mathbf{y} - \mathbf{X}\mathbf{\hat{\beta}}\)

Hat matrix

With the hat matrix \(\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\) we have:

\(\hat{\mathbf{y}} = \mathbf{Hy}\)

Similarly, the residuals can be expressed as:

\(\mathbf{e} = (\mathbf{I} - \mathbf{H})\mathbf{y}\)

Essential linear algebra for inference

Let \(\mathbf{A}\) be a \(k \times k\) matrix, \(\mathbf{y}\) be a \(k \times 1\) random vector with mean \(\mu\) and variance-covariance matrix \(\Sigma\), then:

\(Var(\mathbf{A}\mathbf{y}) = \mathbf{A}\Sigma\mathbf{A}^T\)
\(E(\mathbf{A}\mathbf{y}) = \mathbf{A}E(\mathbf{y})\)
\(E(\hat{\mathbf{\beta}}) = \mathbf{\beta}\)
\(Var(\hat{\mathbf{\beta}}) = \sigma^2\cdot(\mathbf{X}^T\mathbf{X})^{-1}\)
\(\widehat{Var(\hat{\mathbf{\beta}})} = MSE\cdot(\mathbf{X}^T\mathbf{X})^{-1}\)

Confidence intervals of \(\beta_j\)

The distribution of \(\hat{\beta}_j\) are: \[\frac{\hat{\beta}_j - \beta_j}{\sqrt{\widehat{Var(\hat{\beta}_j})}} \sim t_{n-k-1}\]
Thus the \(100\cdot(1-\alpha)\%\) confidence interval for \(\beta_j\) is: \[\hat{\beta}_j \pm t_{n-k-1, 1-\alpha/2}\cdot\sqrt{\widehat{Var(\hat{\beta}_j)}}\]

General F-test

For comparing two nested models \(M_{full}\) and \(M_{reduced}\) we can use the F-test:

\[F = \frac{(SSE_{reduced} - SSE_{full})/(df_{E}(full) - df_{E}(reduced))}{SSE_{full}/df_{full}}\]

\[F = \frac{(SSR_{full} - SSR_{reduced})/(df_{R}(full) - df_{R}(reduced))}{SSE_{full}/df_{full}}\]

The two formulas are equivalent.