Multiple Linear Regression

$$\begin{array}{c} y_{i}=\beta_{0}+\beta_{1}x_{i1}+\beta_{2}x_{i2}+\cdots+\beta_{p}x_{ip}+\epsilon_{i} = \beta_{0} + \sum_{j=1}^{p}\beta_{j}x_{ij} + \epsilon_{i}\\\\ \big\Downarrow\\\\ \mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\epsilon} \end{array}$$

$\mathbf{y}$

$$\left[ \begin{matrix} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \end{matrix} \right]$$

$\mathbf{X}$

$$\left[ \begin{matrix} 1 & x_{11} & x_{12} & \cdots & x_{1p} \\ 1 & x_{21} & x_{22} & \cdots & x_{2p} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n1} & x_{n2} & \cdots & x_{np} \\ \end{matrix} \right] = \left[ \begin{matrix} \mathbf{1}_{n} & \mathbf{x}_{1} & \mathbf{x}_{2} & \cdots & \mathbf{x}_{p} \end{matrix} \right]$$ $\mathbf{1}_{n}$

$n \times 1$ vector of ones.

$\mathbf{x}_{j}$

$$\left[ \begin{matrix} x_{1j}\\ x_{2j}\\ \vdots\\ x_{nj} \end{matrix} \right]$$

$\boldsymbol{\beta}$

$$\left[ \begin{matrix} \beta_{0}\\ \beta_{1}\\ \beta_{2}\\ \vdots\\ \beta_{p} \end{matrix} \right]$$

$\boldsymbol{\epsilon}$

$$\left[ \begin{matrix} \epsilon_{1} \\ \epsilon_{2} \\ \vdots \\ \epsilon_{n} \\ \end{matrix} \right]$$

Assumptions

Fitting a separate linear regression?

1. unclear how to make a single prediction of response given levels of different predictors, since each of the predictors is associated with a separate regression equotion.
2. each regression of different predictors ignores others predictors.

Estimating the Regression Coefficients

least squares

$$\begin{array}{c} \left\{ \begin{array}{ll} {\rm min}\bigg(\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^2\bigg)\\ \begin{array}{ll} \hat{y}_{i} & =\hat{\beta}_{0}+\hat{\beta}_{1}x_{i1}+\hat{\beta}_{12}x_{i2}+\hat{\beta}_{3}x_{i3}+\cdots+\hat{\beta}_{p}x_{ip}\\ & =\hat{\beta}_{0}+\sum_{j=1}^{p}\hat{\beta}_{j}x_{ij}+\epsilon \end{array} \end{array} \right.\\\\ \left\{ \begin{array}{ll} \mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}\\ \mathbf{y}= \left[ \begin{matrix} y_{1}\\y_{2}\\\vdots\\y_{n} \end{matrix} \right]; \ \ \mathbf{X}= \left[ \begin{matrix} 1&x_{11}&x_{12}&\cdots& x_{1p}\\ 1&x_{21}&x_{22}&\cdots& x_{2p}\\ \vdots&\vdots&\vdots&\ddots&\vdots\\ 1&x_{n1}&x_{n2}&\cdots& x_{np} \end{matrix} \right]\\ \boldsymbol{\beta}= \left[ \begin{matrix} \beta_{0}\\\beta_{1}\\\vdots\\\beta_{p} \end{matrix} \right];\ \ \boldsymbol{\epsilon}= \left[ \begin{matrix} \epsilon_{1}\\\epsilon_{2}\\\vdots\\\epsilon_{n} \end{matrix} \right] \end{array} \right.\\\\ \big\Downarrow\\\\ {\rm min}\bigg(\sum_{i=1}^{n}(y_{i}-\boldsymbol{x_{i}\beta})^{2}\bigg) = {\rm min}\bigg(\text{trace}\big((\mathbf{y}-\mathbf{X}\boldsymbol{\beta})\cdot(\mathbf{y}-\mathbf{X}\boldsymbol{\beta})^{\mathbf{T}}\big)\bigg)= \text{min}\big((\mathbf{y}-\mathbf{X}\boldsymbol{\beta})^{\mathbf{T}}\cdot(\mathbf{y}-\mathbf{X}\boldsymbol{\beta})\big) \\\\ \big\Downarrow\\\\ \color{#D0104C}{ \hat{\boldsymbol{\beta}}=(\mathbf{X'X})^{-1}\mathbf{X'y}\\} \end{array}$$

Hat matrix

$$\begin{array}{ll} \because & \hat{\mathbf{y}} = \mathbf{X}\hat{\boldsymbol{\beta}} = \mathbf{X}(\mathbf{X^{T}X})^{-1}\mathbf{X^{T}y} \\ & = \mathbf{Hy} \\ \therefore & \mathbf{H} = \mathbf{X}(\mathbf{X^{T}X})\mathbf{X^{T}} \\ \end{array}$$

Symmetric and idempotent matrix

$$\begin{array}{ll} \mathbf{HH} = \mathbf{H} \\ \mathbf{H^{T}} = \mathbf{H} \end{array}$$

How to explain the different result?

• simple linear regression

  • the amount of response increase as the predictor increase one-units.

• multiple linear regression

  • the amount of response increase as increase one predictor one-units and hold others predictors fixed.

Important Questions

Is There a Relationship Between the Response and Predictors?

Sums of Squares

$SST$

• Sums of Squares Total
• $SST = \sum_{i=1}^{n}({y_{i}-\bar{y}})^{2}$

  $\begin{array}{ll} \begin{array}{ll} SST & = \sum_{i=1}^{n}(y_{i} - \bar{y})^{2} = ( \mathbf{y} - \bar{y}\mathbf{I}_{n} )^{\mathbf{T}} ( \mathbf{y} - \bar{y}\mathbf{I}_{n} ) \\ & = ( \mathbf{y} - \frac{1}{n} \cdot \mathbf{I}_{n \times n}\mathbf{y} )^{\mathbf{T}} ( \mathbf{y} - \frac{1}{n} \cdot \mathbf{I}_{n \times n}\mathbf{y} ) \\ & = [ (1 - \frac{1}{n} \cdot \mathbf{I}_{n \times n}) \cdot \mathbf{y} ]^{\mathbf{T}} [ (1 - \frac{1}{n} \cdot \mathbf{I}_{n \times n}) \cdot \mathbf{y} ] \\ & = \mathbf{y^{T}}[( 1 - \frac{1}{n} \cdot \mathbf{I}_{n \times n} )^{\mathbf{T}}( 1 - \frac{1}{n} \cdot \mathbf{I}_{n \times n} )] \mathbf{y} \\ \end{array}\\ \begin{array}{ll} \because & [( 1 - \frac{1}{n} \cdot \mathbf{I}_{n \times n} )^{\mathbf{T}}( 1 - \frac{1}{n} \cdot \mathbf{I}_{n \times n} )] = 1 + \frac{1}{n^{2}} \cdot \mathbf{I}_{n \times n}^{2} - 2 \cdot \frac{1}{n} \cdot \mathbf{I}_{n \times n} \\ & = 1 - \frac{1}{n} \cdot \mathbf{I}_{n \times n}\\ \therefore & SST = \mathbf{y^{T}}(1 - \frac{1}{n} \cdot \mathbf{I}_{n \times n})\mathbf{y} \\ \end{array} \end{array}$

• $df_{T} = n-1$

$SSR$

• Sum of Squares Regression
• $SSR = \sum_{i=1}^{n}(\hat{y}_{i} - \bar{y})^{2}$

  $\begin{array}{ll} \begin{array}{ll} SSR & = \sum_{i=1}^{n}(\hat{y}_{i} - \bar{y})^{2} = ( \mathbf{Hy} - \bar{y}\mathbf{I}_{n} )^{\mathbf{T}} ( \mathbf{Hy} - \bar{y}\mathbf{I}_{n} ) \\ & = ( \mathbf{Hy} - \frac{1}{n} \cdot \mathbf{I}_{n \times n}\mathbf{y} )^{\mathbf{T}} ( \mathbf{Hy} - \frac{1}{n} \cdot \mathbf{I}_{n \times n}\mathbf{y} ) \\ & = [ (\mathbf{H} - \frac{1}{n} \cdot \mathbf{I}_{n \times n}) \cdot \mathbf{y} ]^{\mathbf{T}} [ (\mathbf{H} - \frac{1}{n} \cdot \mathbf{I}_{n \times n}) \cdot \mathbf{y} ] \\ & = \mathbf{y^{T}}[( \mathbf{H} - \frac{1}{n} \cdot \mathbf{I}_{n \times n} )^{\mathbf{T}}( \mathbf{H} - \frac{1}{n} \cdot \mathbf{I}_{n \times n} )] \mathbf{y} \\ \end{array}\\ \begin{array}{ll} \because & [( \mathbf{H} - \frac{1}{n} \cdot \mathbf{I}_{n \times n} )^{\mathbf{T}}( \mathbf{H} - \frac{1}{n} \cdot \mathbf{I}_{n \times n} )] = [( \mathbf{H} - \frac{1}{n} \cdot \mathbf{I}_{n \times n} )( \mathbf{H} - \frac{1}{n} \cdot \mathbf{I}_{n \times n} )] \\ & = \mathbf{H} + \frac{1}{n}\mathbf{I}_{n \times n} - \frac{1}{n} \cdot \mathbf{H}\mathbf{I}_{n \times n} - \frac{1}{n} \cdot \mathbf{I}_{n \times n}\mathbf{H}\\ & = \mathbf{H} + \frac{1}{n}\mathbf{I}_{n \times n} - 2 \frac{1}{n} \cdot \mathbf{H}\mathbf{I}_{n \times n} \\ & = 1 - \frac{1}{n} \cdot \mathbf{I}_{n \times n}\\ \therefore & SST = \mathbf{y^{T}}(1 - \frac{1}{n} \cdot \mathbf{I}_{n \times n})\mathbf{y} \\ \end{array} \end{array}$

Hypothesis test of model

For all coffients

$$\begin{array}{c} \begin{array}{ll} H_{0}: & \beta_{1} = \beta_{2} = \cdots = \beta_{p} = 0 \\ H_{1}: & \text{at least one } \beta_{j} \text{ is non-zero.} \end{array}\\\\\\ \left\{ \begin{array}{ll} TSS = \sum_{i=1}^{n}(y_{i}-\bar{y})^{2} = (\mathbf{y}-\bar{y} \cdot \mathbf{1})^{\mathbf{T}} \cdot (\mathbf{y}-\bar{y} \cdot \mathbf{1}) \\ RSS = \sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2} = ( \mathbf{y} - \mathbf{X}\hat{\boldsymbol{\beta}} )^{\mathbf{T}} \cdot ( \mathbf{y} - \mathbf{X}\hat{\boldsymbol{\beta}} ) \end{array} \right.\\\\ F=\frac{(TSS-RSS)/p}{RSS/(n-p-1)} \end{array}$$

For particular subset of $q$ of the coffients

• 此处例子为检验最后q个 coffcients $$\begin{array}{c} \begin{array}{ll} H_{0}: & \beta_{p-q+1} = \beta_{p-q+2} = \cdots = \beta_{p} = 0 \\ H_{1}: & \text{at least one } \beta_{j}\ (p-q+1 \le j \le p) \text{ is non-zero.} \end{array}\\\\\\ \left\{ \begin{array}{ll} RSS_{0} = \sum_{i=1}^{n}(y_{i}-\hat{y}_{i1})^{2} \\ RSS = \sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2} = ( \mathbf{y} - \mathbf{X}\hat{\boldsymbol{\beta}} )^{\mathbf{T}} \cdot ( \mathbf{y} - \mathbf{X}\hat{\boldsymbol{\beta}} ) \end{array} \right.\\ RSS_{0}: \text{the residual sum of second model, which fiting by all variables } \textbf{execpt last } q \text{ variables.} \\\\\\ F=\frac{(RSS_{0}-RSS)/q}{RSS/(n-p-1)} \end{array}$$

Hypothesis test of coffients

$$Var(\hat{\boldsymbol{\beta}}) = \sigma^{2}(\mathbf{X'X})^{-1}$$

t-test

$$t = \frac{\hat{\beta}_{i}-C}{\hat{\sigma}_{\beta_{i}}}$$