Simple Linear Regression

$$\begin{array}{cc} Y \approx \beta_{0}+\beta_{1}X \end{array}$$

Estimating the Coefficients

least squares

$$\begin{array}{cc} \left\{ \begin{array}{ll} {\rm min}\bigg(\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^2\bigg)\\ \hat{y}_{i}=\beta_{0}+\beta_{1}x_i \end{array} \right.\\\\ \big\Downarrow\\\\ {\rm min}\bigg(\sum_{i=1}^{n}(y_{i}-\beta_{0}-\beta_{1}x_i)^2 \bigg)\\\\ \big\Downarrow\\\\ \color{#D0104C}{ \left\{ \begin{array}{ll} \hat{\beta}_{1} =\frac{\sum_{i=1}^{n}(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sum_{i=1}^{n}(x-\bar{x})^{2}} =\frac{\sum_{i=1}^{n}x_{i}y_{i}-n\cdot{\bar{x}\bar{y}}}{\sum_{i=1}^{n}x_{i}^{2}-n\cdot{\bar{x}^2}} =\frac{S_{xy}}{S_{x}}\\ \hat{\beta}_{0}=\bar{y}-\hat{\beta}_{1}\bar{x} \end{array} \right.} \end{array}$$

Assessing the Accuracy of Coefficient Estiments

$$\begin{array}{ll} Y=\beta_{0}+\beta_{1}X+\epsilon \end{array}$$

$\beta_{0}$

intercept term

the expected value of $Y$ when $X=0$

$\beta_{1}$

slope

the average increase in $Y$ with one-unit $X$

$\epsilon$

Error term

Reasons

• the true relationship may not a linear

• other variables that cause variation in Y

• measure error

population regression line

• best linear approximation to true relationship between $X$ and $Y$ when we know all population's situation

least squares line

• the linear approximation by least square estimate based on the observed data

stand error

$$\begin{array}{ll} Var(\bar{y})=SE(\bar{y})^2=\frac{\sigma^{2}}{n} \end{array}$$

Stand Error of $\beta_{1}$

$SE^{2}(\hat{\beta}_1)=\frac{\sigma^{2}}{S_{xx}}=\frac{\sigma^{2}}{\sum_{i=1}^{n}(x_{i}-\bar{x})^2}$

• as x spread out, the SE of slope decrease

Stand Error of $\beta_{0}$

$SE^{2}(\hat{\beta}_0)=\frac{\sigma^{2}}{S_{xx}}=\sigma^{2}\bigg[\frac{1}{n}+\frac{\bar{x}^{2}}{\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}}\bigg]$

• as x spread out, the SE of intercept decrease

• as mean of x near to zero, the SE of intercept decrease

How to estimate residual standard error $\sigma$

$\hat{\sigma}=RSE=\sqrt{\frac{RSS}{n-2}}=\sqrt{\frac{\sum_{i=1}^{n}(Y_i-\hat{Y}_i)^2}{n-2}}$

Hypothesis statistic

t-statistic

$$\left\{ \begin{array}{ll} t=\frac{\hat{\beta}_{1}-0}{SE(\hat{\beta})}\\ df=n-2 \end{array} \right.$$

Assessing the Accuracy of the Model

$RSE$

$$RSE=\sqrt{\frac{RSS}{n-2}}=\sqrt{\frac{\sum_{i=1}^{n}{(y_{i}-\hat{y}_{i})^{2}}}{n-2}}$$

Residual Standard Error

• an estimate of the standard deviation of $\epsilon$
• measure of the lack of fit of the model to the data
• measured in the units of $Y$

$R^{2}$

$$\left\{ \begin{aligned} & R^{2}=\frac{TSS-RSS}{TSS}=1-\frac{RSS}{TSS}\\ & TSS=\sum_{i=1}^{n}(y_i-\bar{y})^{2} \end{aligned} \right.$$

not depend on the units of Y

$TSS$

total sum of squares the total variance in the response $Y$

$RSS$

the amount of variability that is left unexplained after performing regression

$R^{2}=r^{2}$

$r$

the pearson correlation coefficient between $X$ and $Y$