How to Assessing Model Accuracy ?

The Quality of Fit

$$\begin{array}{c} MSE=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{f}(x_i))^2 \end{array}$$

training MSE

• computed using the training data
• LESS IMPORTANT!

test MSE

• $Ave(y_0-\hat{f}(x_0))^2$
• $(x_0,y_0)$ is the observations which is not used to training model. Test Observations
• MORE IMPORTANT!
• How to get test observations?
  • cross-validation (Chapter 05)

training MSE v.s. test MSE

• model more flexible (degree of freedom increase)
  • training MSE decrease
  • test MSE may decrease firstly, then become increase

• overfitting
  • when a model yields small training MSE but a large test MSE
  • as a result of model learning some patterns caused by random error.

The Bias-Variance Trade-Off

$$\begin{array}{c} EE(y_0-\hat{f}(x_0))^2 = Var(\hat{f}(x_0))+\big[Bias(\hat{f}(x_0))\big]^2 + Var(\epsilon)\\ (y_0-\hat{f}(x_0))^2 = \boldsymbol{Variance} + \boldsymbol{Bias} + \boldsymbol{Noise} \end{array}$$

variance

• the error caused by different training data
  • the more flexible model (higher degree of freedom) has larger variance

bias

• the error caused by the simplify model
  • the more flexible model (higher degree of freedom) has smaller bias

The Classification Setting

The Quality of Fit

• error rate $$\begin{array}{cc} I(y_{i}\neq\hat{y}_{i})= \left\{ \begin{array}{ll} 0, \rm{if}(y_i=\hat{y}_i)\\ 1, \rm{if}(y_i \neq \hat{y}_i)\\ \end{array} \right. \end{array}$$

  • training error rate $$\begin{array}{c} \frac{1}{n}\sum_{i=1}^{n}I(y_i\neq\hat{y}_i) \end{array}$$

  • test error rate $$\begin{array}{c} Ave(I(y_{0}\neq\hat{y}_{0})) \end{array}$$

The Bayes Classifier

$$\begin{array}{c} Pr(Y=j|X=x_0) \end{array}$$

when given the observed predictor vector $x_{0}$
the probability that $Y=j$

Idea Situation

two-class example

• class 1 and class 2
• $\left\{\begin{array}{ll} \hat{Y}\in \text{class 1},\text{ if}\ Pr(Y = 1|X=x_0)>0.5;\\ \hat{Y}\in \text{class 2},\text{ if}\ Pr(Y = 1|X=x_0)<0.5;\\ \end{array} \right.$

Bayes decision boundary

• the points where the probability is exactly 0.5
• determined the Bayes classifier's prediction

  the purple dashed line

Bayes error rate

$$\begin{array}{cc} 1-E\bigg({\rm\mathop{max}\limits_{j}Pr}(Y=j|X)\bigg) \end{array}$$

• The lowest possible test error rate produced by Bayes classifier.
• Analogous to irreducible error

K-Nearest Neighbors

$$\begin{array}{cc} {\rm Pr}(Y=j|X=x_{0})=\frac{1}{K}\sum_{i \in \mathcal{N}_{0}}I(y_{i}=j) \end{array}$$

$x_0$

test observation

$K$

the number of points which be selected near the $x_{0}$

$K \uparrow$

• Border less flexible
• high variance, low bias

$K \downarrow$

• Border more flexible
• high bias, low variance

$\mathcal{N}_{0}$

the selected training data ($K$ points)