Section04 Independence, Basis and Dimension

Basis vectors: Independent vectors that "span the space".

Every vector in the space is a unique combination of the basis vectors.

Heart of this Section

• Independent vectors
• Spanning space
• Basis for a space
• Dimension of a space

Linear Independence

Defination

1. The Columns of $\boldsymbol{A}$ are linearly independent when the only solution to $\boldsymbol{Ax} = \boldsymbol{0}$ is $\boldsymbol{x} = \boldsymbol{0}$. No other combination of $\boldsymbol{Ax}$ of the columns gives the zero vector.
2. The sequence of vectors $\boldsymbol{v}_{1}, \cdots, \boldsymbol{v}_{n}$ is linear independent if the only combinaton that give the zero vector is $0 \boldsymbol{v}_{1}+ 0 \boldsymbol{v}_{2} + \cdots + 0 \boldsymbol{v}_{n}$ ($x_{1}\boldsymbol{v}_{1} + x_{2}\boldsymbol{v}_{2} + \cdots + x_{n}\boldsymbol{v}_{n} = \boldsymbol{0}$ only happens when all $x$'s are zero.)
3. There is not a vector which is the linear combiantion of others.

Attentions

Desctiption

• Independent vectors but not Independent Matrices.

Dependent and Independent Situations

Dependent

• $\boldsymbol{0}$ is dependent with any vectors.
• Three vectors in $\mathbb{R}^{2}$ can't be independent!
  • Understand from free variable: If $\boldsymbol{A}$ is a 2 by 3 matrix, then there are at least one free variable, therefore, the nullspace of $\boldsymbol{A}$ is not only consist of the zero vector. These three vectors are dependent.
• Any set of $n$ vectors in $\mathbb{R}^{m}$ must be linearly dependent if $n > m$. (There must be free variables of the solution for $\boldsymbol{Ax} = \boldsymbol{0}$)

Independent

• When $\boldsymbol{A}$ is a Full column rank matrix ($n = r$), There are $n$ pivots and no free variables. Only $\boldsymbol{x} = \boldsymbol{0}$ is in the nullspace. ($n \le m$)

Vectors that Span a Subspace

Defination

Span

• A set of vectors spans a space if their linear combination fill the space.

Row Space

• The row space of a matrix is the subspace of $\mathbb{R}^{n}$ spanned by the rows. (Represented by $\boldsymbol{C}(\boldsymbol{A}^{T})$, the column space of $\boldsymbol{A}^{T}$)

Column Space and Row Space

• The row are in $\mathbb{R}^{n}$ spanning the row space. (It's a subspace of $\mathbb{R}^{n}$)
• The column are in $\mathbb{R}^{m}$ spanning the column space. (It's a subspace of $\mathbb{R}^{m}$)

A Basis for a Vector Space

Defination

Basis

• A basis for a vector space is a sequence of vectors with two properties

  • The basis vectors are linearly independent.
  • They span the space.

• There is one and only one way to write $\boldsymbol{v}$ as a combination of the basis vectors. (Each $\boldsymbol{v}$ is unique.)

  • proof $$\begin{array}{ll} & \text{Suppose } \boldsymbol{v} = a_{1}v_{1} + a_{2}v_{2} + \cdots + a_{n}v_{n} \text{ and also } \boldsymbol{v} = b_{1}v_{1} + b_{2}v_{2} + \cdots + b_{n}v_{n} \\ \therefore & \boldsymbol{v} - \boldsymbol{v} = (a_{1} - b_{1}) v_{1} + (a_{2} - b_{2}) v_{2} + \cdots + (a_{n} - b_{n}) v_{n} = 0\\ \because & v_{1}, v_{2}, \cdots , v_{n} \text{ are independent. There isn't nonzero solution for } \boldsymbol{Vx} = 0\\ \therefore & (a_{1} - b_{1}) = (a_{2} - b_{2}) = \cdots = (a_{n} - b_{n}) = 0\\ \therefore & \boldsymbol{a} = \boldsymbol{b} \text{ and the vector } \boldsymbol{v} \text{ is unique.} \\ \end{array}$$

Standard Basis

• The columns of the $n$ by $n$ identity matrix give the Standard Basis for $\mathbb{R}^{n}$.

Common Basis

• The vectors $\boldsymbol{v}_{1}, \cdots, \boldsymbol{v}_{n}$ are a basis for $\mathbb{R}^{n}$ exactly when they are the columns of an $n$ by $n$ invertible matrix.
• The Pivot columns of $\boldsymbol{A}$ are basis for its column space. The pivot rows of $\boldsymbol{A}$ are a basis for its row space. So are the pivot rows of its echelon form $\boldsymbol{R}$.
• $\boldsymbol{R}$
  • The nonzero rows of $\boldsymbol{R}$ could be the basis for the row space of $\boldsymbol{A}$.
  • But the column space of $\boldsymbol{R}$ is not as the same as $\boldsymbol{A}$'s

Example

• Given five vectors in $\mathbb{R}^{7}$, how do you find a basis for the space they span?
  1. Make these five vectors as the rows of $\boldsymbol{A}$, elimate to find the nonzero rows of $\boldsymbol{R}$.
  2. Make these five vectors as the columns of $\boldsymbol{B}$, elimate to find the pivot columns of $\boldsymbol{B}$ (not $\boldsymbol{R}$!).

Dimension of a Vector Space

• All bases for a vector space contain the same number of vectors.

  • If $\boldsymbol{v}_{1}, \cdots, \boldsymbol{v}_{m}$ and $\boldsymbol{w}_{1}, \cdots, \boldsymbol{w}_{n}$ are both bases for the same vector space, then $m = n$.
    • proof $$\begin{array}{ll} & \text{Suppose there are more } \boldsymbol{w}\text{'s than } \boldsymbol{v} \text{'s (} \boldsymbol{n > m} \text{)} \\ \because & \boldsymbol{v} \text{ is the bases of vector space.} \\ \therefore & \boldsymbol{W} = \left[ \begin{matrix} \boldsymbol{w}_{1} & \boldsymbol{w}_{2} & \cdots & \boldsymbol{w}_{n} \end{matrix} \right] = \left[ \begin{matrix} \boldsymbol{v}_{1} & \boldsymbol{v}_{2} & \cdots & \boldsymbol{v}_{m} \end{matrix} \right] \left[ \begin{matrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{matrix} \right]\\ & = \boldsymbol{VA}\\ \because & m > n \\ \therefore & \text{There is a nonzero solution to } \boldsymbol{Ax} = \boldsymbol{0} \\ \therefore & \text{There is a nonzero solution to } \boldsymbol{Wx} = \boldsymbol{VAx} = 0\\ \therefore & \boldsymbol{w}_{1}, \cdots, \boldsymbol{w}_{n} \text{ are linearly dependent. They couldn't be the basis for vector space.} \\ \end{array}$$

• The number of vectors is the dimension of vector space.

  • Defination of Dimension
    • The dimension of a space is the number of vectors in evey basis.

• $\textbf{The dimension of the column space} = \textbf{Rank of the matrix.}$

Bases for Matrix Spaces and Function Spaces

Dimension of Matrices

Function Spaces

Properties from Exercises

1. Supppose $\boldsymbol{v}_{1}, \cdots, \boldsymbol{v}_{n}$ is a basis for $\mathbb{R}^{n}$ and the $n$ by $n$ matrix $\boldsymbol{A}$ is invertible. $\boldsymbol{Av}_{1}, \boldsymbol{Av}_{2}, \cdots, \boldsymbol{Av}_{n}$ is also a basis for $\mathbb{R}^{n}$.
   • proof from Matrix Perspective $$\begin{array}{ll} & \text{Put } \boldsymbol{v}_{1}, \boldsymbol{v}_{2}, \cdots, \boldsymbol{v}_{n} \text{ as columns of new matrix } \boldsymbol{V} \\ \because & \boldsymbol{A} \text{ and } \text{V} \text{ is invertible.} \\ \therefore & \boldsymbol{AV} \text{ is invertible.}\\ \therefore & \text{The columns of } \boldsymbol{AV} (\boldsymbol{Av}_{i})\text{ could be the basis of } \mathbb{R}^{n} \\ \end{array}$$
   • proof from Vector Perspective $$\begin{array}{ll} & \text{Suppose } \sum_{i=1}^{n}{c_{i}\boldsymbol{Av}_{i}} = \boldsymbol{0} \\ \therefore & \boldsymbol{A}\sum_{i=1}^{n}c_{i}\boldsymbol{v}_{i} = \boldsymbol{0} \\ \because & \boldsymbol{A} \text{ is invertible.}\\ \therefore & \boldsymbol{A}^{-1}\boldsymbol{A} \sum_{i=1}^{n}{c_{i}\boldsymbol{v}_{i}} = \boldsymbol{A}^{-1}\boldsymbol{0} \\ & \sum_{i=1}^{n}{c_{i} \boldsymbol{v}_{i}} = \boldsymbol{c}^{T}\boldsymbol{v} = \boldsymbol{0} \\ \because & \boldsymbol{v}_{1}, \cdots, \boldsymbol{v}_{n} \text{ are independent.} \\ \therefore & \boldsymbol{c} = \boldsymbol{0} \\ \therefore & \boldsymbol{Av}_{1}, \cdots, \boldsymbol{Av}_{n} \text{ are independent, are the basis of } \mathbb{R}^{n}. \\ \end{array}$$

Properties from Exercies

1. The row space and nullspace stay fixed after elimination. But the column space might be changed during the elimination.
2. $\cap$
   • example: Space $\boldsymbol{S}$ is the subspace of vectors $(a,b,c,d)$ with $a+c+d = 0$; Space $\boldsymbol{T}$ is the subspace of vectors $(a, b, c, d)$ with $a+b = 0$ and $c = 2d$. q1: find a basis for $\boldsymbol{S}$ and $\boldsymbol{T}$. q2: find the instersection $\boldsymbol{S}\cap \boldsymbol{T}$.
   • q1 $$\begin{array}{ll} \because & \boldsymbol{S} \text{ is the null space of } \left[ \begin{matrix} 1 & 0 & 1 & 1 \end{matrix} \right] \\ & \text{and } \boldsymbol{T} \text{ is the nullspace of } \left[ \begin{matrix} 1 & 1 & 0 & 0 \\ 0 & 0 & 1 & -2 \end{matrix} \right] \\ \therefore & \text{One basis of } \boldsymbol{S} \text{ is:} \\ & \left[ \begin{matrix} 0 \\ 1 \\ 0 \\ 0 \end{matrix} \right] , \left[ \begin{matrix} -1 \\ 0 \\ 1 \\ 0 \end{matrix} \right], \left[ \begin{matrix} -1 \\ 0 \\ 0 \\ 1 \end{matrix} \right] \\ & \text{One basis of } \boldsymbol{T} \text{ is:} \\ & \left[ \begin{matrix} -1 \\ 1 \\ 0 \\ 0 \end{matrix} \right], \left[ \begin{matrix} 0 \\ 0 \\ 2 \\ 1 \end{matrix} \right] \end{array}$$
   • q2 $$\begin{array}{ll} \because & \boldsymbol{S} \cap \boldsymbol{T} \text{ is the nullspace of } \left[ \begin{matrix} 1 & 0 & 1 & 1 \\ 1 & 1 & 0 & 0 \\ 0 & 0 & 1 & -2 \end{matrix} \right]\\ \therefore & \left[ \begin{matrix} -3 \\ 3 \\ 2 \\ 1 \end{matrix} \right] \text{ is a basis of the space of } \boldsymbol{S} \cap \boldsymbol{T}\\ \end{array}$$
3. Suppose $\boldsymbol{A}$ is 5 by 4 with rank 4, $\boldsymbol{Ax} = \boldsymbol{b}$ has no solution when $\left[\begin{matrix}\boldsymbol{A} & \boldsymbol{b}\end{matrix}\right]$ is invertible ($\boldsymbol{b}$ is not the linearly combination of the $\boldsymbol{A}$'s columns), but when this augment matrix is singular ($\boldsymbol{b}$ is the linearly combination of $\boldsymbol{A}$'s columns), there is solutions to $\boldsymbol{Ax} = \boldsymbol{b}$.
4. $\boldsymbol{V} + \boldsymbol{W} = \{\boldsymbol{v} + \boldsymbol{w}:(\boldsymbol{v} \in \boldsymbol{V})(\boldsymbol{w} \in \boldsymbol{W}) \}(\boldsymbol{V} \text{ and } \boldsymbol{W} \text{ are the subspaces of the same space.})$
   • $\dim(\boldsymbol{V}+\boldsymbol{W}) = \dim(\boldsymbol{V}) + \dim(\boldsymbol{W}) - \dim(\boldsymbol{V} \cap \boldsymbol{W})$
   • The meaning of $\boldsymbol{V} \cap \boldsymbol{W}$: the vectors of $\boldsymbol{V}$ which $\in \boldsymbol{V} \cap \boldsymbol{W}$, is the linearly combination of vectors of $\boldsymbol{W}$ which $\in \boldsymbol{V} \cap \boldsymbol{W}$.
   • The dimension of $\boldsymbol{V} + \boldsymbol{W}$ is equal to the dimension of $\left[\begin{matrix}\boldsymbol{V} & \boldsymbol{W}\end{matrix}\right]$. It consist of three parts: the vectors which is contained in $\boldsymbol{V}$ only, the vectors which is contained in $\boldsymbol{W}$ only, the vectors which is contained in the $\boldsymbol{V}$ and $\boldsymbol{W}$ at the same time (contained in $\boldsymbol{V} \cap \boldsymbol{W}$).
5. If $\boldsymbol{A}^{2} = \boldsymbol{0}$, each columns of $\boldsymbol{A}$ is in the nullspace of $\boldsymbol{A}$. $$\begin{array}{ll} \therefore & \boldsymbol{C}(\boldsymbol{A}) \subseteq \boldsymbol{\boldsymbol{N}}(\boldsymbol{A}) \\ \therefore & r \le n-r \\ \end{array}$$