Orthogonality and Least Squares

Skip to main content

\(\newcommand{\avec}{{\mathbf a}} \newcommand{\bvec}{{\mathbf b}} \newcommand{\cvec}{{\mathbf c}} \newcommand{\dvec}{{\mathbf d}} \newcommand{\dtil}{\widetilde{\mathbf d}} \newcommand{\evec}{{\mathbf e}} \newcommand{\fvec}{{\mathbf f}} \newcommand{\nvec}{{\mathbf n}} \newcommand{\pvec}{{\mathbf p}} \newcommand{\qvec}{{\mathbf q}} \newcommand{\svec}{{\mathbf s}} \newcommand{\tvec}{{\mathbf t}} \newcommand{\uvec}{{\mathbf u}} \newcommand{\vvec}{{\mathbf v}} \newcommand{\wvec}{{\mathbf w}} \newcommand{\xvec}{{\mathbf x}} \newcommand{\yvec}{{\mathbf y}} \newcommand{\zvec}{{\mathbf z}} \newcommand{\rvec}{{\mathbf r}} \newcommand{\mvec}{{\mathbf m}} \newcommand{\zerovec}{{\mathbf 0}} \newcommand{\onevec}{{\mathbf 1}} \newcommand{\real}{{\mathbb R}} \newcommand{\twovec}[2]{\left[\begin{array}{r}#1 \\ #2 \end{array}\right]} \newcommand{\ctwovec}[2]{\left[\begin{array}{c}#1 \\ #2 \end{array}\right]} \newcommand{\threevec}[3]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \end{array}\right]} \newcommand{\cthreevec}[3]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \end{array}\right]} \newcommand{\fourvec}[4]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]} \newcommand{\cfourvec}[4]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]} \newcommand{\fivevec}[5]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]} \newcommand{\cfivevec}[5]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]} \newcommand{\mattwo}[4]{\left[\begin{array}{rr}#1 \amp #2 \\ #3 \amp #4 \\ \end{array}\right]} \newcommand{\laspan}[1]{\text{Span}\{#1\}} \newcommand{\bcal}{{\cal B}} \newcommand{\ccal}{{\cal C}} \newcommand{\scal}{{\cal S}} \newcommand{\wcal}{{\cal W}} \newcommand{\ecal}{{\cal E}} \newcommand{\coords}[2]{\left\{#1\right\}_{#2}} \newcommand{\gray}[1]{\color{gray}{#1}} \newcommand{\lgray}[1]{\color{lightgray}{#1}} \newcommand{\rank}{\operatorname{rank}} \newcommand{\row}{\text{Row}} \newcommand{\col}{\text{Col}} \renewcommand{\row}{\text{Row}} \newcommand{\nul}{\text{Nul}} \newcommand{\var}{\text{Var}} \newcommand{\corr}{\text{corr}} \newcommand{\len}[1]{\left|#1\right|} \newcommand{\bbar}{\overline{\bvec}} \newcommand{\bhat}{\widehat{\bvec}} \newcommand{\bperp}{\bvec^\perp} \newcommand{\xhat}{\widehat{\xvec}} \newcommand{\vhat}{\widehat{\vvec}} \newcommand{\uhat}{\widehat{\uvec}} \newcommand{\what}{\widehat{\wvec}} \newcommand{\Sighat}{\widehat{\Sigma}} \newcommand{\lt}{<} \newcommand{\gt}{>} \newcommand{\amp}{&} \definecolor{fillinmathshade}{gray}{0.9} \newcommand{\fillinmath}[1]{\mathchoice{\colorbox{fillinmathshade}{$\displaystyle \phantom{\,#1\,}$}}{\colorbox{fillinmathshade}{$\textstyle \phantom{\,#1\,}$}}{\colorbox{fillinmathshade}{$\scriptstyle \phantom{\,#1\,}$}}{\colorbox{fillinmathshade}{$\scriptscriptstyle\phantom{\,#1\,}$}}} \)

Chapter 6 Orthogonality and Least Squares

We introduced vectors as a means to develop visual intuition about our basic questions concerning linear systems. For example, vectors allow us to reinterpret questions about the existence of solutions to linear systems as questions about the span of a set of vectors. Questions about the uniqueness of solutions led to the concept of linear independence.

In this chapter, we will begin to think of vectors as geometric objects that have lengths and that form angles. In some cases, this will simplify our search for solutions to a linear system. Perhaps more importantly, we will be able to measure the distance between vectors. This means that if a system \(A\xvec=\bvec\) is inconsistent, we can look for \(\xhat\text{,}\) the vector for which \(A\xhat\) is as close to \(\bvec\) as possible. This leads to the method of least squares, which underpins regression, a key tool in data science.