Orthogonal bases and projections

Section 6.3 Orthogonal bases and projections

We know that a linear system

A x = b

is inconsistent when

b

is not in

Col (A),

the column space of

A .

Later in this chapter, we’ll develop a strategy for dealing with inconsistent systems by finding

\hat{b},

the vector in

Col (A)

that minimizes the distance to

b .

The equation

A x = \hat{b}

is therefore consistent and its solution set can provide us with useful information about the original system

A x = b .

🔗

In this section and the next, we’ll develop some techniques that enable us to find

\hat{b},

the vector in a given subspace

W

that is closest to a given vector

b .

🔗

Preview Activity 6.3.1.

For this activity, it will be helpful to recall the distributive property of dot products:

v \cdot (c_{1} w_{1} + c_{2} w_{2}) = c_{1} v \cdot w_{1} + c_{2} v \cdot w_{2} .

We’ll work with the basis of

R^{2}

formed by the vectors

w_{1} = [\begin{array}{r} 1 \\ 2 \end{array}], w_{2} = [\begin{array}{r} - 2 \\ 1 \end{array}] .

Verify that the vectors $w_{1}$ and $w_{2}$ are orthogonal.
🔗

🔗
Suppose that $b = [\begin{array}{r} 7 \\ 4 \end{array}]$ and find the dot products $w_{1} \cdot b$ and $w_{2} \cdot b .$
🔗

🔗
We would like to express $b$ as a linear combination of $w_{1}$ and $w_{2},$ which means that we need to find weights $c_{1}$ and $c_{2}$ such that

$b = c_{1} w_{1} + c_{2} w_{2} .$

To find the weight $c_{1},$ dot both sides of this expression with $w_{1} :$

$b \cdot w_{1} = (c_{1} w_{1} + c_{2} w_{2}) \cdot w_{1},$

and apply the distributive property.

🔗

🔗
In a similar fashion, find the weight $c_{2} .$
🔗

🔗
Verify that $b = c_{1} w_{1} + c_{2} w_{2}$ using the weights you have found.
🔗

🔗

🔗

We frequently ask to write a given vector as a linear combination of given basis vectors. In the past, we have done this by solving a linear system. The preview activity illustrates how this task can be simplified when the basis vectors are orthogonal to each other. We’ll explore this and other uses of orthogonal bases in this section.

🔗

Subsection 6.3.1 Orthogonal sets

The preview activity dealt with a basis of

R^{2}

formed by two orthogonal vectors. More generally, we will consider a set of orthogonal vectors, as described in the next definition.

🔗

Definition 6.3.1.

By an orthogonal set of vectors, we mean a set of nonzero vectors each of which is orthogonal to the others.

🔗

Example 6.3.2.

The 3-dimensional vectors

w_{1} = [\begin{array}{r} 1 \\ - 1 \\ 1 \end{array}], w_{2} = [\begin{array}{r} 1 \\ 1 \\ 0 \end{array}], w_{3} = [\begin{array}{r} 1 \\ - 1 \\ - 2 \end{array}] .

form an orthogonal set, which can be verified by computing

\begin{array}{rcl} w_{1} \cdot w_{2} & = & 0 \\ w_{1} \cdot w_{3} & = & 0 \\ w_{2} \cdot w_{3} & = & 0 . \end{array}

Notice that this set of vectors forms a basis for

R^{3} .

🔗

Example 6.3.3.

The vectors

w_{1} = [\begin{array}{r} 1 \\ 1 \\ 1 \\ 1 \end{array}], w_{2} = [\begin{array}{r} 1 \\ 1 \\ - 1 \\ - 1 \end{array}], w_{3} = [\begin{array}{r} 1 \\ - 1 \\ 1 \\ - 1 \end{array}]

form an orthogonal set of 4-dimensional vectors. Since there are only three vectors, this set does not form a basis for

R^{4} .

It does, however, form a basis for a 3-dimensional subspace

W

R^{4} .

🔗

Suppose that a vector

b

is a linear combination of an orthogonal set of vectors

w_{1}, w_{2}, \dots, w_{n};

that is, suppose that

c_{1} w_{1} + c_{2} w_{2} + \dots + c_{n} w_{n} = b .

Just as in the preview activity, we can find the weight

c_{1}

by dotting both sides with

w_{1}

and applying the distributive property of dot products:

\begin{aligned} (c_{1} w_{1} + c_{2} w_{2} + \dots + c_{n} w_{n}) \cdot w_{1} & = b \cdot w_{1} \\ c_{1} w_{1} \cdot w_{1} + c_{2} w_{2} \cdot w_{1} + \dots + c_{n} w_{n} \cdot w_{1} & = b \cdot w_{1} \\ c_{1} w_{1} \cdot w_{1} & = b \cdot w_{1} \\ c_{1} & = \frac{b \cdot w_{1}}{w_{1} \cdot w_{1}} . \end{aligned}

Notice how the presence of an orthogonal set causes most of the terms in the sum to vanish. In the same way, we find that

c_{i} = \frac{b \cdot w_{i}}{w_{i} \cdot w_{i}}

so that

b = \frac{b \cdot w_{1}}{w_{1} \cdot w_{1}} w_{1} + \frac{b \cdot w_{2}}{w_{2} \cdot w_{2}} w_{2} + \dots + \frac{b \cdot w_{n}}{w_{n} \cdot w_{n}} w_{n} .

🔗

We’ll record this fact in the following proposition.

🔗

Proposition 6.3.4.

If a vector

b

is a linear combination of an orthogonal set of vectors

w_{1}, w_{2}, \dots, w_{n},

then

b = \frac{b \cdot w_{1}}{w_{1} \cdot w_{1}} w_{1} + \frac{b \cdot w_{2}}{w_{2} \cdot w_{2}} w_{2} + \dots + \frac{b \cdot w_{n}}{w_{n} \cdot w_{n}} w_{n} .

🔗

Using this proposition, we can see that an orthogonal set of vectors must be linearly independent. Suppose, for instance, that

w_{1}, w_{2}, \dots, w_{n}

is a set of nonzero orthogonal vectors and that one of the vectors is a linear combination of the others, say,

w_{3} = c_{1} w_{1} + c_{2} w_{2} .

We therefore know that

w_{3} = \frac{w_{3} \cdot w_{1}}{w_{1} \cdot w_{1}} w_{1} + \frac{w_{3} \cdot w_{2}}{w_{2} \cdot w_{2}} w_{2} = 0,

which cannot happen since we know that

w_{3}

is nonzero. This tells us that

🔗

Proposition 6.3.5.

An orthogonal set of vectors

w_{1}, w_{2}, \dots, w_{n}

is linearly independent.

🔗

If the vectors in an orthogonal set have dimension

m,

they form a linearly independent set in

R^{m}

and are therefore a basis for the subspace

W = Span {w_{1}, w_{2}, \dots, w_{n}} .

If there are

m

vectors in the orthogonal set, they form a basis for

R^{m} .

🔗

Activity 6.3.2.

Consider the vectors

w_{1} = [\begin{array}{r} 1 \\ - 1 \\ 1 \end{array}], w_{2} = [\begin{array}{r} 1 \\ 1 \\ 0 \end{array}], w_{3} = [\begin{array}{r} 1 \\ - 1 \\ - 2 \end{array}] .

Verify that this set forms an orthogonal set of $3$ -dimensional vectors.
🔗
xxxxxxxxxx
Language:
Messages
🔗
Explain why we know that this set of vectors forms a basis for $R^{3} .$
🔗

🔗
Suppose that $b = [\begin{array}{r} 2 \\ 4 \\ - 4 \end{array}] .$ Find the weights $c_{1},$ $c_{2},$ and $c_{3}$ that express $b$ as a linear combination $b = c_{1} w_{1} + c_{2} w_{2} + c_{3} w_{3}$ using Proposition 6.3.4.
🔗

🔗
If we multiply a vector $v$ by a positive scalar $s,$ the length of $v$ is also multiplied by $s;$ that is, $| s v | = s | v | .$
🔗
Using this observation, find a vector $u_{1}$ that is parallel to $w_{1}$ and has length 1. Such vectors are called unit vectors.
xxxxxxxxxx
Language:
Messages
🔗
🔗
Similarly, find a unit vector $u_{2}$ that is parallel to $w_{2}$ and a unit vector $u_{3}$ that is parallel to $w_{3} .$
🔗

🔗
Construct the matrix $Q = [\begin{matrix} u_{1} & u_{2} & u_{3} \end{matrix}]$ and find the product $Q^{T} Q .$ Use Proposition 6.2.8 to explain your result.
🔗

🔗

🔗

This activity introduces an important way of modifying an orthogonal set so that the vectors in the set have unit length. Recall that we may multiply any nonzero vector

w

by a scalar so that the new vector has length 1. For instance, we know that if

s

is a positive scalar, then

| s w | = s | w | .

To obtain a vector

u

having unit length, we want

| u | = | s w | = s | w | = 1

so that

s = 1 / | w | .

Therefore,

u = \frac{1}{| w |} w

becomes a unit vector parallel to

w .

🔗

Orthogonal sets in which the vectors have unit length are called orthonormal and are especially convenient.

🔗

Definition 6.3.6.

An orthonormal set is an orthogonal set of vectors each of which has unit length.

🔗

Example 6.3.7.

The vectors

u_{1} = [\begin{array}{r} 1 / \sqrt{2} \\ 1 / \sqrt{2} \end{array}], u_{2} = [\begin{array}{r} - 1 / \sqrt{2} \\ 1 / \sqrt{2} \end{array}]

are an orthonormal set of vectors in

R^{2}

and form an orthonormal basis for

R^{2} .

🔗

If we form the matrix

Q = [\begin{matrix} u_{1} & u_{2} \end{matrix}] = [\begin{matrix} 1 / \sqrt{2} & - 1 / \sqrt{2} \\ 1 / \sqrt{2} & 1 / \sqrt{2} \end{matrix}],

we find that

Q^{T} Q = I

since Proposition 6.2.8 tells us that

Q^{T} Q = [\begin{matrix} u_{1} \cdot u_{1} & u_{1} \cdot u_{2} \\ u_{2} \cdot u_{1} & u_{2} \cdot u_{2} \end{matrix}] = [\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix}]

🔗

The previous activity and example illustrate the next proposition.

🔗

Proposition 6.3.8.

If the columns of the

m \times n

matrix

Q

form an orthonormal set, then

Q^{T} Q = I_{n},

the

n \times n

identity matrix.

🔗

Subsection 6.3.2 Orthogonal projections

We now turn to an important problem that will appear in many forms in the rest of our explorations. Suppose, as shown in Figure 6.3.9, that we have a subspace

W

R^{m}

and a vector

b

that is not in that subspace. We would like to find the vector

\hat{b}

W

that is closest to

b,

meaning the distance between

\hat{b}

and

b

is as small as possible.

🔗

Figure 6.3.9. Given a plane in $R^{3}$ and a vector $b$ not in the plane, we wish to find the vector $\hat{b}$ in the plane that is closest to $b .$
🔗

To get started, let’s consider a simpler problem where we have a line

L

R^{2},

defined by the vector

w,

and another vector

b

that is not on the line, as shown on the left of Figure 6.3.10. We wish to find

\hat{b},

the vector on the line that is closest to

b,

as illustrated in the right of Figure 6.3.10.

🔗

Figure 6.3.10. Given a line $L$ and a vector $b,$ we seek the vector $\hat{b}$ on $L$ that is closest to $b .$
🔗

To find

\hat{b},

we require that

b - \hat{b}

be orthogonal to

L .

For instance, if

y

is another vector on the line, as shown in Figure 6.3.11, then the Pythagorean theorem implies that

{| b - y |}^{2} = | b - \hat{b} |^{2} + | \hat{b} - y |^{2}

which means that

| b - y | \geq | b - \hat{b} | .

Therefore,

\hat{b}

is closer to

b

than any other vector on the line

L .

🔗

Figure 6.3.11. The vector $\hat{b}$ is closer to $b$ than $y$ because $b - \hat{b}$ is orthogonal to $L .$
🔗

Definition 6.3.12.

Given a vector

b

R^{m}

and a subspace

W

R^{m},

the orthogonal projection of

b

onto

W

is the vector

\hat{b}

W

that is closest to

b .

It is characterized by the property that

b - \hat{b}

is orthogonal to

W .

🔗

Activity 6.3.3.

This activity demonstrates how to determine the orthogonal projection of a vector onto a subspace of

R^{m} .

Let’s begin by considering a line $L,$ defined by the vector $w = [\begin{array}{r} 2 \\ 1 \end{array}],$ and a vector $b = [\begin{array}{r} 2 \\ 4 \end{array}]$ not on $L,$ as illustrated in Figure 6.3.13.
🔗

Figure 6.3.13. Finding the orthogonal projection of $b$ onto the line defined by $w .$
🔗
1. To find $\hat{b},$ first notice that $\hat{b} = s w$ for some scalar $s .$ Since $b - \hat{b} = b - s w$ is orthogonal to $w,$ what do we know about the dot product
  
  $(b - s w) \cdot w ?$
  
  🔗
  
  🔗
2. Apply the distributive property of dot products to find the scalar $s .$ What is the vector $\hat{b},$ the orthogonal projection of $b$ onto $L ?$
  🔗
  
  🔗
3. More generally, explain why the orthogonal projection of $b$ onto the line defined by $w$ is
  
  $\hat{b} = \frac{b \cdot w}{w \cdot w} w .$
  
  🔗
  
  🔗
🔗
🔗
The same ideas apply more generally. Suppose we have an orthogonal set of vectors $w_{1} = [\begin{array}{r} 2 \\ 2 \\ - 1 \end{array}]$ and $w_{2} = [\begin{array}{r} 1 \\ 0 \\ 2 \end{array}]$ that define a plane $W$ in $R^{3} .$ If $b = [\begin{array}{r} 3 \\ 9 \\ 6 \end{array}]$ another vector in $R^{3},$ we seek the vector $\hat{b}$ on the plane $W$ closest to $b .$ As before, the vector $b - \hat{b}$ will be orthogonal to $W,$ as illustrated in Figure 6.3.14.
🔗

Figure 6.3.14. Given a plane $W$ defined by the orthogonal vectors $w_{1}$ and $w_{2}$ and another vector $b,$ we seek the vector $\hat{b}$ on $W$ closest to $b .$
🔗
1. The vector $b - \hat{b}$ is orthogonal to $W .$ What does this say about the dot products: $(b - \hat{b}) \cdot w_{1}$ and $(b - \hat{b}) \cdot w_{2} ?$
  🔗
  
  🔗
2. Since $\hat{b}$ is in the plane $W,$ we can write it as a linear combination $\hat{b} = c_{1} w_{1} + c_{2} w_{2} .$ Then
  
  $b - \hat{b} = b - (c_{1} w_{1} + c_{2} w_{2}) .$
  
  Find the weight $c_{1}$ by dotting $b - \hat{b}$ with $w_{1}$ and applying the distributive property of dot products. Similarly, find the weight $c_{2} .$
  
  🔗
  
  🔗
3. What is the vector $\hat{b},$ the orthogonal projection of $b$ onto the plane $W ?$
  🔗
  
  🔗
🔗
🔗
Suppose that $W$ is a subspace of $R^{m}$ with orthogonal basis $w_{1}, w_{2}, \dots, w_{n}$ and that $b$ is a vector in $R^{m} .$ Explain why the orthogonal projection of $b$ onto $W$ is the vector

$\hat{b} = \frac{b \cdot w_{1}}{w_{1} \cdot w_{1}} w_{1} + \frac{b \cdot w_{2}}{w_{2} \cdot w_{2}} w_{2} + \dots + \frac{b \cdot w_{n}}{w_{n} \cdot w_{n}} w_{n} .$

🔗

🔗
Suppose that $u_{1}, u_{2}, \dots, u_{n}$ is an orthonormal basis for $W;$ that is, the vectors are orthogonal to one another and have unit length. Explain why the orthogonal projection is

$\hat{b} = (b \cdot u_{1}) u_{1} + (b \cdot u_{2}) u_{2} + \dots + (b \cdot u_{n}) u_{n} .$

🔗

🔗
If $Q = [\begin{matrix} u_{1} & u_{2} & \dots & u_{n} \end{matrix}]$ is the matrix whose columns are an orthonormal basis of $W,$ use Proposition 6.2.8 to explain why $\hat{b} = Q Q^{T} b .$
🔗

🔗

🔗

In all the cases considered in the activity, we are looking for

\hat{b},

the vector in a subspace

W

closest to a vector

b,

which is found by requiring that

b - \hat{b}

be orthogonal to

W .

This means that

(b - \hat{b}) \cdot w = 0

for any vector

w

W .

🔗

If we have an orthogonal basis

w_{1}, w_{2}, \dots, w_{n}

for

W,

then

\hat{b} = c_{1} w_{1} + c_{w} w_{2} + \dots + c_{n} w_{n} .

Therefore,

\begin{aligned} (b - \hat{b}) \cdot w_{i} & = 0 \\ b \cdot w_{i} & = \hat{b} \cdot w_{i} \\ b \cdot w_{i} & = (c_{1} w_{1} + c_{2} w_{2} + \dots + c_{n} w_{n}) \cdot w_{i} \\ b \cdot w_{i} & = c_{i} w_{i} \cdot w_{i} \\ c_{i} & = \frac{b \cdot w_{i}}{w_{i} \cdot w_{i}} . \end{aligned}

This leads to the projection formula:

🔗

Proposition 6.3.15. Projection formula.

W

is a subspace of

R^{m}

having an orthogonal basis

w_{1}, w_{2}, \dots, w_{n}

and

b

is a vector in

R^{m},

then the orthogonal projection of

b

onto

W

\hat{b} = \frac{b \cdot w_{1}}{w_{1} \cdot w_{1}} w_{1} + \frac{b \cdot w_{2}}{w_{2} \cdot w_{2}} w_{2} + \dots + \frac{b \cdot w_{n}}{w_{n} \cdot w_{n}} w_{n} .

🔗

Caution.

Remember that the projection formula given in Proposition 6.3.15 applies only when the basis

w_{1}, w_{2}, \dots, w_{n}

W

is orthogonal.

🔗

If we have an orthonormal basis

u_{1}, u_{2}, \dots, u_{n}

for

W,

the projection formula simplifies to

\hat{b} = (b \cdot u_{1}) u_{1} + (b \cdot u_{2}) u_{2} + \dots + (b \cdot u_{n}) u_{n} .

If we then form the matrix

Q = [\begin{matrix} u_{1} & u_{2} & \dots & u_{n} \end{matrix}],

this expression may be succintly written

\begin{aligned} \hat{b} & = (b \cdot u_{1}) u_{1} + (b \cdot u_{2}) u_{2} + \dots + (b \cdot u_{n}) u_{n} \\ = [\begin{array}{c} u_{1} & u_{2} & \dots & u_{n} \end{array}] [\begin{array}{c} u_{1} \cdot b \\ u_{2} \cdot b \\ ⋮ \\ u_{n} \cdot b \end{array}] \\ = Q Q^{T} b \end{aligned}

🔗

This leads to the following proposition.

🔗

Proposition 6.3.16.

u_{1}, u_{2}, \dots, u_{n}

is an orthonormal basis for a subspace

W

R^{m},

then the matrix transformation that projects vectors in

R^{m}

orthogonally onto

W

is represented by the matrix

P = Q Q^{T}

where

Q = [\begin{matrix} u_{1} & u_{2} & \dots & u_{n} \end{matrix}] .

🔗

Example 6.3.17.

In the previous activity, we looked at the plane

W

defined by the two orthogonal vectors

w_{1} = [\begin{array}{r} 2 \\ 2 \\ - 1 \end{array}], w_{2} = [\begin{array}{r} 1 \\ 0 \\ 2 \end{array}] .

We can form an orthonormal basis by scalar multiplying these vectors to have unit length:

u_{1} = \frac{1}{3} [\begin{array}{r} 2 \\ 2 \\ - 1 \end{array}] = [\begin{array}{r} 2 / 3 \\ 2 / 3 \\ - 1 / 3 \end{array}], u_{2} = \frac{1}{\sqrt{5}} [\begin{array}{r} 1 \\ 0 \\ 2 \end{array}] = [\begin{array}{r} 1 / \sqrt{5} \\ 0 \\ 2 / \sqrt{5} \end{array}] .

Using these vectors, we form the matrix

Q = [\begin{matrix} 2 / 3 & 1 / \sqrt{5} \\ 2 / 3 & 0 \\ - 1 / 3 & 2 / \sqrt{5} \end{matrix}] .

The projection onto the plane

W

is then given by the matrix

Q Q^{T} = [\begin{matrix} 2 / 3 & 1 / \sqrt{5} \\ 2 / 3 & 0 \\ - 1 / 3 & 2 / \sqrt{5} \end{matrix}] [\begin{matrix} 2 / 3 & 2 / 3 & - 1 / 3 \\ 1 / \sqrt{5} & 0 & 2 / \sqrt{5} \end{matrix}] = [\begin{matrix} 29 / 45 & 4 / 9 & 8 / 45 \\ 4 / 9 & 4 / 9 & - 2 / 9 \\ 8 / 45 & - 2 / 9 & 41 / 45 \end{matrix}] .

🔗

Let’s check that this works by considering the vector

b = [\begin{array}{r} 1 \\ 0 \\ 0 \end{array}]

and finding

\hat{b},

its orthogonal projection onto the plane

W .

In terms of the original basis

w_{1}

and

w_{2},

the projection formula from Proposition 6.3.15 tells us that

\hat{b} = \frac{b \cdot w_{1}}{w_{1} \cdot w_{1}} w_{1} + \frac{b \cdot w_{2}}{w_{2} \cdot w_{2}} w_{2} = [\begin{matrix} 29 / 45 \\ 4 / 9 \\ 8 / 45 \end{matrix}]

🔗

Alternatively, we use the matrix

Q Q^{T},

as in Proposition 6.3.16, to find that

\hat{b} = Q Q^{T} b = [\begin{matrix} 29 / 45 & 4 / 9 & 8 / 45 \\ 4 / 9 & 4 / 9 & - 2 / 9 \\ 8 / 45 & - 2 / 9 & 41 / 45 \end{matrix}] [\begin{array}{r} 1 \\ 0 \\ 0 \end{array}] = [\begin{matrix} 29 / 45 \\ 4 / 9 \\ 8 / 45 \end{matrix}] .

🔗

Activity 6.3.4.

Suppose that $L$ is the line in $R^{3}$ defined by the vector $w = [\begin{array}{r} 1 \\ 2 \\ - 2 \end{array}] .$
xxxxxxxxxx

Language:
Messages
1. Find an orthonormal basis $u$ for $L .$
  🔗
  
  🔗
2. Construct the matrix $Q = [\begin{matrix} u \end{matrix}]$ and use it to construct the matrix $P$ that projects vectors orthogonally onto $L .$
  🔗
  
  🔗
3. Use your matrix to find $\hat{b},$ the orthogonal projection of $b = [\begin{array}{r} 1 \\ 1 \\ 1 \end{array}]$ onto $L .$
  🔗
  
  🔗
4. Find $rank (P)$ and explain its geometric significance.
  🔗
  
  🔗
🔗
🔗
The vectors

$w_{1} = [\begin{array}{r} 1 \\ 1 \\ 1 \\ 1 \end{array}], w_{2} = [\begin{array}{r} 0 \\ 1 \\ 1 \\ - 2 \end{array}]$
form an orthogonal basis of $W,$ a two-dimensional subspace of $R^{4} .$
xxxxxxxxxx

Language:
Messages
1. Use the projection formula from Proposition 6.3.15 to find $\hat{b},$ the orthogonal projection of $b = [\begin{array}{r} 9 \\ 2 \\ - 2 \\ 3 \end{array}]$ onto $W .$
  🔗
  
  🔗
2. Find an orthonormal basis $u_{1}$ and $u_{2}$ for $W$ and use it to construct the matrix $P$ that projects vectors orthogonally onto $W .$ Check that $P b = \hat{b},$ the orthogonal projection you found in the previous part of this activity.
  🔗
  
  🔗
3. Find $rank (P)$ and explain its geometric significance.
  🔗
  
  🔗
4. Find a basis for $W^{⊥} .$
  🔗
  
  🔗
5. Find a vector $b^{⊥}$ in $W^{⊥}$ such that
  
  $b = \hat{b} + b^{⊥} .$
  
  🔗
  
  🔗
6. If $Q$ is the matrix whose columns are $u_{1}$ and $u_{2},$ find the product $Q^{T} Q$ and explain your result.
  🔗
  
  🔗
🔗
🔗

🔗

This activity demonstrates one issue of note. We found

\hat{b},

the orthogonal projection of

b

onto

W,

by requiring that

b - \hat{b}

be orthogonal to

W .

In other words,

b - \hat{b}

is a vector in the orthogonal complement

W^{⊥},

which we may denote

b^{⊥} .

This explains the following proposition, which is illustrated in Figure 6.3.19

🔗

Proposition 6.3.18.

W

is a subspace of

R^{n}

with orthogonal complement

W^{⊥},

then any

n

-dimensional vector

b

can be uniquely written as

b = \hat{b} + b^{⊥}

where

\hat{b}

is in

W

and

b^{⊥}

is in

W^{⊥} .

The vector

\hat{b}

is the orthogonal projection of

b

onto

W

and

b^{⊥}

is the orthogonal projection of

b

onto

W^{⊥} .

🔗

Figure 6.3.19. A vector $b$ along with $\hat{b},$ its orthogonal projection onto the line $L,$ and $b^{⊥},$ its orthogonal projection onto the orthogonal complement $L^{⊥} .$
🔗

Let’s summarize what we’ve found. If

Q

is a matrix whose columns

u_{1}, u_{2}, \dots, u_{n}

form an orthonormal set in

R^{m},

then

$Q^{T} Q = I_{n},$ the $n \times n$ identity matrix, because this product computes the dot products between the columns of $Q .$
🔗

🔗
$Q Q^{T}$ is the matrix the projects vectors orthogonally onto $W,$ the subspace of $R^{m}$ spanned by $u_{1}, \dots, u_{n} .$
🔗

🔗

As we’ve said before, matrix multiplication depends on the order in which we multiply the matrices, and we see this clearly here.

🔗

Because

Q^{T} Q = I,

there is a temptation to say that

Q

is invertible. This is usually not the case, however. Remember that an invertible matrix must be a square matrix, and the matrix

Q

will only be square if

n = m .

In this case, there are

m

vectors in the orthonormal set so the subspace

W

spanned by the vectors

u_{1}, u_{2}, \dots, u_{m}

R^{m} .

b

is a vector in

R^{m},

then

\hat{b} = Q Q^{T} b

is the orthogonal projection of

b

onto

R^{m} .

In other words,

Q Q^{T} b

is the closest vector in

R^{m}

b,

and this closest vector must be

b

itself. Therefore,

Q Q^{T} b = b,

which means that

Q Q^{T} = I .

In this case,

Q

is an invertible matrix.

🔗

Example 6.3.20.

Consider the orthonormal set of vectors

u_{1} = [\begin{array}{r} 1 / \sqrt{3} \\ - 1 / \sqrt{3} \\ 1 / \sqrt{3} \end{array}], u_{2} = [\begin{array}{r} 1 / \sqrt{2} \\ 1 / \sqrt{2} \\ 0 \end{array}]

and the matrix they define

Q = [\begin{matrix} 1 / \sqrt{3} & 1 / \sqrt{2} \\ - 1 / \sqrt{3} & 1 / \sqrt{2} \\ 1 / \sqrt{3} & 0 \end{matrix}] .

In this case,

u_{1}

and

u_{2}

span a plane, a 2-dimensional subspace of

R^{3} .

We know that

Q^{T} Q = I_{2}

and

Q Q^{T}

projects vectors orthogonally onto the plane. However,

Q

is not a square matrix so it cannot be invertible.

🔗

Example 6.3.21.

Now consider the orthonormal set of vectors

u_{1} = [\begin{array}{r} 1 / \sqrt{3} \\ - 1 / \sqrt{3} \\ 1 / \sqrt{3} \end{array}], u_{2} = [\begin{array}{r} 1 / \sqrt{2} \\ 1 / \sqrt{2} \\ 0 \end{array}], u_{3} = [\begin{array}{r} 1 / \sqrt{6} \\ - 1 / \sqrt{6} \\ - 2 / \sqrt{6} \end{array}]

and the matrix they define

Q = [\begin{matrix} 1 / \sqrt{3} & 1 / \sqrt{2} & 1 / \sqrt{6} \\ - 1 / \sqrt{3} & 1 / \sqrt{2} & - 1 / \sqrt{6} \\ 1 / \sqrt{3} & 0 & - 2 / \sqrt{6} \end{matrix}] .

Here,

u_{1},

u_{2},

and

u_{3}

form a basis for

R^{3}

so that both

Q^{T} Q = I_{3}

and

Q Q^{T} = I_{3} .

Therefore,

Q

is a square matrix and is invertible.

🔗

Moreover, since

Q^{T} Q = I,

we see that

Q^{- 1} = Q^{T}

so finding the inverse of

Q

is as simple as writing its transpose. Matrices with this property are very special and will play an important role in our upcoming work. We will therefore give them a special name.

🔗

Definition 6.3.22.

A square

m \times m

matrix

Q

whose columns form an orthonormal basis for

R^{m}

is called orthogonal.

🔗

This terminology can be a little confusing. We call a basis orthogonal if the basis vectors are orthogonal to one another. However, a matrix is orthogonal if the columns are orthogonal to one another and have unit length. It pays to keep this in mind when reading statements about orthogonal bases and orthogonal matrices. In the meantime, we record the following proposition.

🔗

Proposition 6.3.23.

An orthogonal matrix

Q

is invertible and its inverse

Q^{- 1} = Q^{T} .

🔗

Subsection 6.3.3 Summary

This section introduced orthogonal sets and the projection formula that allows us to project vectors orthogonally onto a subspace.

Given an orthogonal set $w_{1}, w_{2}, \dots, w_{n}$ that spans an $n$ -dimensional subspace $W$ of $R^{m},$ the orthogonal projection of $b$ onto $W$ is the vector in $W$ closest to $b$ and may be written as

$\hat{b} = \frac{b \cdot w_{1}}{w_{1} \cdot w_{1}} w_{1} + \frac{b \cdot w_{2}}{w_{2} \cdot w_{2}} w_{2} + \dots + \frac{b \cdot w_{n}}{w_{n} \cdot w_{n}} w_{n} .$

🔗

🔗
If $u_{1}, u_{2}, \dots, u_{n}$ is an orthonormal basis of $W$ and $Q$ is the matrix whose columns are $u_{i},$ then the matrix $P = Q Q^{T}$ projects vectors orthogonally onto $W .$
🔗

🔗
If the columns of $Q$ form an orthonormal basis for an $n$ -dimensional subspace of $R^{m},$ then $Q^{T} Q = I_{n} .$
🔗

🔗
An orthogonal matrix $Q$ is a square matrix whose columns form an orthonormal basis. In this case, $Q Q^{T} = Q^{T} Q = I$ so that $Q^{- 1} = Q^{T} .$
🔗

🔗

🔗

Exercises 6.3.4 Exercises

1.

Suppose that

w_{1} = [\begin{array}{r} 1 \\ 1 \\ 1 \end{array}], w_{2} = [\begin{array}{r} 1 \\ - 2 \\ 1 \end{array}] .

Verify that $w_{1}$ and $w_{2}$ form an orthogonal basis for a plane $W$ in $R^{3} .$
🔗

🔗
Use Proposition 6.3.15 to find $\hat{b},$ the orthogonal projection of $b = [\begin{array}{r} 2 \\ 1 \\ - 1 \end{array}]$ onto $W .$
🔗

🔗
Find an orthonormal basis $u_{1},$ $u_{2}$ for $W .$
🔗

🔗
Find the matrix $P$ representing the matrix transformation that projects vectors in $R^{3}$ orthogonally onto $W .$ Verify that $\hat{b} = P b .$
🔗

🔗
Determine $rank (P)$ and explain its geometric significance.
🔗

🔗

🔗

2.

Consider the vectors

w_{1} = [\begin{array}{r} 1 \\ 1 \\ 1 \end{array}], w_{2} = [\begin{array}{r} - 1 \\ 0 \\ 1 \end{array}], w_{3} = [\begin{array}{r} 1 \\ - 2 \\ 1 \end{array}] .

Explain why these vectors form an orthogonal basis for $R^{3} .$
🔗

🔗
Suppose that $A = [\begin{matrix} w_{1} & w_{2} & w_{3} \end{matrix}]$ and evaluate the product $A^{T} A .$ Why is this product a diagonal matrix and what is the significance of the diagonal entries?
🔗

🔗
Express the vector $b = [\begin{array}{r} - 3 \\ - 6 \\ 3 \end{array}]$ as a linear combination of $w_{1},$ $w_{2},$ and $w_{3} .$
🔗

🔗
Multiply the vectors $w_{1},$ $w_{2},$ $w_{3}$ by appropriate scalars to find an orthonormal basis $u_{1},$ $u_{2},$ $u_{3}$ of $R^{3} .$
🔗

🔗
If $Q = [\begin{matrix} u_{1} & u_{2} & u_{3} \end{matrix}],$ find the matrix product $Q Q^{T}$ and explain the result.
🔗

🔗

🔗

3.

Suppose that

w_{1} = [\begin{array}{r} 1 \\ 1 \\ 0 \\ - 1 \end{array}], w_{2} = [\begin{array}{r} 1 \\ 0 \\ 1 \\ 1 \end{array}]

form an orthogonal basis for a subspace

W

R^{4} .

Find $\hat{b},$ the orthogonal projection of $b = [\begin{array}{r} 2 \\ - 1 \\ - 6 \\ 7 \end{array}]$ onto $W .$
🔗

🔗
Find the vector $b^{⊥}$ in $W^{⊥}$ such that $b = \hat{b} + b^{⊥} .$
🔗

🔗
Find a basis for $W^{⊥} .$ and express $b^{⊥}$ as a linear combination of the basis vectors.
🔗

🔗

🔗

4.

Consider the vectors

w_{1} = [\begin{array}{r} 1 \\ 1 \\ 0 \\ 0 \end{array}], w_{2} = [\begin{array}{r} 0 \\ 0 \\ 1 \\ 1 \end{array}], b = [\begin{array}{r} 2 \\ - 4 \\ 1 \\ 3 \end{array}] .

If $L$ is the line defined by the vector $w_{1},$ find the vector in $L$ closest to $b .$ Call this vector ${\hat{b}}_{1} .$
🔗

🔗
If $W$ is the subspace spanned by $w_{1}$ and $w_{2},$ find the vector in $W$ closest to $b .$ Call this vector ${\hat{b}}_{2} .$
🔗

🔗
Determine whether ${\hat{b}}_{1}$ or ${\hat{b}}_{2}$ is closer to $b$ and explain why.
🔗

🔗

🔗

5.

Suppose that

w = [\begin{array}{r} 2 \\ - 1 \\ 2 \end{array}]

defines a line

L

R^{3} .

Find the orthogonal projections of the vectors $[\begin{array}{r} 1 \\ 0 \\ 0 \end{array}],$ $[\begin{array}{r} 0 \\ 1 \\ 0 \end{array}],$ $[\begin{array}{r} 0 \\ 0 \\ 1 \end{array}]$ onto $L .$
🔗

🔗
Find the matrix $P = \frac{1}{{| w |}^{2}} w w^{T} .$
🔗

🔗
Use Proposition 2.5.6 to explain why the columns of $P$ are related to the orthogonal projections you found in the first part of this exercise.
🔗

🔗

🔗

6.

Suppose that

v_{1} = [\begin{array}{r} 1 \\ 0 \\ 3 \end{array}], v_{2} = [\begin{array}{r} 2 \\ 2 \\ 2 \end{array}]

form the basis for a plane

W

R^{3} .

Find a basis for the line that is the orthogonal complement $W^{⊥} .$
🔗

🔗
Given the vector $b = [\begin{array}{r} 6 \\ - 6 \\ 2 \end{array}],$ find $y,$ the orthogonal projection of $b$ onto the line $W^{⊥} .$
🔗

🔗
Explain why the vector $z = b - y$ must be in $W$ and write $z$ as a linear combination of $v_{1}$ and $v_{2} .$
🔗

🔗

🔗

7.

Determine whether the following statements are true or false and explain your thinking.

If the columns of $Q$ form an orthonormal basis for a subspace $W$ and $w$ is a vector in $W,$ then $Q Q^{T} w = w .$
🔗

🔗
An orthogonal set of vectors in $R^{8}$ can have no more than 8 vectors.
🔗

🔗
If $Q$ is a $7 \times 5$ matrix whose columns are orthonormal, then $Q Q^{T} = I_{7} .$
🔗

🔗
If $Q$ is a $7 \times 5$ matrix whose columns are orthonormal, then $Q^{T} Q = I_{5} .$
🔗

🔗
If the orthogonal projection of $b$ onto a subspace $W$ satisfies $\hat{b} = 0,$ then $b$ is in $W^{⊥} .$
🔗

🔗

🔗

8.

Suppose that

Q

is an orthogonal matrix.

Remembering that $v \cdot w = v^{T} w,$ explain why

$Q x \cdot (Q y) = x \cdot y .$

🔗

🔗
Explain why $| Q x | = | x | .$
🔗

This means that the length of a vector is unchanged after multiplying by an orthogonal matrix.
🔗

🔗
If $λ$ is a real eigenvalue of $Q,$ explain why $λ = \pm 1 .$
🔗

🔗

🔗

9.

Explain why the following statements are true.

If $Q$ is an orthogonal matrix, then $det Q = \pm 1 .$
🔗

🔗
If $Q$ is a $8 \times 4$ matrix whose columns are orthonormal, then $Q Q^{T}$ is an $8 \times 8$ matrix whose rank is 4.
🔗

🔗
If $\hat{b}$ is the orthogonal projection of $b$ onto a subspace $W,$ then $b - \hat{b}$ is the orthogonal projection of $b$ onto $W^{⊥} .$
🔗

🔗

🔗

10.

This exercise is about

2 \times 2

orthogonal matrices.

In Section 2.6, we saw that the matrix $[\begin{matrix} \cos θ & - \sin θ \\ \sin θ & \cos θ \end{matrix}]$ represents a rotation by an angle $θ .$ Explain why this matrix is an orthogonal matrix.
🔗

🔗
We also saw that the matrix $[\begin{matrix} \cos θ & \sin θ \\ \sin θ & - \cos θ \end{matrix}]$ represents a reflection in a line. Explain why this matrix is an orthogonal matrix.
🔗

🔗
Suppose that $u_{1} = [\begin{array}{r} \cos θ \\ \sin θ \end{array}]$ is a 2-dimensional unit vector. Use a sketch to indicate all the possible vectors $u_{2}$ such that $u_{1}$ and $u_{2}$ form an orthonormal basis of $R^{2} .$
🔗

🔗
Explain why every $2 \times 2$ orthogonal matrix is either a rotation or a reflection.
🔗

🔗

🔗

You have attempted 1 of 1 activities on this page.

🔗

Prev Top Next