Ch4 Lecture 1
Geometrical Intuitions
Planes
More copilot insanity. I wrote the first sentence, and then it autocompleted as follows:
Have students put a dot in the center of the card and say this is the origin. . . . Have them draw a line and say this is the x-axis. . . . Have them draw a line perpendicular to the x-axis and say this is the y-axis.
. . .
Have them draw a line perpendicular to the x and y axes and say this is the z-axis.
. . .
Have them draw a line perpendicular to the x and y axes and say this is the w-axis.
. . .
Have them draw a line perpendicular to the x, y, and w axes and say this is the v-axis.
. . .
Have them draw a line perpendicular to the x, y, w, and v axes and say this is the u-axis.
. . .
Have them draw a line perpendicular to the x, y, w, v, and u axes and say this is the t-axis. Have them draw a line perpendicular to the x, y, w, v, u, and t axes and say this is the s-axis. Have them draw a line perpendicular to the x, y, w, v, u, t, and s axes and say this is the r-axis. Have them draw a line perpendicular to the x, y, w, v, u, t, s, and r axes and say this is the q-axis. Have them draw a line perpendicular to the x, y, w, v, u, t, s, r, and q axes and say this is the p-axis. Have them draw a line perpendicular to the x, y, w, v, u, t, s, r, q, and p axes and say this is the o-axis. Have them draw a line perpendicular to the x, y, w, v, u, t, s, r, q, p, and o axes and say this is the n-axis. Have them draw a line perpendicular to the x, y, w, v, u, t, s, r, q, p, o, and n axes and say this is the m-axis. Have them draw a line perpendicular to the x, y, w, v, u, t, s, r, q, p, o, n, and m axes and say this is the l-axis. Have them draw a line perpendicular to the x, y, w, v, u, t, s, r, q, p, o, n, m, and l axes and say this is the k-axis. Have them draw a line perpendicular to the x, y, w, v, u, t, s…
Finding the normal vector when we know two vectors in the plane
For \(\mathbb{R}^3\), we can find the normal vector by taking the cross product of two vectors in the plane.
Let \(\mathbf{i}, \mathbf{j}\), and \(\mathbf{k}\) represent the standard basis \(\mathbf{e}_{1}, \mathbf{e}_{2}, \mathbf{e}_{3}\).
\[ \mathbf{u} \times \mathbf{v}=\left(u_{2} v_{3}-u_{3} v_{2}\right) \mathbf{i}+\left(u_{3} v_{1}-u_{1} v_{3}\right) \mathbf{j}+\left(u_{1} v_{2}-u_{2} v_{1}\right) \mathbf{k}=\left|\begin{array}{ccc} \mathbf{i} & \mathbf{j} & \mathbf{k} \\ u_{1} & u_{2} & u_{3} \\ v_{1} & v_{2} & v_{3} \end{array}\right| \]
. . .
The cross product is orthogonal to both \(\mathbf{u}\) and \(\mathbf{v}\): \(\mathbf{u} \cdot \mathbf{u} \times \mathbf{v}=0\) and \(\mathbf{v} \cdot \mathbf{u} \times \mathbf{v}=0\)
. . .
We can use the cross product to find the normal vector to the plane, simply by taking the cross product of two vectors in the plane.
Finding the normal vector from the equation of the plane
If the plane is specified \(ax + by + cz = d\), then a normal vector is \(\vec a = ( a, b, c )\).
. . .
Check:
- Find three points on the plane.
- Set the first point where \(x = 0\) and \(y = 0\), so \(z = d/c\): \((0, 0, d/c)\).
- Set second point is where \(x=0\) and \(z = 0\), so \(y=d/b\): \((0,d/b,0)\).
- The third point is \((d/a,0,0)\).
- Now find two displacement vectors in the plane:
- \(\vec u = (0, d/b, 0) - (0, 0, d/c) = (0, d/b, -d/c)\) and
- \(\vec v = (d/a, 0, 0) - (0, 0, d/c)=(d/a,0,-d/c)\).
Now take the cross product of these two vectors to find the normal vector:
\[ \begin{aligned} \vec u \times \vec v \\ &= \left|\begin{array}{ccc} \mathbf{i} & \mathbf{j} & \mathbf{k} \\ 0 & d/b & -d/c \\ d/a & 0 & -d/c \end{array}\right| \\ &= - \mathbf{i} \left( \frac{d}{b} \frac{d}{c} \right) + \mathbf{j} \left( \frac{d}{a} \frac{d}{c} \right) - \mathbf{k} \left( \frac{d}{a} \frac{d}{b} \right) \\ &= -d^2 \left[ \left( \frac{1}{bc} \right) \mathbf{i} - \left( \frac{1}{ac} \right) \mathbf{j} + \left( \frac{1}{ab} \right) \mathbf{k} \right]\\ &= -\frac{d^2}{abc} \left[\left( a \right) \mathbf{i} - \left( b \right) \mathbf{j} + \left( c \right) \mathbf{k} \right]\\ \end{aligned} \]
This is parallel to \(\vec{a} = (a, b, c)\).
Distances from the plane
Back onto the plane
A plane not at the origin
Hyperplane
A hyperplane in \(\mathbb{R}^{n}\) is the set of all \(\mathbf{x} \in \mathbb{R}^{n}\) such that \(\mathbf{a} \cdot \mathbf{x}=b\), where the nonzero vector \(\mathbf{a} \in \mathbb{R}^{n}\) and scalar \(b\) are given.
. . .
Let \(H\) be the hyperplane in \(\mathbb{R}^{n}\) defined by the equation \(\mathbf{a} \cdot \mathbf{x}=b\) and let \(\mathbf{x}_{*} \in H\). Then
\(\mathbf{a}^{\perp}=\left\{\mathbf{y} \in \mathbb{R}^{n} \mid \mathbf{a} \cdot \mathbf{y}=0\right\}\) is a subspace of \(\mathbb{R}^{n}\) of dimension \(n-1\).
\(H=\mathbf{x}_{*}+\mathbf{a}^{\perp}=\left\{\mathbf{x}_{*}+\mathbf{y} \mid \mathbf{y} \in \mathbf{a}^{\perp}\right\}\).
Example
Find an equation that defines the plane containing the three (noncollinear) points \(P, Q\), and \(R\) with coordinates \((1,0,2),(2,1,0)\), and \((3,1,1)\), respectively.
. . .
\[ \begin{aligned} & \overrightarrow{P Q}=(2,1,0)-(1,0,2)=(1,1,-2) \\ & \overrightarrow{P R}=(3,1,1)-(1,0,2)=(2,1,-1) \end{aligned} \]
. . .
\[ \mathbf{u} \times \mathbf{v}=\left|\begin{array}{ccc} \mathbf{i} & \mathbf{j} & \mathbf{k} \\ 1 & 1 & -2 \\ 2 & 1 & -1 \end{array}\right|=\mathbf{i}-3 \mathbf{j}-\mathbf{k} \]
. . .
The normal vector is \(\mathbf{a}=(1,-3,-1)\).
. . .
The equation of the plane is \(\mathbf{a} \cdot \mathbf{x}=b\)
What is \(b\)? We can plug in any of the points and solve for it. Let’s use \(P\):
. . .
\(b=\mathbf{a} \cdot \mathbf{P}=(1,-3,-1) \cdot(1,0,2)=1+0-2=-1\).
Projections to the displaced plane
Summary
Plane through the origin: \(\vec a\cdot \vec x = 0\)
Plane displaced from the origin: \(\vec a\cdot \vec x = q\)
Component of \(\vec x\) along the vector \(\vec a\) (scalar projection): \(\vec a\cdot \vec x / || \vec a ||\)
Projection of \(\vec x\) onto \(\vec a\): \(\vec a / ||a|| \quad \vec a\cdot \vec x/||\vec a||\)
Projection of \(\vec x\) onto the plane for which \(\vec a\) is the normal vector, displaced by \(q \vec a\) from the origin: \(\vec x\) - projection onto the normal vector = $x - a (ax - q) $
Projection Formula for Vectors
Let \(\mathbf{u}\) and \(\mathbf{v}\) be vectors, \(\mathbf{v} \neq \mathbf{0}\).
\[ \mathbf{p}=\frac{\mathbf{v} \cdot \mathbf{u}}{\mathbf{v} \cdot \mathbf{v}} \mathbf{v} \text { and } \quad \mathbf{q}=\mathbf{u}-\mathbf{p} \]
. . .
Then \(\mathbf{p}\) is parallel to \(\mathbf{v}, \mathbf{q}\) is orthogonal to \(\mathbf{v}\), and \(\mathbf{u}=\mathbf{p}+\mathbf{q}\).
. . . We say \(\operatorname{proj}_{\mathbf{v}} \mathbf{u}=\frac{\mathbf{v} \cdot \mathbf{u}}{\mathbf{v} \cdot \mathbf{v}} \mathbf{v}\) and \(\operatorname{orth}_{\mathbf{v}} \mathbf{u}=\mathbf{u}-\operatorname{proj}_{\mathbf{v}} \mathbf{u}\)
Least Squares
Taking a step back from linear algebra: fitting a line to data
Straight lines of x and y, now in Rn
Imperfect data
Guessing at a fit
Best guess
Least squares
\(m \mathbf{x}-\mathbf{y}\) should be orthogonal to \(\mathbf{x}\).
\[ \begin{aligned} \mathbf{x} \cdot\left(m \mathbf{x}-\mathbf{y}\right) &=0 \\ m \mathbf{x} \cdot \mathbf{x}-\mathbf{x} \cdot \mathbf{y} &=0 \\ m \mathbf{x} \cdot \mathbf{x} &=\mathbf{x} \cdot \mathbf{y} \\ \mathbf{x} \cdot \mathbf{x}\quad m &=\mathbf{x} \cdot \mathbf{y} \\ \end{aligned} \]
. . .
Change to matrix notation:
\(\mathbf{x} \rightarrow \mathbf{A}\text{,}\qquad\) \(\mathbf{y} \rightarrow \mathbf{b}\text{,}\qquad\) \(m \rightarrow \mathbf{x}\)
. . .
Linear equation becomes \(\mathbf{A} \mathbf{x}= \mathbf{b}\). But it’s not consistent if the data aren’t on the straight line. Best fit equation becomes
. . .
\[ \begin{aligned} \mathbf{x} \cdot \mathbf{x}\quad m &=\mathbf{x} \cdot \mathbf{y} \\ \mathbf{A}^{T} \mathbf{A} \mathbf{x} &=\mathbf{A}^{T} \mathbf{b} \end{aligned} \]
These are the normal equations. We solve them to find \(\mathbf{x}\) which corresponds to the best fit.
Normal equations for a single predictor
When we have a single predictor variable and a single response variable, the \(\mathbf{x}\) we are trying to find is just a scalar.
. . .
\(\mathbf{A}\) is a column vector of the predictor variable, call it now \(\mathbf{a}\)
. . .
\(\mathbf{b}\) is a column vector of the response variable.
. . .
The normal equations become
\[ \begin{aligned} \mathbf{a}^{T} \mathbf{a} \mathbf{x} &=\mathbf{a}^{T} \mathbf{b} \\ \mathbf{a} \cdot \mathbf{a} \mathbf{x} &=\mathbf{a} \cdot \mathbf{b} \\ x = \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{a} \cdot \mathbf{a}} \end{aligned} \]
This is the formula for the slope of the best fit line.
Example
You have a scale that measures in unknown units.
You have items that you know weigh approximately 2, 5, and 7 pounds (approximately.)
. . .
You measure the scale’s output for these items and get \(0.7,2.4\), and 3.2. What is the conversion factor from the scale’s units to pounds?
. . .
You can set up the system of equations:
\(\mathbf{a}= \begin{bmatrix} 2 \\ 5 \\ 7 \end{bmatrix}\) and \(\mathbf{b}= \begin{bmatrix} 0.7 \\ 2.4 \\ 3.2 \end{bmatrix}\)
Can you find \(\mathbf{x}\) such that \(\mathbf{a} \cdot \mathbf{x} = \mathbf{b}\)?
. . .
No, it’s inconsistent. But you can find the best fit.
. . .
\[ \begin{aligned} \mathbf{a}^{T} \mathbf{a} \mathbf{x} &=\mathbf{a}^{T} \mathbf{b} \\ \begin{bmatrix} 2 & 5 & 7 \end{bmatrix} \begin{bmatrix} 2 \\ 5 \\ 7 \end{bmatrix} x &= \begin{bmatrix} 2 & 5 & 7 \end{bmatrix} \begin{bmatrix} 0.7 \\ 2.4 \\ 3.2 \end{bmatrix} x = \frac{2 \cdot 0.7 + 5 \cdot 2.4 + 7 \cdot 3.2}{2^2 + 5^2 + 7^2} \approx 0.459 \end{aligned} \]
So to go from pounds to the scale’s units, multiply by 0.459.
. . .
(This is the opposite of what the book did – they found the conversion factor from the scale’s units to pounds.)