Introduction to Linear Algebra(6) Orthogonal and Least Squares

Orthogonal and Least Squares

• Inner Product,Length, and Orthiginality
• Orthogonal Sets
• An Orthogonal Projection
• Orthonormal Sets
• The GRAM-SCHMIDT PROCESS
• QR Factorization of Matrices
• Least-Squares Problems
• Least-Squares Lines
• Inner Product spaces
• Trend Analysis of Data
• Fourier Series

Inner Product,Length, and Orthiginality

Two vectors $u$u and $v$v in $R^n$Rn are orthogonal if $u\cdot v=0$u⋅v=0.
Two vectors $u$u and $v$v are orthogonal if and only if $||u+v||^2=||u||^2+||v||^2$∣∣u+v∣∣2=∣∣u∣∣2+∣∣v∣∣2.
Let A be an $m \times n$m×n matrix. The orthogonal complement of the row space of $A$A is the null space of $A$A, and teh orthogonal somplement of the column space of $A$A is the null space of $A^\perp$A⊥:$(RowA)^{\perp} = Nul A , (ColA)^{\perp}=NulA^{\perp}$(RowA)⊥=NulA,(ColA)⊥=NulA⊥

Orthogonal Sets

A set of vectors KaTeX parse error: Can't use function '\u' in math mode at position 14: \{u_1,\cdots,\̲u̲_p\} in $R^n$Rn is said to be an orthogonal set if each pair of distinct vectors from theset is orthogonal, that is, if KaTeX parse error: Expected 'EOF', got '\cdotu' at position 4: u_i\̲c̲d̲o̲t̲u̲_j=0 whenever $i\ne j$i̸​=j.
Let $\{u_1,\cdots,u_p\}${u1​,⋯,up​} be an orthogonal baisis for a subspace $W$W of $R^n$Rn. For each $y$y in $W$W, the weights in the lienar combination$y=c_1u_1+\cdots+c_pu_p$y=c1​u1​+⋯+cp​up​
are given by $c_j=\frac{y\cdot u_j}{u_j\cdot u_j} (j=1,\cdots,P)$cj​=uj​⋅uj​y⋅uj​​(j=1,⋯,P)

An Orthogonal Projection

KaTeX parse error: Expected '}', got '\cdotu' at position 24: …proj_Ly=\frac{y\̲c̲d̲o̲t̲u̲}{u\cdotu}u

Orthonormal Sets

A set $\{u_1,\cdots,u_p\}${u1​,⋯,up​} is an orthonormal set if it is an oorthogonal set of
An $m \times n$m×n matrix $U$U has orthonormal columns if and only if $U^TU=I$UTU=I.
Let $U$U be an $m \times n$m×n matrix with orthonormal columns, and let $x$x and $y$y be in $R^n$Rn. Then
a.$||Ux||=||x||$∣∣Ux∣∣=∣∣x∣∣
b.$(Ux)\cdot(uy)=x\cdot y$(Ux)⋅(uy)=x⋅y
c.$(Ux)\cdot(uy) = 0$(Ux)⋅(uy)=0 if and only if $x \cdot y=0$x⋅y=0
The best approsimation Theorem
Let $W$W be a subspace of $R^n$Rn, let $y$y be any vector in $R^n$Rn, and let $\hat{y}$y^​ be the orthegonal projection of $y$y onto $W$W. Then $\hat{y}$y^​ is the closet point in $W$W to $y$y, in the sense that $||y-\hat{y}||&lt;||y-v||$∣∣y−y^​∣∣<∣∣y−v∣∣
for all $v$v in $W$W distinct from $\hat{y}$y^​.
If $\{u_1,\cdots,u_p\}${u1​,⋯,up​} is an orthonormal basis for a subspace $W$W of $R^n$Rn, then$proj_{w}y=(y\cdot u_1)u_1+(y\cdot u_2)u_2+\cdots+(y\cdot u_p)u_p$projw​y=(y⋅u1​)u1​+(y⋅u2​)u2​+⋯+(y⋅up​)up​
If $U=[u_1 u_2 \cdots u_p],then$U=[u1​u2​⋯up​],then $proj_{w}y=UU^Ty$projw​y=UUTy for all $y$y in $R^n$Rn

The GRAM-SCHMIDT PROCESS

Given a basis $x_1,\cdots,x_p$x1​,⋯,xp​ for a nonzero subspace $W$W of $R^n$Rn, define$v_1=x_1$v1​=x1​ $x_2 = x_2-\frac{x_2\cdot v_1}{v_1\cdot v_1}v_1$x2​=x2​−v1​⋅v1​x2​⋅v1​​v1​ $v_3 = x_3 - \frac{x_3\cdot v_1}{v_1 \cdot v_1}v_1 - \frac{x_3 \cdot v_2}{v_2\cdot v_2} v_2$v3​=x3​−v1​⋅v1​x3​⋅v1​​v1​−v2​⋅v2​x3​⋅v2​​v2​ $\vdots$⋮ $v_p = x_p - \frac{x_p\cdot v_1}{v_1 \cdot v_1}v_1 \cdots -\frac{x_p\cdot v_{p-1}}{v_{p-1}\cdot v_{p-1}}v_{p-1}$vp​=xp​−v1​⋅v1​xp​⋅v1​​v1​⋯−vp−1​⋅vp−1​xp​⋅vp−1​​vp−1​
Then ${v_1,\cdots,v_p}$v1​,⋯,vp​ is an orthogonal basis for $W$W. In addition$Span\{v_1,\cdots,v_k\}=Span\{x_1,\cdots, x_k\} for 1 \le k \le p$Span{v1​,⋯,vk​}=Span{x1​,⋯,xk​}for1≤k≤p

QR Factorization of Matrices

If A is an $m \times n$m×n matrix with linearly independent columns, then $A$A cam ne factored as $A = QR$A=QR, where $Q$Q is an $m \times n$m×n matrix whose columns form an orthonormal basis for $ColA$ColA and $R$R is an $n \times n$n×n upper triangular invertible matrix with positive entries on its diagonal.

Least-Squares Problems

If $A$A is $m \times n$m×n and $b$b is in $R^m$Rm, a least-squares solution of $Ax=b$Ax=b ia an $\hat{x}$x^ in $R^n$Rn such that:$||b-A\hat{x}||\le ||b-Ax||$∣∣b−Ax^∣∣≤∣∣b−Ax∣∣
for all $x$x in $R^n$Rn.
Note that
$A\hat{x}$Ax^ is in the $Col A$ColA
Let $A$A be an $m \times n$m×n matrix. THe following statements are logically equivalent:
a. The equation $Ax=b$Ax=b has a unique least-squares solution for each $b$b in $R^m$Rm.
b. The columns of $A$A are linearly independent.
c. The matrix $A^{T}A$ATA is invertible.
When these statments are true, the least-squares solution $\hat{x}$x^ is given by $\hat{x}=(A^{T}A)^{-1}A^{T}b$x^=(ATA)−1ATb
The set of least-squares solutions of $Ax=b$Ax=b coincides with the nonempty set of solutions of the normal equations $A^{T}A=x =A^{T}b$ATA=x=ATb.
Alternative Calculations of Least-Squares Solutions
given an $m \times n$m×n matrix A with linearly independent columns, let $A=QR$A=QR be a QR factorization of $A$A. Then for each b in $R^m$Rm, the equation $Ax=b$Ax=b has a unique least-squares solution.given by $\hat{x}=R^{-1}Q^Tb$x^=R−1QTb

Least-Squares Lines

For a system: predicted y-value: $\beta_0 +\beta_1 x_1$β0​+β1​x1​ Observed y-value:$y_1$y1​
We can write this system as $X \beta=$Xβ=, where $X=$X=, $\beta=[\beta_0 , \beta_1]$β=[β0​,β1​]

Inner Product spaces

An inner product on a vector space $V$V is a function that, to each pair of vectors $u$u and $v$v in $V$V, associates a real number <u,v> and satisfies the following axioms, for all $u,v$u,v, and $w$w in $V$V and all scalars $c$c:
1.$&lt;u,v&gt; = &lt;v,u&gt;$<u,v>=<v,u>
2.$&lt;u+v,w&gt;=&lt;u,w&gt;+&lt;v,w&gt;$<u+v,w>=<u,w>+<v,w>
3.$&lt;cu,v&gt;=c&lt;u,v&gt;$<cu,v>=c<u,v>
4.$&lt;u,u&gt; \ge 0$<u,u>≥0 and $&lt;u,u&gt;=0$<u,u>=0 if and only if $u=0$u=0
A vector space with an inner product is called an inner product space.

Trend Analysis of Data

$\hat{g}=c_0p_0+c_1p_1+c_2p_2+c_3p_3$g^​=c0​p0​+c1​p1​+c2​p2​+c3​p3​ and $\hat{g}$g^​ is called cubic trend function, and $c_0,\cdots,c_3$c0​,⋯,c3​ are the trend coefficients of the data. Thus could use Gram-schmidt process to construct the coefficients $c_0,\cdots,c_3$c0​,⋯,c3​

Fourier Series

Any function could be approsimated as closely as desired by a function of the form$F=a_0/2+a_1cost+\cdots+a_ncosnt+b_1sint+\cdots+b_nsinnt$F=a0​/2+a1​cost+⋯+an​cosnt+b1​sint+⋯+bn​sinnt
the set $\{1,cost,cos2t,\cdots,cosnt,sint,sin2t,\cdots,sinnt\}${1,cost,cos2t,⋯,cosnt,sint,sin2t,⋯,sinnt} si orthogonal with respect to the inner product$&lt;f,g&gt;=\int_0^{2\pi}f(t)g(t)dt$<f,g>=∫02π​f(t)g(t)dt Thus $a_k=\frac{&lt;f,coskt&gt;}{&lt;coskt,coskt&gt;}, b_k=\frac{&lt;f,sinkt&gt;}{&lt;sinkt,sinkt&gt;},k\ge 1.$ak​=<coskt,coskt><f,coskt>​,bk​=<sinkt,sinkt><f,sinkt>​,k≥1.

标签：v1,gt,Linear,Orthogonal,Introduction,cdots,cdot,lt,nm
来源： https://blog.csdn.net/zbzhzhy/article/details/88051059