线性回归
作者:互联网
模型假设
\[ h(x) = \omega ^T x + b \]
损失函数
针对一个样例 \((x_i, y_i)\) 而言,损失为 \(\frac{1}{2}(\omega ^T x_i + b - y_i)^2\),记之为 \(J_i(\omega , b)\),下面对 \(\omega, \ b\) 求导:
\(dJ_i = d(\frac{1}{2}(\omega ^T x_i + b - y_i)^2)\)
\(\ \ \ \ \ = (\omega ^T x_i + b - y_i) \odot d(\omega ^T x_i + b - y_i)\)
由于损失值对 \(b\) 的导数是标量对标量,故而直接求得对 \(b\) 的偏导:
\[
\frac{\partial J_i}{\partial b} = \omega ^T x_i + b - y_i
\]
由于对\(\omega\) 的导数是标量对矩阵求导,故而:
\(dJ_i = (\omega ^T x_i + b - y_i) \odot d(\omega ^T x_i)\)
\(\ \ \ \ \ = (\omega ^T x_i + b - y_i) \odot (d\omega ^T x_i + \omega ^T d x_i )\)
\(\ \ \ \ \ = (\omega ^T x_i + b - y_i) \odot d\omega ^T x_i\)
\(\ \ \ \ \ = tr((\omega ^T x_i + b - y_i) \odot x_i ^T d\omega )\)
最后可得对 $\omega $ 的偏导数:
\[
\frac{\partial J_i}{\partial \omega } = (\omega ^T x_i + b - y_i) \cdot x_i
\]
再考虑所有样本,由于 \(J = \frac{1}{n} \sum_{i=1}^n J_i\),故而可以求得整体损失对 \(\omega , \ b\) 的导数:
\[
\frac{\partial J}{\partial \omega } = \frac{1}{n} \sum_{i=1}^n (\omega ^T x_i + b - y_i) \cdot x_i \\
\frac{\partial J}{\partial b } = \frac{1}{n} \sum_{i=1}^n (\omega ^T x_i + b - y_i)
\]
矩阵计算
分别对 \(\omega , b\) 求导
给出各个变量的维度:\(X \in R^{n \times d}, y \in R^{n \times 1}, \omega \in R^{d \times 1}, b \in R, \textbf{1} \in R^{n \times 1}\)。于是,\(J\) 对 \(\omega , b\) 的导数可以表示如下:
\[
\frac{\partial J}{\partial \omega } = \frac{1}{n} X^T (X \omega + b - y) \\
\frac{\partial J}{\partial b } = \frac{1}{n} \textbf{1}^T (X \omega + b - y)
\]
合并 \(\omega , b\)
记 \(\hat{X} = [x_i ^T, 1], \ \hat{\omega } = [\omega ; b]\),则损失对参数的导数是:
\[
\frac{\partial J}{\partial \hat{\omega} } =\frac{1}{n} \hat{X}^T (\hat{X}\hat{\omega } - y)
\]
令其为零可得闭式解:
\[
\hat{\omega }= (\hat{X}^T \hat{X})^{-1} \hat{X}^T y
\]
标签:frac,导数,hat,回归,odot,线性,partial,omega 来源: https://www.cnblogs.com/luyunan/p/12269240.html