首页 > 编程语言> > 反向传播算法推导过程（非常详细）

反向传播算法推导过程（非常详细）

2021-01-24 16:36:15 作者：互联网

原地址：https://zhuanlan.zhihu.com/p/79657669

1. 前向传播

假设 $X$ 为 $N\times m$ 的矩阵（其中， $N$ 为样本个数（batch size）， $m$ 为特征维数）:

$h_1$ 与 $Z_1$ 的维数为 $m_1 \rightarrow W_1$ 为 $m\times m_1$ 的矩阵， $b_1 \in \mathbb{R}^{m_1},$

$h_2$ 与 $Z_2$ 的维数为 $m_2 \rightarrow W_2$ 为 $m_1\times m_2$ 的矩阵， $b_2 \in \mathbb{R}^{m_2},$

${\vdots}$

$h_L$ 与 $Z_L$ 的维数为 $m_L \rightarrow W_L$ 为 $m_{L-1}\times m_L$ 的矩阵， $b_L \in \mathbb{R}^{m_L}$

前向算法：

$\begin{array}{l}{h_{1}=x W_{1}+\tilde{b}_{1}, Z_{1}=f_{1}\left(h_{1}\right), \tilde{b}_{1}}为b_1^T沿着行方向扩展成N行 \\ {h_{2}=Z_{1} W_{2}+\tilde{b}_{2}, Z_{2}=f_{2}\left(h_{2}\right)} \\ {\vdots} \\ {h_{L}=Z_{L-1} W_{L}+\tilde{b}_{L}, Z_{L}=f_{L}\left(h_{L}\right)} \\ {\text { out }=Z_{L} W_{L+1}+\tilde{b}_{L+1}}\end{array}$

假设输出为 $n$ 维，则 $out$ 为大小为 $N\times n$ 的矩阵，根据MSE或CE准则可以求得 $\frac{\partial J}{\partial out}$ ，对于回归问题与分类问题， $\frac{\partial J}{\partial out}$ 的求解方法如下：

对于回归问题，对out直接计算损失，损失函数为MSE。损失： $J=\frac{1}{2N}\sum_{i=1}^{N}||y_i-\tilde{y_i}||^2$

$\begin{aligned} \frac{\partial J}{\partial y_i}&=\frac{1}{2N}\sum_{i=1}^{N}(y_i-\tilde{y_i})\times 2 \\ &=\frac{1}{N}\sum_{i=1}^{N}(y_i-\tilde{y_i}) \end{aligned}$

对于分类问题，out后接softmax进行分类，然后使用CE(cross entropy)计算loss. $S_k=\frac{e^{y_k}}{\sum_{i=1}^{n}e^{y_i}}$ 一个样本对应的网络的输出 $S(s_1,s_2,...,s_n)$ 是一个概率分布，而这个样本的标注 $\tilde{S}$ 一般为 $(0,0,...,1,0,0,...,0)$ ，也可以看做一个概率分布（硬分布）。cross entropy可以看成是 $S$ 与 $\tilde{S}$ 之间的KL距离：

$D(\tilde{S}||S)=\Sigma\tilde{S}\log\frac{\tilde{S}}{S}$

假设 $\tilde{S}=(0,0,...,1,0,0,...,0)$ ，其中1为第 $k$ 个元素(索引从0开始)，令 $S=(s_0,s_1,...,s_k,...,s_{n-1})$ .

损失： $\begin{aligned} J=D(\tilde{S}||S)&=1\times \log\frac{1}{s_k}\\&=-\log s_k \quad(CE损失函数,可看做目标类别概率最大)\\ &=-\log\frac{e^{y_k}}{\sum_{i=0}^{n-1}e^{y_i}} \end{aligned}$ $\begin{aligned} &\frac{\partial J}{\partial y_m}=\frac{\partial J}{\partial y_m}(\log \sum_{i=0}^{n-1}e^{y_i}-y_k)=\frac{e^{y_m}}{\sum_{i=0}^{n-1}e^{y_i}}-\delta(m=k)=s_m-\delta(m=k) \\ &写成向量形式为:\frac{\partial J}{\partial y}=S-\tilde{S} \end{aligned}$

KL距离（相对熵）：是Kullback-Leibler Divergence的简称，也叫相对熵(Relative Entropy).它衡量的是相同事件空间里的两个概率分布的差异情况。其物理意义是：在相同事件空间里，概率分布 P(x) 对应的每个事件，若用概率分布 Q(x) 编码时，平均每个基本事件(符号)编码长度增加了多少比特。我们用 $D(P||Q)$ 表示KL距离，计算公式如下： $D(P||Q)=\sum_{x\in X}P(x)\log\frac{P(x)}{Q(x)}$ ，当两个概率分布完全相同时，即 P(X)=Q(X) ,其相对熵为0.

2.反向传播

$\text { out }=Z_{L} W_{L+1}+\tilde{b}_{L+1}$ ，为了便于详细说明反向传播算法，假设 $Z_L$ 为 $2\times 3$ 的向量， $W_{L+1}$ 为 $3\times 2$ 的向量： $\begin{array}{l}{Z_{L}=\left(\begin{array}{ccc}{z_{11}} & {z_{12}} & {z_{13}} \\ {z_{21}} & {z_{22}} & {z_{23}}\end{array}\right)_{2 \times 3}, W_{L+1}=\left(\begin{array}{cc}{w_{11}} & {w_{12}} \\ {w_{21}} & {w_{22}} \\ {w_{31}} & {w_{32}}\end{array}\right)_{3 \times 2} \tilde{b}_{L+1}=\left(\begin{array}{cc}{b_{1}} & {b_{2}} \\ {b_{1}} & {b_{2}}\end{array}\right)_{2 \times 2}, \text { out }=\left(\begin{array}{cc}{o_{11}} & {o_{12}} \\ {o_{21}} & {o_{22}}\end{array}\right)} \\ \Rightarrow {Z_{L}W_{L+1}+\tilde{b}_{L+1}=\left(\begin{array}{cc}{z_{11} w_{11}+z_{12} w_{21}+z_{13} w_{31}+b_1} & {z_{11} w_{12}+z_{12} w_{22}+z_{13} w_{32}+b_2} \\ {z_{21} w_{11}+z_{22} w_{21}+z_{23} w_{31}+b_1} & {z_{21} w_{12}+z_{22} w_{22}+z_{23} w_{32}+b_2}\end{array}\right)=\text{out}.}\end{array}$

所以， $\begin{array}{l}{o_{11}=z_{11} w_{11}+z_{12} w_{21}+z_{13} w_{31}+b_{1}} \\ {o_{12}=z_{11} w_{12}+z_{12} w_{22}+z_{13} w_{32}+b_{2}} \\ {o_{21}=z_{21} w_{11}+z_{22} w_{21}+z_{23} w_{31}+b_{1}} \\ {o_{22}=z_{21} w_{12}+z_{22} w_{22}+z_{23} w_{32}+b_{2}}\end{array}$

1) 损失 $J$ 对 $W$ 的导数：

$\begin{aligned} \frac{\partial J}{\partial w_{11}} &=\frac{\partial J}{\partial o_{11}} z_{11}+\frac{\partial J}{\partial o_{21}} z_{21}, \frac{\partial J}{\partial w_{12}}=\frac{\partial J}{\partial o_{12}} z_{11}+\frac{\partial J}{\partial o_{22}} z_{21} \\ \frac{\partial J}{\partial w_{21}} &=\frac{\partial J}{\partial o_{11}} z_{12}+\frac{\partial J}{\partial o_{21}} z_{22}, \frac{\partial J}{\partial w_{22}}=\frac{\partial J}{\partial o_{12}} z_{12}+\frac{\partial J}{\partial o_{22}} z_{22} \\ \frac{\partial J}{\partial w_{31}} &=\frac{\partial J}{\partial o_{11}} z_{13}+\frac{\partial J}{\partial o_{21}} z_{23}, \frac{\partial J}{\partial w_{32}}=\frac{\partial J}{\partial o_{12}} z_{13}+\frac{\partial J}{\partial o_{22}} z_{23} \end{aligned}$

$\Rightarrow \left(\begin{array}{cc}{\frac{\partial J}{\partial w_{11}}} & {\frac{\partial J}{\partial w_{12}}} \\ {\frac{\partial J}{\partial w_{21}}} & {\frac{\partial J}{\partial w_{22}}} \\ {\frac{\partial J}{\partial w_{31}}} & {\frac{\partial J}{\partial w_{32}}}\end{array}\right)=\left(\begin{array}{cc}{z_{11}} & {z_{21}} \\ {z_{12}} & {z_{22}} \\ {z_{13}} & {z_{23}}\end{array}\right)\left(\begin{array}{cc}{\frac{\partial J}{\partial o_{11}}} & {\frac{\partial J}{\partial o_{12}}} \\ {\frac{\partial J}{\partial o_{21}}} & {\frac{\partial J}{\partial o_{22}}}\end{array}\right)$

即， $\frac{\partial J}{\partial W_{L+1}}=Z_L^T\frac{\partial J}{\partial out}$

2) 损失对偏置b的导数等于将 $\frac{\partial J}{\partial out}$ 的每一列加起来：

$\left\{\begin{array}{l}{\frac{\partial J}{\partial b_{1}}=\frac{\partial J}{\partial o_{11}}+\frac{\partial J}{\partial o_{21}}} \\ {\frac{\partial J}{\partial b_{2}}=\frac{\partial J}{\partial o_{12}}+\frac{\partial J}{\partial o_{22}}}\end{array} \Rightarrow\left(\frac{\partial J}{\partial b_{L+1}}\right)^{T}=\left(\frac{\partial J}{\partial b_{1}} \quad \frac{\partial J}{\partial b_{2}}\right)=\left(\frac{\partial J}{\partial o_{11}}+\frac{\partial J}{\partial o_{21}} \quad \frac{\partial J}{\partial o_{12}}+\frac{\partial J}{\partial o_{22}}\right)\right.$

3) 损失 $J$ 对 $Z$ 的导数：

$\begin{aligned} \frac{\partial J}{\partial z_{11}} &=\frac{\partial J}{\partial o_{11}} w_{11}+\frac{\partial J}{\partial o_{12}} w_{12} ; \frac{\partial J}{\partial z_{12}}=\frac{\partial J}{\partial o_{11}} w_{21}+\frac{\partial J}{\partial o_{12}} w_{22} ; \frac{\partial J}{\partial z_{13}}=\frac{\partial J}{\partial o_{11}} w_{31}+\frac{\partial J}{\partial o_{12}} w_{32} \\ \frac{\partial J}{\partial z_{21}} &=\frac{\partial J}{\partial o_{21}} w_{11}+\frac{\partial J}{\partial o_{22}} w_{12} ; \frac{\partial J}{\partial z_{22}}=\frac{\partial J}{\partial o_{21}} w_{21}+\frac{\partial J}{\partial o_{12}} w_{22} ; \frac{\partial J}{\partial z_{23}}=\frac{\partial J}{\partial o_{21}} w_{31}+\frac{\partial J}{\partial o_{22}} w_{32} \end{aligned}$

即，

$\left(\begin{array}{ccc}{\frac{\partial J}{\partial z_{11}}} & {\frac{\partial J}{\partial z_{12}}} & {\frac{\partial J}{\partial z_{13}}} \\ {\frac{\partial J}{\partial z_{21}}} & {\frac{\partial J}{\partial z_{22}}} & {\frac{\partial J}{\partial z_{23}}}\end{array}\right)=\left(\begin{array}{cc}{\frac{\partial J}{\partial o_{11}}} & {\frac{\partial J}{\partial o_{12}}} \\ {\frac{\partial J}{\partial \theta_{21}}} & {\frac{\partial J}{\partial o_{22}}}\end{array}\right)\left(\begin{array}{ccc}{w_{11}} & {w_{21}} & {w_{31}} \\ {w_{12}} & {w_{22}} & {w_{32}}\end{array}\right)$

$\Rightarrow \frac{\partial J}{\partial Z_{L}}=\frac{\partial J}{\partial out}W_{L+1}^T$

4) 损失 $J$ 对 $h$ 的导数：

$Z_L = f_L(h_L)$

$f_L$ 为sigmoid时， $Z_L=\frac{1}{1+e^{-h_L}} .$

$\begin{array}{l}{\frac{\partial J}{\partial h_{L}}=\frac{\partial J}{\partial Z_{L}} \frac{d z_{L}}{d h_{L}}=\frac{\partial J}{\partial Z_{L}} \frac{e^{-h L}}{\left(1+e^{-h_{L}}\right)^{2}}=\frac{\partial J}{\partial Z_{L}} \frac{1}{1+e^{-h_{L}}} \frac{e^{-h_{L}}}{1+e^{-h_{L}}}} \\ {=\frac{\partial J}{\partial Z_{L}} Z_{L}\left(1-Z_{L}\right)}\end{array}$

$f_L$ 为tanh时， ${Z_{L}=\frac{e^{h_{L}}-e^{-h_{L}}}{e^{h_{L}}+e^{-h_{L}}}}$ .

$\begin{array}{l} {\frac{\partial J}{\partial h_{L}}=\frac{\partial J}{\partial Z_{L}} \frac{d Z_{L}}{d h_{L}}=\frac{\partial J}{\partial Z_{L}} \frac{4}{\left(e^{h_{L}}+e^{-h_{L}}\right)^{2}}=\frac{\partial J}{\partial Z_{L}}\left[1-\left(\frac{e^{h_{L}}-e^{-h_{L}}}{e^{h_{L}}+e^{-h_{L}}}\right)^{2}\right]} \\ {=\frac{\partial J}{\partial z_{L}}\left[1-z_{L}^{2}\right]}\end{array}$

$f_L$ 为relu时， $Z_L=relu(h_L)=\left\{\begin{matrix} 0,&h_L\leq 0 \\ h_L,&h_L > 0 \end{matrix}\right. .$

$\begin{array}{l} \frac{\partial J}{\partial h_L}=\frac{\partial J}{\partial Z_L}\frac{\partial Z_L}{\partial h_L}=\left\{\begin{matrix} 0,&h_L\leq 0 \\ \frac{\partial J}{\partial Z_L},&h_L > 0 \end{matrix}\right. \end{array}$

3. 梯度更新

对于不同算法，梯度更新方式如下： $\frac{\partial J}{\partial out} \Rightarrow \left \{\begin{matrix} \frac{\partial J}{\partial W_{L+1}}=Z_L^T\frac{\partial J}{\partial out} \\ \frac{\partial J}{\partial Z_{L}}=\frac{\partial J}{\partial out}W_{L+1}^T \\ \left(\frac{\partial J}{\partial b}\right)^{T}=SumCol(\frac{\partial J}{\partial out}) \\ W_{L+1}^{t+1} = W_{L+1}^t-\eta \frac{\partial J}{\partial W_{L+1}} \\ b_{L+1}^{t+1} = b_{L+1}^t-\eta \frac{\partial J}{\partial b_{L+1}} \end{matrix} \right. \Rightarrow \frac{\partial J}{\partial h_L}=\frac{\partial J}{\partial Z_L}\frac{\partial Z_L}{\partial h_L} \Rightarrow \left \{\begin{matrix} \frac{\partial J}{\partial W_{L}}=Z_{L-1}^T\frac{\partial J}{\partial h_L} \\ \frac{\partial J}{\partial Z_{L-1}}=\frac{\partial J}{\partial h_L}W_{L}^T \\ \vdots \\ \vdots \end{matrix}\right. \Rightarrow \cdots$

标签：L%,7D%,推导,概率分布,5Cpartial,+%,算法,7B%,反向
来源： https://www.cnblogs.com/aabbcc/p/14321276.html