本博文具有大量公式，由于简书不支持公式，公式渲染完全版请移步博客
DNN中的反向传播
反向传播算法是神经网络的训练的基本算法组成之一，在训练神经网络时，训练分为两个步骤：计算梯度和更新权值。其中反向传播负责的是梯度的计算，而训练算法的区分主要在更新权值的方式上。对于DNN，基本的反向传播思路为：
$$\cfrac{dz}{dw_{i}} = \cfrac{dz}{da_{i+1}} \times \cfrac{da_{i+1}}{dw_{i}}$$
其中，$\cfrac{dz}{dw_{i}}$为输出（多为代价函数输出）对第i层的权值的梯度，$\cfrac{da_{i+1}}{dw_{i}}$为本层输出对权值的梯度。于是梯度的计算被分为反向传播链条上的几个部分，将复杂的求导分割为层内运算的求导，上一层的梯度可以由本层的梯度递归的求出。
卷积神经网络中的反向传播
卷积神经网络相比于多层感知机，增加了两种新的层次——卷积层与池化层。由于反向传播链的存在，要求出这两种层结构的梯度，仅需要解决输出对权值的梯度即可。
池化层的梯度
池化层用于削减数据量，在这一层上前向传播的数据会有损失，则在反向传播时，传播来的梯度也会有所损失。一般来说，池化层没有参数，于是仅需要计算梯度反向传播的结果。
理论分析
池化层的反向传播的方法是upsample，先将矩阵还原成原大小，之后：
- 对于最大值池化，将梯度放置于每个池化区域取得最大值的位置，其他位置为0
- 对于平均值池化，则把的所有子矩阵的各个池化局域的值取平均后放在还原后的子矩阵位置
例如对于矩阵：
$$ \left( \begin{array}{} 1& 2 \ 3& 4 \end{array} \right)$$
假设经过2*2的池化，还原为原来大小：
$$\left( \begin{array}{ccc} 0&0&0&0 \ 0&1& 2&0 \ 0&3&4&0 \ 0&0&0&0 \end{array} \right)$$
若是最大值池化，假设每个窗口的最大值位置都是左上，则传播结果为：
$$\left( \begin{array}{ccc} 1&0&2&0 \ 0&0& 0&0 \ 3&0&4&0 \ 0&0&0&0 \end{array} \right)$$
若是经过平均值池化，则传播结果为：
$$\left( \begin{array}{ccc} 0.25&0.25&0.5&0.5 \ 0.25&0.25& 0.5&0.5 \ 0.75&0.75&1&1 \ 0.75&0.75&1&1 \end{array} \right)$$
代码验证
最大值池化
test_model = pt.nn.MaxPool2d(2)
test = pt.autograd.Variable(pt.arange(0, 16).view(1, 1, 4, 4), requires_grad=True)
print(test)
result = test_model(test)
print(result)
result.backward(result.data)
print(test.grad)
Variable containing:
(0 ,0 ,.,.) =
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
[torch.FloatTensor of size 1x1x4x4]
Variable containing:
(0 ,0 ,.,.) =
5 7
13 15
[torch.FloatTensor of size 1x1x2x2]
Variable containing:
(0 ,0 ,.,.) =
0 0 0 0
0 5 0 7
0 0 0 0
0 13 0 15
[torch.FloatTensor of size 1x1x4x4]
平均值池化
test_model = pt.nn.AvgPool2d(2)
test = pt.autograd.Variable(pt.arange(0, 16).view(1, 1, 4, 4), requires_grad=True)
print(test)
result = test_model(test)
print(result)
result.backward(result.data)
print(test.grad)
Variable containing:
(0 ,0 ,.,.) =
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
[torch.FloatTensor of size 1x1x4x4]
Variable containing:
(0 ,0 ,.,.) =
2.5000 4.5000
10.5000 12.5000
[torch.FloatTensor of size 1x1x2x2]
Variable containing:
(0 ,0 ,.,.) =
0.6250 0.6250 1.1250 1.1250
0.6250 0.6250 1.1250 1.1250
2.6250 2.6250 3.1250 3.1250
2.6250 2.6250 3.1250 3.1250
[torch.FloatTensor of size 1x1x4x4]
卷积层梯度
卷积层具有权值，因此梯度计算包括反向传播的梯度和权值梯度
反向传播梯度
理论分析
对于卷积网络，前向传播公式为：
$$a^l= \sigma(z^l) = \sigma(a{l-1}*Wl +b^l) $$
其中$*$为卷积运算（不为乘法运算），DNN的反向传播公式为：
$$\delta^{l} = \cfrac{\partial J(W,b)}{\partial z^l} = \cfrac{\partial J(W,b)}{\partial z^{l+1}}\cfrac{\partial z^{l+1}}{\partial z^{l}} = \delta^{l+1}\cfrac{\partial z^{l+1}}{\partial z^{l}}$$
其中$\delta^{l}$为第l层的梯度，$\cfrac{\partial z^{l+1}}{\partial z^{l}}$为卷积层的输出对输入的梯度，则反向传播的梯度为：
$$\delta^{l-1} = \delta^{l}\cfrac{\partial z^{l}}{\partial z^{l-1}} = \delta{l}*rot180(W{l}) \odot \sigma{'}(z{l-1})$$
其中$rot180(W^{l})$为卷积核旋转180度的函数，*为卷积。
举一个例子来说，对于以下卷积等式：
$$\left( \begin{array}{ccc} a_{11}&a_{12}&a_{13} \ a_{21}&a_{22}&a_{23}\ a_{31}&a_{32}&a_{33} \end{array} \right) * \left( \begin{array}{ccc} w_{11}&w_{12}\ w_{21}&w_{22} \end{array} \right) = \left( \begin{array}{ccc} z_{11}&z_{12}\ z_{21}&z_{22} \end{array} \right)$$
对于$a_{11}$，有$z_{11} = a_{11}w_{11} + a_{12}w_{12} + a_{21}w_{21} + a_{22}w_{22}$，仅$z_{11}$与其有关，则有$\nabla a_{11} = \delta_{11}w_{11}$。
对于$a_{22}$，所有z项都和该数有关，有$\nabla a_{22} = \delta_{11}w_{22} + \delta_{12}w_{21} + \delta_{21}w_{12} + \delta_{22}w_{11}$
依次类推，可得：
$\left( \begin{array}{ccc} 0&0&0&0 \ 0&\delta_{11}& \delta_{12}&0 \ 0&\delta_{21}&\delta_{22}&0 \ 0&0&0&0 \end{array} \right) * \left( \begin{array}{ccc} w_{22}&w_{21}\ w_{12}&w_{11} \end{array} \right) = \left( \begin{array}{ccc} \nabla a_{11}&\nabla a_{12}&\nabla a_{13} \ \nabla a_{21}&\nabla a_{22}&\nabla a_{23}\ \nabla a_{31}&\nabla a_{32}&\nabla a_{33} \end{array} \right)$
代码验证
data = pt.autograd.Variable(pt.arange(0, 9).view(1, 1, 3, 3), requires_grad=True)
weight = pt.autograd.Variable(pt.arange(0,4).view(1,1,2,2),requires_grad=True)
bias = pt.autograd.Variable(pt.zeros(1),requires_grad=True)
result = pt.nn.functional.conv2d(data,weight,bias)
print(result)
result.backward(result.data)
print(data.grad)
Variable containing:
(0 ,0 ,.,.) =
19 25
37 43
[torch.FloatTensor of size 1x1x2x2]
Variable containing:
(0 ,0 ,.,.) =
0 19 25
38 144 118
74 197 129
[torch.FloatTensor of size 1x1x3x3]
该代码中，前向传播为：
$$\left( \begin{array}{ccc} 0&1&2 \ 3&4&5\ 6&7&8 \end{array} \right) * \left( \begin{array}{ccc} 0&1\ 2&3 \end{array} \right) = \left( \begin{array}{ccc} 19&25\ 37&43 \end{array} \right)$$
反向传播为：
$$\left( \begin{array}{ccc} 0&0&0&0 \ 0&19&25&0 \ 0&37&43&0 \ 0&0&0&0 \end{array} \right) * \left( \begin{array}{ccc} 3&2\ 1&0 \end{array} \right) = \left( \begin{array}{ccc} 0&19&25 \ 38&144&118\ 74&197&129 \end{array} \right)$$
权值梯度
理论推导
类似于上一节的公式，对权值的梯度为：
$$\cfrac{\partial J(W,b)}{\partial W^{l}} = \cfrac{\partial J(W,b)}{\partial z^{l}}\cfrac{\partial z^{l}}{\partial W^{l}} =rot180(\deltal*a{l-1})$$
同样对于前向传播：
$$\left( \begin{array}{ccc} a_{11}&a_{12}&a_{13} \ a_{21}&a_{22}&a_{23}\ a_{31}&a_{32}&a_{33} \end{array} \right) * \left( \begin{array}{ccc} w_{11}&w_{12}\ w_{21}&w_{22} \end{array} \right) = \left( \begin{array}{ccc} z_{11}&z_{12}\ z_{21}&z_{22} \end{array} \right)$$
对于$z$，有以下：
- $z_{11} = a_{11}w_{11} + a_{12}w_{12} + a_{21}w_{21} + a_{22}w_{22}$
- $z_{12} = a_{12}w_{11} + a_{13}w_{12} + a_{22}w_{21} + a_{23}w_{22}$
- $z_{21} = a_{21}w_{11} + a_{22}w_{12} + a_{31}w_{21} + a_{32}w_{22}$
- $z_{22} = a_{22}w_{11} + a_{23}w_{12} + a_{32}w_{21} + a_{33}w_{22}$
有梯度：
- $\nabla w_{11} = \delta_{11}a_{11} + \delta_{12}a_{12} + \delta_{21}a_{21} + \delta_{22}a_{22}$
- $\nabla w_{12} = \delta_{11}a_{12} + \delta_{12}a_{13} + \delta_{21}a_{22} + \delta_{22}a_{12}$
- $\nabla w_{21} = \delta_{11}a_{21} + \delta_{12}a_{22} + \delta_{21}a_{31} + \delta_{22}a_{32}$
- $\nabla w_{22} = \delta_{11}a_{22} + \delta_{12}a_{23} + \delta_{32}a_{21} + \delta_{33}a_{22}$
反向传播为：
$\left( \begin{array}{ccc} 0&0&0&0 \ 0&\delta_{11}& \delta_{12}&0 \ 0&\delta_{21}&\delta_{22}&0 \ 0&0&0&0 \end{array} \right) * \left( \begin{array}{ccc} a_{11}&a_{12}&a_{13}\ a_{21}&a_{22}&a_{23}\ a_{31}&a_{32}&a_{33} \end{array} \right) = \left( \begin{array}{ccc} \nabla w_{22}&\nabla w_{21}\ \nabla w_{12}&\nabla w_{11} \end{array} \right)$
代码验证
data = pt.autograd.Variable(pt.arange(0, 9).view(1, 1, 3, 3), requires_grad=True)
weight = pt.autograd.Variable(pt.arange(0,4).view(1,1,2,2),requires_grad=True)
bias = pt.autograd.Variable(pt.zeros(1),requires_grad=True)
result = pt.nn.functional.conv2d(data,weight,bias)
print(result)
result.backward(result.data)
print(weight.grad)
Variable containing:
(0 ,0 ,.,.) =
19 25
37 43
[torch.FloatTensor of size 1x1x2x2]
Variable containing:
(0 ,0 ,.,.) =
308 432
680 804
[torch.FloatTensor of size 1x1x2x2]
反向传播为：
$$\left( \begin{array}{ccc} 0&0&0&0 \ 0&19&25&0 \ 0&37&43&0 \ 0&0&0&0 \end{array} \right) * \left( \begin{array}{ccc} 0&1&2 \ 3&4&5\ 6&7&8 \end{array} \right) = \left( \begin{array}{ccc} 804&680\ 432&308 \end{array} \right)$$
偏置梯度
偏置梯度较为简单，为上一层梯度和：
$$\cfrac{\partial J(W,b)}{\partial b^{l}} = \cfrac{\partial J(W,b)}{\partial z^{l}} \cdot \cfrac{dz{l}}{dbl}= \sum\limits_{u,v}(\delta^l)_{u,v}$$
data = pt.autograd.Variable(pt.arange(0, 9).view(1, 1, 3, 3), requires_grad=True)
weight = pt.autograd.Variable(pt.arange(0,4).view(1,1,2,2),requires_grad=True)
bias = pt.autograd.Variable(pt.zeros(1),requires_grad=True)
result = pt.nn.functional.conv2d(data,weight,bias)
print(result)
result.backward(result.data)
print(bias.grad)
Variable containing:
(0 ,0 ,.,.) =
19 25
37 43
[torch.FloatTensor of size 1x1x2x2]
Variable containing:
124
[torch.FloatTensor of size 1]
有$124 = 19 + 25 + 37 + 43$
参考
