Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [41] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1 , where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

Deep convolutional neural networks [22, 21] have led to a series of breakthroughs for image classification [21, 50, 40]. Deep networks naturally integrate low/mid/highlevel features [50] and classifiers in an end-to-end multilayer fashion, and the “levels” of features can be enriched by the number of stacked layers (depth). Recent evidence [41, 44] reveals that network depth is of crucial importance, and the leading results [41, 44, 13, 16] on the challenging ImageNet dataset [36] all exploit “very deep” [41] models, with a depth of sixteen [41] to thirty [16]. Many other nontrivial visual recognition tasks [8, 12, 7, 32, 27] have also greatly benefited from very deep models.

Driven by the significance of depth, a question arises: Is learning better networks as easy as stacking more layers? An obstacle to answering this question was the notorious problem of vanishing/exploding gradients [1, 9], which hamper convergence from the beginning. This problem, however, has been largely addressed by normalized initialization [23, 9, 37, 13] and intermediate normalization layers [16], which enable networks with tens of layers to start converging for stochastic gradient descent (SGD) with backpropagation [22].

When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example.

The degradation (of training accuracy) indicates that not all systems are similarly easy to optimize. Let us consider a shallower architecture and its deeper counterpart that adds more layers onto it. There exists a solution by construction to the deeper model: the added layers are identity mapping, and the other layers are copied from the learned shallower model. The existence of this constructed solution indicates that a deeper model should produce no higher training error than its shallower counterpart. But experiments show that our current solvers on hand are unable to find solutions that are comparably good or better than the constructed solution (or unable to do so in feasible time).

深度卷积神经网络语句引领了图像分类任务中的许多突破。深的网络可以自然地融合低/中/高层次的特征以及端到端多层的分类器，并且，特征的层次可以通过堆叠层的数量（深度）去丰富。最近有证据揭示，网络深度十分重要，并且，在困难的ImageNet数据集上领先的结果都利用了“非常深”的网络，如16层到30层。（这里给非常深打引号我觉得非常有意思，更显示出了作者的152层真是对于网络深度很大的突破。）许多其他的重要的视觉感知任务已经从非常深的模型中受益很多了。

收深度的重要性的驱动，一个问题产生了：学习更好的网络和堆叠更多的层一样简单吗？回答这个问题的阻碍是臭名昭著的梯度消失和梯度爆炸问题，这从一开始就阻碍了模型的收敛。但是这个问题已经很大程度上被标准化初始化和中间归一化层解决了，它们使数十层的网络能够使用SGD通过BP算法达到收敛。

当更深的网络能够收敛时，又出现了一个衰退问题：随着网络深度的增加，精度达到饱和后迅速下降。出乎意料的是，根据文献的报道以及我们充分的实验证明，这样的衰退不是由过拟合导致的，向适当深度的模型添加更多层会导致更高的训练错误。Figure 1中给出了一种典型的例子。

训练准确度的衰退表明不是所有系统都同样易于优化。 让我们考虑一下一个较浅的架构以及在它上面加更多层的更深对应部分。有一个解决上述问题的方案：添加层是恒等映射，其他层都复制浅层模型中的学习结果。这种解决方案的存在表明，更深的模型不应产生比其浅层对应模型更高的训练误差。但实验表明，我们现有的求解器无法找到与上述解决方案相当或更好的解决方案，或者说是无法在可行的时间内找到。

In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as H(x), we let the stacked nonlinear layers fit another mapping of F(x) := H(x)−x. The original mapping is recast into F(x)+x. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers. The formulation of F(x) +x can be realized by feedforward neural networks with “shortcut connections” (Fig. 2). Shortcut connections [2, 34, 49] are those skipping one or more layers. In our case, the shortcut connections simply perform identity mapping, and their outputs are added to the outputs of the stacked layers (Fig. 2). Identity shortcut connections add neither extra parameter nor computational complexity. The entire network can still be trained end-to-end by SGD with backpropagation, and can be easily implemented using common libraries (e.g., Caffe [19]) without modifying the solvers.

We present comprehensive experiments on ImageNet [36] to show the degradation problem and evaluate our method. We show that: 1) Our extremely deep residual nets are easy to optimize, but the counterpart “plain” nets (that simply stack layers) exhibit higher training error when the depth increases; 2) Our deep residual nets can easily enjoy accuracy gains from greatly increased depth, producing results substantially better than previous networks.

Similar phenomena are also shown on the CIFAR-10 set [20], suggesting that the optimization difficulties and the effects of our method are not just akin to a particular dataset. We present successfully trained models on this dataset with over 100 layers, and explore models with over 1000 layers.

On the ImageNet classification dataset [36], we obtain excellent results by extremely deep residual nets. Our 152- layer residual net is the deepest network ever presented on ImageNet, while still having lower complexity than VGG nets [41]. Our ensemble has 3.57% top-5 error on the ImageNet test set, and won the 1st place in the ILSVRC 2015 classification competition. The extremely deep representations also have excellent generalization performance on other recognition tasks, and lead us to further win the 1st places on: ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation in ILSVRC & COCO 2015 competitions. This strong evidence shows that the residual learning principle is generic, and we expect that it is applicable in other vision and non-vision problems.

本文中，我们通过引入一个深度残差学习框架解决了该退化问题。我们让堆叠层去你和一个残差映射，而不是让他直接去拟合需要的底层映射（就是最终我们需要的映射）。我们将需要的底层映射表示为H(x)，让堆叠的非线性层去拟合F(x):=H(x)-x的映射，原始的底层映射H(x)即被表示为F(x)+x。我们假定相较于优化原始底层映射，优化残差映射会更加简单。极端一点来说，如果一个恒等映射是最优的，对于堆叠的非线性层来说，相较于拟合一个恒等映射，将残差向零推进会更加的简单。F(x)+x这个公式可以被带有shortcut connection（残差/跳跃/捷径链接）的前驱神经网络实现。shortcut connection，也就是跳跃了一层或多层的连接。在我们的案例中，残差连接只是使用了一个恒等映射，残差连接的输出被加入到堆叠层的输出当中。恒等的残差链接既不会增加参数的个数，也不会提高模型的算法复杂度（因为它只是一个加法而已）。整个网络可以被使用SGD的端到端的BP算法训练出来，并且可以使用通用库轻松实现，无需修改求解器。

我们在 ImageNet上进行了全面的实验，展示了衰退问题并评估了我们的方法。我们展示出：1）我们极深的残差网络很容易优化。当深度增加时，对应的“普通”网络（简单的无残差连接的堆叠层）表现出了更高的训练误差；2）我们的深度残差网络可以很容易地随着网络深度的大幅度增加，获得准确度的提升，产生的结果比以前的网络要好得多。

同样的现象在CIFAR-10的数据集上也体现了出来，这表明，深层网络优化的困难和我们方法的效果不是仅仅局限于一个特定的数据集。我们在这个数据集上展现出了超过100层的成功训练的模型，并探索了超过1000层的模型。

接下里的都是实验结果......表现出了残差网络模型的高准确性和很好的泛化能力。

Residual Representations. In image recognition, VLAD [18] is a representation that encodes by the residual vectors with respect to a dictionary, and Fisher Vector [30] can be formulated as a probabilistic version [18] of VLAD. Both of them are powerful shallow representations for image retrieval and classification [4, 48]. For vector quantization, encoding residual vectors [17] is shown to be more effective than encoding original vectors.

In low-level vision and computer graphics, for solving Partial Differential Equations (PDEs), the widely used Multigrid method [3] reformulates the system as subproblems at multiple scales, where each subproblem is responsible for the residual solution between a coarser and a finer scale. An alternative to Multigrid is hierarchical basis preconditioning [45, 46], which relies on variables that represent residual vectors between two scales. It has been shown [3, 45, 46] that these solvers converge much faster than standard solvers that are unaware of the residual nature of the solutions. These methods suggest that a good reformulation or preconditioning can simplify the optimization.

Shortcut Connections. Practices and theories that lead to shortcut connections [2, 34, 49] have been studied for a long time. An early practice of training multi-layer perceptrons (MLPs) is to add a linear layer connected from the network input to the output [34, 49]. In [44, 24], a few intermediate layers are directly connected to auxiliary classifiers for addressing vanishing/exploding gradients. The papers of [39, 38, 31, 47] propose methods for centering layer responses, gradients, and propagated errors, implemented by shortcut connections. In [44], an “inception” layer is composed of a shortcut branch and a few deeper branches.

Concurrent with our work, “highway networks” [42, 43] present shortcut connections with gating functions [15]. These gates are data-dependent and have parameters, in contrast to our identity shortcuts that are parameter-free. When a gated shortcut is “closed” (approaching zero), the layers in highway networks represent non-residual functions. On the contrary, our formulation always learns residual functions; our identity shortcuts are never closed, and all information is always passed through, with additional residual functions to be learned. In addition, high way networks have not demonstrated accuracy gains with extremely increased depth (e.g., over 100 layers).

残差表示

在图像识别领域，VLAD是一种由残差向量相对于字典进行编码的表示，Fisher Vector可以表示为 VLAD 的概率版本。它们都是面向图像检索和分类任务的强大的浅层表示。 对于矢量量化，编码残差向量被证明比编码原始矢量更有效。

在低层次视觉和计算机图形学中，为了求解偏微分方程 PDE，被广泛使用的多重网格方法Muiltigrid将系统重新表述为多个尺度的子问题，其中每个子问题负责粗粒度尺度和细粒度尺度间的残差解。Multigrid的一种替代方法是分层基础预处理，它依赖于两个尺度间残差向量的变量。有研究已经表明这些求解器的收敛速度比未利用残差性质的标准求解器快得多。这些方法表明，好的重构或预处理可以简化优化过程。

残差链接

关于残差连接的实践和理论已经被研究了很长一段时间。一个训练多层感知器MLP的早期实践是添加一个连接输入到输出的线性层 。在[44, 24]中，一些中间层被直接连接到辅助分类器来解决梯度消失/爆炸问题。 [39, 38, 31, 47]的论文提出了一个通过残差连接实现的层响应、梯度和传播误差的居中方法。在[44]中，一个“原始”层由一个shortcut分支和一些更深的分支组成。

在我们的工作的同时，highway networks展示了带有门控功能的残差连接。这些门控函数依赖于数据并且带来了额外的参数，而与我们的恒等残差连接与之相反，是无参的。当门控捷径关闭（接近零）时，highway networks中的层代表为无残差函数。相反，我们的公式总是在学习残差函数，我们的恒等残差连接永远不会关闭，所有信息总是连带着额外的待学习的残差函数被传输。此外，highway networks没有表现出随着深度增加的准确性提升（如超过 100 层）。

Let us consider H(x) as an underlying mapping to be fit by a few stacked layers (not necessarily the entire net), with x denoting the inputs to the first of these layers. If one hypothesizes that multiple nonlinear layers can asymptotically approximate complicated functions2 , then it is equivalent to hypothesize that they can asymptotically approximate the residual functions, i.e., H(x) − x (assuming that the input and output are of the same dimensions). So rather than expect stacked layers to approximate H(x), we explicitly let these layers approximate a residual function F(x) := H(x) − x. The original function thus becomes F(x)+x. Although both forms should be able to asymptotically approximate the desired functions (as hypothesized), the ease of learning might be different.

This reformulation is motivated by the counterintuitive phenomena about the degradation problem (Fig. 1, left). As we discussed in the introduction, if the added layers can be constructed as identity mappings, a deeper model should have training error no greater than its shallower counterpart. The degradation problem suggests that the solvers might have difficulties in approximating identity mappings by multiple nonlinear layers. With the residual learning reformulation, if identity mappings are optimal, the solvers may simply drive the weights of the multiple nonlinear layers toward zero to approach identity mappings.

In real cases, it is unlikely that identity mappings are optimal, but our reformulation may help to precondition the problem. If the optimal function is closer to an identity mapping than to a zero mapping, it should be easier for the solver to find the perturbations with reference to an identity mapping, than to learn the function as a new one. We show by experiments (Fig. 7) that the learned residual functions in general have small responses, suggesting that identity mappings provide reasonable preconditioning.

让我们把H(x)看作几个人堆叠层（不一定是整个网络）拟合的底层映射，其中x表示这些层中第一个层的输入。如果假设多个非线性层可以逐渐逼近复杂函数，那么就等价于假设它们可以逐渐逼近残差函数，即H(x)−x（假设输入和输出的维度相同）。因此，与其期望堆叠层逼近H(x)，我们让这些层去逼近残差函数F(x):=H(x)−x。初始的函数H(x)也就因此变成了F(x)+x。尽管这两种形式都能够逐渐逼近我们需要的函数（如假设所述），但学习的难易程度可能是不同的。

这种重构表示的想法的是由上述违反直觉的衰退问题现象所启发的。正如我们在引入中所讨论的，如果添加的层可以构建为恒等映射，那么更深的模型的训练误差应该不大于它对应的较浅的模型。衰退问题表明求解器可能难以通过多个非线性层去逼近恒等映射。通过残差学习的表示重构，如果恒等映射是最优的，求解器可以简单地将多个非线性层的权重推向零以接近恒等映射。

在实际情况下，恒等映射不可能是最优的，但我们的重构表示可能有助于先决问题。如果最优函数更接近恒等映射而不是零映射，那么求解器就应该更容易参考恒等映射找到扰动项，而不是把待求解函数当作新函数去学习。 我们通过实验表明，学习到的残差函数通常具有较小的响应，这表明恒等映射提供了合理的预处理。

我们测试了各种普通/残差网络，观察到了一致的现象。为了提供讨论的实例，我们在下文描述了进行ImageNet任务的两个模型：

普通网络

我们的普通网络基础架构主要受 VGG 网络的思想启发。卷积层大多使用的是 3×3 的过滤器，并遵循两个简单的设计规则：（i）对于相同的输出特征图大小，层具有相同数量的过滤器； (ii) 如果特征图大小减半，就通过过滤器数量的加倍来保持每层的时间复杂度。我们直接通过步长为 2 的卷积层进行下采样。整个网络以全局平均池化层和带有 softmax的1000类全连接层结束，加权层的总数为 34。

值得注意的是，相比于VGG网络，我们的模型具有更少的过滤器和更低的计算复杂度。我们的34层基础架构有36亿次FLOP（乘加），仅为 VGG-19（196 亿次 FLOP）的 18%。

残差网络

基于上述普通网络，我们添加了残差连接，将网络转换为其对应的残差版本。当输入和输出的维度相同时（Figure 3 中的实线残差连接），可以直接使用恒等映射方式进行残差连接。当维度增加时（Figure 3 中的虚线残差连接），我们考虑两种选择：（A）残差连接仍然执行恒等映射，填充额外的零以增加维度，这个选项没有引入额外的参数； (B) 对恒等映射进行投影用于匹配维度（由 1×1 卷积完成）。对于这两个选项，当残差连接跨越两种尺寸的特征图时，它们将由步长为2的卷积完成尺寸适配。