Residual connection

The model processes a text description through many layers. If every layer had to rebuild all the information, forwarding it could easily become difficult. Retaining the input and adding only the computed change is helpful.

A residual connection combines a block's input with its output by addition. The block can learn a correction rather than replace the whole earlier description. If the input list of numbers is h and the computed correction is u, h + u is passed onward.

Input (2, 5) and correction (1, −2) give (3, 3). We do not copy the input unchanged: we retain both numbers' direct contribution and modify them. These connections help transmit signals and train deep networks. He and colleagues, §3, present residual learning.

In a Transformer, addition surrounds the main parts of the blocks. A residual connection does not guarantee removal of an earlier layer's error: the correction can reduce or amplify it. This depends on its direction and magnitude, as the quantization error example shows.