Batch normalization for replicated layers of neural network

1. A method for training a machine-trained (MT) network comprising a plurality of layers, each layer comprising a plurality of computation nodes, the method comprising:

propagating a plurality of network inputs through (i) a set of related layers of the MT network, wherein each respective computation node in a first one of the related layers has respective corresponding computation nodes in each of the other related layers of the set of related layers and each respective set of corresponding computation nodes, comprising a respective computation node in each of the related layers, receives a same respective set of layer inputs and applies different respective sets of weights to the respective set of layer inputs to generate a respective output, and (ii) an element-wise addition layer comprising computation nodes, wherein each respective computation node in the element-wise addition layer adds the outputs of a different respective set of corresponding nodes from the set of related layers to generate a sum for the respective computation node in the element-wise addition layer;

after propagating the plurality of network inputs through the set of related layers and the element-wise addition layer:

for each respective layer of the related layers, using outputs generated by the respective computation nodes of the respective related layer to determine a respective first set of batch normalization parameters specific to the respective layer;

using the sums generated by the element-wise addition layer to determine a second set of batch normalization parameters that are shared between the set of related layers; and

for each respective related layer of the related layers, applying (i) the respective first set of batch normalization parameters specific to the respective layer and (ii) the second set of batch normalization parameters to the outputs generated by the respective layer; and

propagating the network inputs, after applying the first and second sets of batch normalization parameters, through subsequent layers of the MT network.

2. The method of claim 1 , wherein the set of related layers and the element-wise addition layer collectively replace a single layer of a previously-trained version of the MT network.

3. The method of claim 1 , wherein propagating the outputs through the subsequent layers of the MT network comprises generating a plurality of network outputs.

4. The method of claim 3 , wherein training the MT network further comprises:

computing a network error based at least in part on comparing the generated plurality of network outputs to expected network outputs; and

back-propagating the network error to modify the weights of the sets of weights of the MT network.

5. The method of claim 3 , wherein;

the respective computation nodes of each respective related layer are grouped into respective channels based on the respective sets of weights applied to the respective sets of inputs to the respective computation nodes; and

each respective channel in one of the related layers has corresponding respective channels in each of the other related layers.

6. The method of claim 5 , wherein, for each respective layer of the set of related layers, using the outputs generated by the respective computation nodes of the respective layer comprises, for each respective channel of the respective layer, computing a mean of the outputs generated by the respective computation nodes of the respective channel while propagating the plurality of network inputs through the MT network.

7. The method of claim 6 , wherein applying, for each respective related layer, the first set of batch normalization parameters specific to the respective layer to the outputs generated by the respective layer comprises, for each respective computation node of the respective related layer, subtracting the computed mean for the respective channel to which the respective computation node belongs from each of the outputs generated by the respective computation node for the plurality of network inputs.

8. The method of claim 7 further comprising, for each respective computation node of each respective related layer, adding a trained shift value for the respective channel to which the respective computation node belongs to each of the outputs generated by the respective computation node for the plurality of network inputs.

9. The method of claim 5 , wherein the computation nodes of the element-wise addition layer are grouped into channels, wherein each respective channel of the element-wise addition layer comprises respective computation nodes that add the outputs of respective computation nodes of respective corresponding channels of the related layers.

10. The method of claim 9 , wherein using the sums generated by the element-wise addition layer to determine the second set of batch normalization parameters comprises, for each respective channel of the element-wise addition layer, computing a variance and a standard deviation of the sums generated by the respective computation nodes of the element-wise addition layer while propagating the plurality of network inputs through the MT network.

11. The method of claim 10 , wherein applying, for each respective related layer, the second set of batch normalization parameters comprises, for each respective computation node of the respective related layer, dividing each of the outputs generated by the respective computation node by the standard deviation computed for the respective channel of the element-wise addition layer corresponding to the respective channel to which the respective computation node belongs.

12. The method of claim 11 further comprising, for each respective computation node of each respective related layer, multiplying each of the outputs generated by the respective computation node by a learned scale value for the respective channel of the element-wise addition layer corresponding to the respective channel to which the respective computation node belongs.

13. A non-transitory machine-readable medium storing a program which when executed by at least one processing unit trains a machine-trained (MT) network comprising a plurality of layers, each layer comprising a plurality of computation nodes, the program comprising sets of instructions for:

propagating a plurality of network inputs through (i) a set of related layers of the MT network, wherein each respective computation node in a first one of the related layers has respective corresponding computation nodes in each of the other related layers of the set of related layers and each respective set of corresponding computation nodes, comprising a respective computation node in each of the related layers, receives a same respective set of layer inputs and applies different respective sets of weights to the respective set of layer inputs to generate a respective output, and (ii) an element-wise addition layer comprising computation nodes, wherein each respective computation node in the element-wise addition layer adds outputs of a different respective set of corresponding nodes from the set of related layers to generate a sum for the respective computation node in the element-wise addition layer;

after propagating the plurality of network inputs through the set of related layers and the element-wise addition layer:

for each respective layer of the related layers, using outputs generated by the respective computation nodes of the respective related layer to determine a respective first set of batch normalization parameters specific to the respective layer;

using the sums generated by the element-wise addition layer to determine a second set of batch normalization parameters that are shared between the set of related layers; and

for each respective related layer of the related layers, applying (i) the respective first set of batch normalization parameters specific to the respective layer and (ii) the second set of batch normalization parameters to the outputs generated by the respective layer; and

propagating the network inputs, after applying the first and second sets of batch normalization parameters, through subsequent layers of the MT network.

14. The non-transitory machine-readable medium of claim 13 , wherein the set of related layers and the element-wise addition layer collectively replace a single layer of a previously-trained version of the MT network.

15. The non-transitory machine-readable medium of claim 13 , wherein the set of instructions for propagating the outputs through the subsequent layers of the MT network comprises a set of instructions for generating a plurality of network outputs.

16. The non-transitory machine-readable medium of claim 15 , wherein the set of instructions for training the MT network further comprises sets of instructions for:

computing a network error based at least in part on comparing the generated plurality of network outputs to expected network outputs; and

back-propagating the network error to modify the weights of the sets of weights of the MT network.

17. The non-transitory machine-readable medium method of claim 15 , wherein:

the respective computation nodes of each respective related layer are grouped into respective channels based on the respective sets of weights applied to the respective sets of inputs to the respective computation nodes; and

each respective channel in one of the related layers has corresponding respective channels in each of the other related layers.

18. The non-transitory machine-readable medium of claim 17 , wherein, for each respective layer of the set of related layers, the set of instructions for using the outputs generated by the respective computation nodes of the respective layer comprises a set of instructions for computing, for each respective channel of the respective layer, a mean of the outputs generated by the respective computation nodes of the respective channel while propagating the plurality of network inputs through the MT network.

19. The non-transitory machine-readable medium of claim 18 wherein:

the set of instructions for applying, for each respective related layer, the first set of batch normalization parameters specific to the respective layer to the outputs generated by the respective layer comprises a set of instructions for, for each respective computation node of the respective related layer, subtracting the computed mean for the respective channel to which the respective computation node belongs from each of the outputs generated by the respective computation node for the plurality of network inputs; and

the program further comprises a set of instructions for, for each respective computation node of each respective related layer, adding a trained shift value for the respective channel to which the respective computation node belongs to each of the outputs generated by the respective computation node for the plurality of network inputs.

20. The non-transitory machine-readable medium of claim 17 , wherein the computation nodes of the element-wise addition layer are grouped into channels, wherein each respective channel of the element-wise addition layer comprises respective computation nodes that add the outputs of respective computation nodes of respective corresponding channels of the related layers.

21. The non-transitory machine-readable medium of claim 20 , wherein the set of instructions for using the sums generated by the element-wise addition layer to determine the second set of batch normalization parameters comprises a set of instructions for computing, for each respective channel of the element-wise addition layer, a variance and standard deviation of the sums generated by the respective computation nodes of the element-wise addition layer while propagating the plurality of network inputs through the MT network.

22. The non-transitory machine-readable medium of claim 21 , wherein the program further comprises a set of instructions for:

for each respective computation node of each respective related layer:

dividing each of the outputs generated by the respective computation node by the standard deviation computed for the respective channel of the element-wise addition layer corresponding to the respective channel to which the respective computation node belongs; and

multiplying each of the outputs generated by the respective computation node by a learned scale value for the respective channel of the element-wise addition layer corresponding to the respective channel to which the respective computation node belongs.