Parallel residual neural network architecture and system and method for training a residual neural network
View Patent ↗Systems and methods for training a residual neural network are described. One of the methods includes: initializing one or more weights of each of a plurality of residual units; inputting a plurality of training cases to a first warp unit in a series; using each training case to optimize the one or more weights for each residual unit in parallel in the first warp unit in the series; starting with the output of the first warp unit in the series, iteratively propagating the output of each warp unit to the input of a next respective warp unit in the series, for each respective warp unit, using each training case to optimize the one or more weights for each residual unit in parallel in the respective warp unit; and storing the output of the last warp unit in the series and the weights for each residual unit.
1. A method for training a residual neural network executed on one or more processing units, the residual neural network comprising a plurality of warp units connected in series, each warp unit comprising an input, an output, a plurality of independent residual units connected in parallel from the input to the output, a direct connection from the input to the output, and at least one derivative unit connected from the input to the output in parallel to the residual units, each residual unit comprising one or more weights, the method comprising:
initializing the one or more weights of each residual unit;
inputting a plurality of training cases to the first warp unit in the series;
using each training case to optimize the one or more weights for each residual unit in parallel in the first warp unit in the series;
starting with the output of the first warp unit in the series, iteratively propagating the output of each warp unit to the input of the next respective warp unit in the series, for each respective warp unit, using each training case to optimize the one or more weights for each residual unit in parallel in the respective warp unit; and
storing the output of the last warp unit in the series and the weights for each residual unit.
2. The method of claim 1 , wherein the at least one derivative unit is determined using at least one multiplication of one of the residual units to a derivative of another one of the residual units.
3. The method of claim 1 , wherein the at least one derivative unit is determined as a derivative of at least one of the residual units.
4. The method of claim 1 , wherein the at least one derivative unit is determined as a derivative of at least one of the residual units multiplied by the input.
5. The method of claim 1 , wherein the output is determined from the input using first-order Taylor Series Expansion of the residual units and the at least one derivative unit.
6. The method of claim 1 , further comprising performing back propagation comprising propagating a weight gradient of a loss function from the last warp unit in the series to the first warp unit, in each warp unit, the one or more weights of each residual unit are independently updated in parallel.
7. The method of claim 6 , wherein the weight gradient for each warp unit is determined as a gradient of the output of the warp unit multiplied by a sum comprising the gradients of the residual units and gradients of the at least one derivative unit.
8. The method of claim 7 , wherein the gradient of each of the at least one derivative units comprising at least one multiplication of the gradient of one of the residual units to the gradient of another one of the residual units.
9. The method of claim 8 , wherein the sum further comprises the identity matrix.
10. The method of claim 1 , wherein at least two of the warp units comprising a differing quantity of residual units from each other.
11. A system for training a residual neural network, the system comprising one or more processors and one or more non-transitory computer storage media, the one or more non-transitory computer storage media causing the one or more processors to execute the residual neural network comprising:
a plurality of warp units comprising an input and an output and a direct connection from the input to the output;
for each of the warp units, a plurality of residual units connected in parallel, each residual unit comprising one or more weights;
for each of the warp units, at least one derivative unit connected in parallel to the residual units; and
a warp operator to receive the inputs and outputs from each of the warp units and to train the residual neural network, the training comprising optimizing the weights of each residual unit based on a plurality of training cases.
12. The system of claim 11 , wherein the weights for each residual unit in a warp unit is determined on a separate processing unit.
13. The system of claim 11 , wherein the training of the residual neural network comprising;
initializing the one or more weights of each residual unit;
inputting a plurality of training cases to the first warp unit in the series;
using each training case to optimize the one or more weights for each residual unit in parallel in the first warp unit in the series;
starting with the output of the first warp unit in the series, iteratively propagating the output of each warp unit to the input of the next respective warp unit in the series, for each respective warp unit, using each training case to optimize the one or more weights for each residual unit in parallel in the respective warp unit; and
storing the output of the last warp unit in the series and the weights for each residual unit in the one or more non-transitory computer storage media.
14. The system of claim 13 , wherein the at least one derivative unit is determined using at least one multiplication of one of the residual units to a derivative of another one of the residual units.
15. The system of claim 13 , wherein the at least one derivative unit is determined as a derivative of at least one of the residual units.
16. The system of claim 13 , wherein the at least one derivative unit is determined as a derivative of at least one of the residual units multiplied by the input.
17. The system of claim 13 , wherein the output is determined from the input using first-order Taylor Series Expansion of the residual units and the at least one derivative unit.
18. The system of claim 13 , the warp operator further performing, with the warp units, back propagation, the back propagation comprising propagating a weight gradient of a loss function from the last warp unit in the series to the first warp unit, in each warp unit, the one or more weights of each residual unit are independently updated in parallel.
19. The system of claim 18 , wherein the weight gradient for each warp unit is determined as a gradient of the output of the warp unit multiplied by a sum comprising the gradients of the residual units and gradients of the at least one derivative unit.
20. The system of claim 19 , wherein the gradient of each of the at least one derivative units comprising at least one multiplication of the gradient of one of the residual units to the gradient of another one of the residual units.