Lossless tiling in convolution networks—weight gradient calculation
Disclosed is a data processing system that includes compile time logic configured to process a processing graph to generate a modified processing graph, which includes a plurality of forward processing nodes of a forward pass and a plurality of backward processing nodes of a backward pass. The data processing system also includes runtime logic configured with the compile time logic to execute the modified processing graph to generate, at a backward processing node of the plurality of backward processing nodes, a plurality of partial weight gradients, based on processing a corresponding plurality of gradient tiles of a gradient tensor, and generate, based on the plurality of partial weight gradients, a final weight gradient corresponding to the gradient tensor.
1. A non-transitory computer readable storage medium impressed with computer program instructions, the instructions, when executed on a processor, implement a method comprising:
generating a plurality of partial weight gradients associated with weights of a convolution kernel of a processing node of a forward pass of a neural network, based on processing a corresponding plurality of gradient tiles of a gradient tensor, such that (i) a first partial weight gradient of the plurality of partial weight gradients is generated based on processing a corresponding first gradient tile of the plurality of gradient tiles of the gradient tensor, and (ii) a second partial weight gradient of the plurality of partial weight gradients is generated based on processing a corresponding second gradient tile of the plurality of gradient tiles of the gradient tensor;
generating, based on the plurality of partial weight gradients, a final weight gradient corresponding to the gradient tensor; and
training the weights of the convolution kernel of the processing node of the forward pass of the neural network, based on the final weight gradient.
2. The non-transitory computer readable storage medium of claim 1 , wherein generating the final weight gradient comprises:
summing the plurality of partial weight gradients, to generate the final weight gradient.
3. The non-transitory computer readable storage medium of claim 2 , wherein generating the final weight gradient comprises:
averaging the sum of the plurality of partial weight gradients, to generate the final weight gradient.
4. The non-transitory computer readable storage medium of claim 1 , wherein each of the plurality of partial weight gradients and the final weight gradient has a same dimension.
5. The non-transitory computer readable storage medium of claim 1 , wherein the plurality of partial weight gradients is generated based on processing the corresponding plurality of gradient tiles of the gradient tensor and a corresponding plurality of input tiles of an input tensor.
6. The non-transitory computer readable storage medium of claim 5 , wherein:
the plurality of input tiles of the input tensor is generated by an (L) th layer of a forward pass of a processing graph;
the plurality of gradient tiles of the gradient tensor is generated by an (L+1) th layer of a backward pass of the processing graph; and
plurality of partial weight gradients is generated by an (L) th layer of the backward pass of the processing graph.
7. The non-transitory computer readable storage medium of claim 5 , wherein generating the plurality of partial weight gradients comprises:
generating the first partial weight gradient of the plurality of partial weight gradients, based on processing the first gradient tile of the plurality of gradient tiles and a first input tile of the plurality of input tiles; and
generating the second partial weight gradient of the plurality of partial weight gradients, based on processing the second gradient tile of the plurality of gradient tiles and a second input tile of the plurality of input tiles.
8. The non-transitory computer readable storage medium of claim 6 , wherein the method further comprises:
training weights of the (L) th layer of the forward pass, based on the final weight gradient generated for the (L) th layer of the backward pass.
9. The non-transitory computer readable storage medium of claim 1 , wherein the method further comprises:
generating, by one or more on-chip reconfigurable processors, the plurality of partial weight gradients;
storing, on one or more on-chip memory, the generated plurality of partial weight gradients;
generating the final weight gradient, based on the plurality of partial weight gradients stored on the one or more on-chip memory; and
writing the final weight gradient to an off-chip memory.
10. The non-transitory computer readable storage medium of claim 9 , wherein one or more, or all, of the plurality of partial weight gradients is not stored in the off-chip memory.
11. The non-transitory computer readable storage medium of claim 1 , wherein each of the plurality of partial weight gradients, the final weight gradient, and the convolution kernel of the processing node has a same dimension.
12. A data processing system, comprising:
storage medium storing instructions associated with compile time logic and runtime logic; and
one or more processors coupled to the storage medium and executing the compile time logic and runtime logic,
wherein the compile time logic, when executed by any of the one or more processors, processes a processing graph to generate a modified processing graph comprising a plurality of forward processing nodes of a forward pass and a plurality of backward processing nodes of a backward pass of a neural network model; and
wherein the runtime logic, when executed by any of the one or more processors, is configured by the compile time logic to execute the modified processing graph to:
generate, at a backward processing node of the plurality of backward processing nodes, a plurality of partial weight gradients, based on processing a corresponding plurality of gradient tiles of a gradient tensor, such that (i) a first partial weight gradient of the plurality of partial weight gradients is generated based on processing a corresponding first gradient tile of the plurality of gradient tiles of the gradient tensor, and (ii) a second partial weight gradient of the plurality of partial weight gradients is generated based on processing a corresponding second gradient tile of the plurality of gradient tiles of the gradient tensor,
generate, based on the plurality of partial weight gradients, a final weight gradient corresponding to the gradient tensor, and
train weights of a forward processing node of the plurality of forward processing nodes of the forward pass, based on the final weight gradient.
13. A computer implemented method, comprising:
generating a plurality of partial weight gradients associated with weights of a convolution kernel of a processing node of a forward pass of a neural network, based on processing a corresponding plurality of gradient tiles of a gradient tensor, such that (i) a first partial weight gradient of the plurality of partial weight gradients is generated based on processing a corresponding first gradient tile of the plurality of gradient tiles of the gradient tensor, and (ii) a second partial weight gradient of the plurality of partial weight gradients is generated based on processing a corresponding second gradient tile of the plurality of gradient tiles of the gradient tensor;
generating, based on the plurality of partial weight gradients, a final weight gradient corresponding to the gradient tensor; and
training weights of the convolution kernel of the processing node of the forward pass of the neural network, based on the final weight gradient.
14. The method of claim 13 , wherein generating the final weight gradient comprises:
summing the plurality of partial weight gradients, to generate the final weight gradient.
15. The method of claim 14 , wherein generating the final weight gradient comprises:
averaging the sum of the plurality of partial weight gradients, to generate the final weight gradient.
16. The method of claim 13 , wherein each of the plurality of partial weight gradients and the final weight gradient has a same dimension.
17. The method of claim 13 , wherein the plurality of partial weight gradients is generated based on processing the corresponding plurality of gradient tiles of the gradient tensor and a corresponding plurality of input tiles of an input tensor.
18. The method of claim 17 , wherein:
the plurality of input tiles of the input tensor is generated by an (L) th layer of a forward pass of a processing graph;
the plurality of gradient tiles of the gradient tensor is generated by an (L+1) th layer of a backward pass of the processing graph; and
plurality of partial weight gradients is generated by an (L) th layer of the backward pass of the processing graph.
19. The method of claim 18 , wherein generating the plurality of partial weight gradients comprises:
generating the first partial weight gradient of the plurality of partial weight gradients, based on processing the first gradient tile of the plurality of gradient tiles and a first input tile of the plurality of input tiles; and
generating the second partial weight gradient of the plurality of partial weight gradients, based on processing the second gradient tile of the plurality of gradient tiles and a second input tile of the plurality of input tiles.
20. The method of claim 18 , further comprising:
training weights of the (L) th layer of the forward pass, based on the final weight gradient generated for the (L) th layer of the backward pass.