Method and system for graph manipulations for machine learning inference engine
An image is received/divided into a first/second slices, at a first convolutional layer in a CNN. A first/second tensor data for a second/third convolutional layer are generated for the first slice respectively using a kernel. The first tensor data is divided into a first/second portions. The second portion is needed by the second convolutional layer of the second slice to complement partial tensor data generated by the second slice. The second tensor data is divided into a first/second portions where the second portion is needed by the third convolutional layer of the second slice to complement partial tensor data generated by the second slice. The first/second/third convolutional layers of the first slice are processed using a subset of processing tiles. The second portion of the first tensor data and the second portion of the second tensor data generated by the first slice are stored in a memory component.
1 . A compiler implemented method, comprising:
receiving an image at a first convolutional layer in a convolutional neural network (CNN);
dividing the image into a first slice and a second slice associated with the first convolutional layer;
generating a first tensor data associated with a second convolutional layer for the first slice using a kernel with associated weights and using a stride value;
generating a second tensor data associated with a third convolutional layer for the first slice using the kernel with associated weights and using the stride value;
dividing the first tensor data associated with the second convolutional layer into a first portion and a second portion, wherein the second portion of the first tensor data is needed by the second convolutional layer of the second slice to complement partial tensor data generated by the second slice;
dividing the second tensor data associated with the third convolutional layer into a first portion and a second portion, wherein the second portion of the second tensor data is needed by the third convolutional layer of the second slice to complement partial tensor data generated by the second slice;
processing the first convolutional layer of the first slice, the second convolutional layer of the first slice, and the third convolutional layer of the first slice using a subset of processing tiles from a plurality of processing tiles; and
storing the second portion of the first tensor data and the second portion of the second tensor data generated by the first slice in a memory component.
2 . The compiler implemented method of claim 1 , wherein the method further comprises:
generating a first tensor data associated with the second convolutional layer for the second slice using the kernel with associated weights and using the stride value; and
generating a second tensor data associated with the third convolutional layer for the second slice using the kernel with associated weights and using the stride value.
3 . The compiler implemented method of claim 2 further comprising complementing partial tensor data in the second convolutional layer and the third convolutional layer that are generated by the second slice with the second portion of the first tensor data and the second portion of the second tensor data generated by the first slice.
4 . The compiler implemented method of claim 2 , wherein a portion of tensor data being generated in the second slice is generated independent of the first tensor data of the first slice and further independent of the second tensor data of the first slice.
5 . The compiler implemented method of claim 4 , wherein another portion of the tensor data being generated in the second slice is generated subsequent to the first tensor data of the first slice and further independent of the second tensor data of the first slice.
6 . The compiler implemented method of claim 1 , wherein the subset of processing tiles from the plurality of processing tiles is the same as another subset of processing tiles from the plurality of processing tiles that process the second slice.
7 . The compiler implemented method of claim 1 , wherein the memory component is an on-chip memory (OCM) within the plurality of processing tiles.
8 . The compiler implemented method of claim 1 , wherein the memory component is a double data rate (DDR) memory external to the plurality of processing tiles.
9 . The compiler implemented method of claim 1 , wherein data in the second slice that includes the first convolutional layer, the second convolutional layer, and the third convolutional layer includes an independent portion data, wherein the independent portion data has no dependency on data from the first slice.
10 . The compiler implemented method of claim 9 , wherein the independent portion data is processed in parallel with the generating the first and the second tensor data for the first slice.
11 . A system comprising:
a means for receiving an image as a first convolution layer in a convolutional neural network (CNN);
a means for dividing the image into a first slice and a second slice associated with the first convolutional layer;
a means for generating a first tensor data associated with a second convolutional layer for the first slice using a kernel with associated weights and using a stride value;
a means for generating a second tensor data associated with a third convolutional layer for the first slice using the kernel with associated weights and using the stride value;
a means for dividing the first tensor data associated with the second convolutional layer into a first portion and a second portion, wherein the second portion of the first tensor data is needed by the second convolutional layer of the second slice to complement partial tensor data generated by the second slice;
a means for dividing the second tensor data associated with the third convolutional layer into a first portion and a second portion, wherein the second portion of the second tensor data is needed by the third convolutional layer of the second slice to complement partial tensor data generated by the second slice;
a means for processing the first convolutional layer of the first slice, the second convolutional layer, and the third convolutional layer using a subset of processing tiles from a plurality of processing tiles; and
a means for storing the second portion of the first tensor data and the second portion of the second tensor data generated by the first slice in a memory component.
12 . The system of claim 11 further comprising:
means for generating a first tensor data associated with the second convolutional layer for the second slice using the kernel with associated weights and using the stride value; and
means for generating a second tensor data associated with the third convolutional layer for the second slice using the kernel with associated weights and using the stride value.
13 . The system of claim 12 further comprising:
means for complementing partial tensor data in the second convolutional layer and the third convolutional layer that are generated by the second slice with the second portion of the first tensor data and the second portion of the second tensor data generated by the first slice.
14 . The system of claim 12 , wherein a portion of tensor data being generated in the second slice is generated independent of the first tensor data of the first slice and further independent of the second tensor data of the first slice.
15 . The system of claim 14 , wherein another portion of the tensor data being generated in the second slice is generated subsequent to the first tensor data of the first slice and further independent of the second tensor data of the first slice.
16 . The system of claim 11 , wherein the subset of processing tiles from the plurality of processing tiles is the same as the another subset of processing tiles from the plurality of processing tiles.
17 . The system of claim 11 , wherein the memory component is an on-chip memory (OCM) within the plurality of processing tiles.
18 . The system of claim 11 , wherein the memory component is a double data rate (DDR) memory external to the plurality of processing tiles.
19 . The system of claim 11 , wherein data in the second slice that includes the first convolutional layer, the second convolutional layer, and the third convolutional layer includes an independent portion data, wherein the independent portion data has no dependency on data from the first slice.
20 . The system of claim 19 , wherein the independent portion data is processed in parallel with the generating the first and the second tensor data for the first slice.