Spatially sparse convolutional neural networks for inking applications
A spatially sparse convolutional neural network (CNN) framework is introduced to that leverages high sparsity of input data to significantly reduce the computational cost of applications that employ CNNs (e.g., inking applications and others) by avoiding unnecessary floating point mathematical operations. The framework, which is compatible with parallelized operations, includes (1) a data structure for sparse tensors that both (a) reduces storage burden and (b) speeds computations; (2) a set of sparse tensor operations that accelerate convolution computations; and (3) the merging of pooling and convolutional layers. Practical applications involving handwriting recognition and/or stroke analysis demonstrate a notable reduction in storage and computational burdens.
1. A spatially sparse convolutional neural network (CNN) system for inking applications, the system comprising:
at least one processor; and
a computer-readable medium storing instructions that are operative when executed by the at least one processor to:
receive input data as a sparse tensor;
store the input data in a sparse tensor data structure, wherein the sparse tensor data structure includes a non-zero element having a value and an index;
perform a convolution operation using the input data in the sparse tensor data structure and a kernel, wherein the non-zero element of the input data is multiplied by an element of the kernel that is indexed based at least on the index of the non-zero element of the input data;
transpose a three-dimensional (3D) tensor output of the convolution operation from a first order to a second order, wherein each of the first order and the second order comprises a width, height, and channel, wherein the width is a number of columns of pixels, the height is a number of rows of pixels, and the channel represents a number of convolutions stages or processes; and
determine, based at least on the transposed 3D tensor output of the convolution operation, an output character or object representing the input data.
2. The system of claim 1 , wherein the received input data comprises a three dimensional (3D) tensor.
3. The system of claim 1 , wherein the sparse tensor data structure omits zero-value elements of the input data.
4. The system of claim 1 , wherein the sparse tensor data structure includes an end-of-dimension indicator.
5. The system of claim 1 , wherein the convolution operation omits multiplication operations for zero-value elements of the input data.
6. The system of claim 1 , wherein performing the convolution operation comprises performing the convolution operation on channel-height-width (CHW) ordered data.
7. The system of claim 1 , wherein the 3D transposed 3D tensor output is a 3D matrix.
8. The system of claim 1 , wherein the first order is width-height-channel (WHC) order and the second order is channel-height-width (CHW) order.
9. The system of claim 1 , wherein the input data represents handwriting.
10. A method of spatially sparse convolution for inking applications, the method comprising:
receiving input data as a sparse tensor;
storing the input data in a sparse tensor data structure, wherein the sparse tensor data structure includes a non-zero element having a value and an index;
performing a convolution operation using the input data in the sparse tensor data structure and a kernel, wherein the non-zero element of the input data is multiplied by an element of the kernel that is indexed based at least on the index of the non-zero element of the input data;
transposing an output of the convolution operation from a first order to a second order, wherein each of the first order and the second order comprises a width, height, and channel, wherein the width is a number of columns of pixels, the height is a number of rows of pixels, and the channel represents a number of convolutions stages or processes; and
determining, based at least on the transposed output of the convolution operation, an output character or object representing the input data.
11. The method of claim 10 , wherein the received input data comprises a three dimensional (3D) tensor.
12. The method of claim 10 , wherein the sparse tensor data structure omits zero-value elements of the input data.
13. The method of claim 10 , wherein the sparse tensor data structure includes an end-of-dimension indicator.
14. The method of claim 10 , wherein the convolution operation omits multiplication operations for zero-value elements of the input data.
15. The method of claim 10 , wherein performing the convolution operation comprises performing the convolution operation on channel-height-width (CHW) ordered data.
16. The method of claim 10 , further comprising:
transposing an output of the convolution operation from width-height-channel (WHC) order to channel-height-width (CHW) order.
17. The method of claim 10 , wherein the input data is pixel data.
18. The method of claim 10 , wherein the input data represents handwriting.
19. One or more computer storage devices having computer-executable instructions stored thereon for spatially sparse convolution, which, on execution by a computer, cause the computer to perform operations comprising:
receiving input data as a sparse tensor, wherein the received input data comprises a three dimensional (3D) tensor;
storing the input data in a sparse tensor data structure, wherein the sparse tensor data structure includes a non-zero element having a value and an index, wherein the sparse tensor data structure omits zero-value elements of the input data, and wherein the sparse tensor data structure includes an end-of-dimension indicator;
performing a convolution operation using the input data in the sparse tensor data structure and a kernel, wherein the non-zero element of the input data is multiplied by an element of the kernel that is indexed based at least on the index of the non-zero element of the input data, wherein the convolution operation omits multiplication operations for zero-value elements of the input data, and wherein performing the convolution operation comprises performing the convolution operation on channel-height-width (CHW) ordered data; and
transposing an output of the convolution operation from a first order to a second order, wherein each of the first order and the second order comprises a width, height, and channel, wherein the width is a number of columns of pixels, the height is a number of rows of pixels, and the channel represents a number of convolutions stages or processes.
20. The one or more computer storage devices of claim 19 , wherein the input data is pixel data representing handwriting, and wherein the operations further comprise:
determining, based at least on the convolution operation, an output character or object representing the input data.