Generic modular sparse three-dimensional (3D) convolution design utilizing sparse 3D group convolution
Embodiments are generally directed to sparse 3D convolution acceleration in a convolutional layer of an artificial neural network model. An embodiment of an apparatus includes one or more processors including a graphics processor to process data; and a memory for storage of data, including feature maps. The one or more processors are to provide for sparse 3D convolution acceleration by applying a shared 3D convolutional kernel/filter to an input feature map to produce an output feature map, including increasing sparsity of the input feature map by partitioning it into multiple disjoint input groups; generation of multiple disjoint output groups corresponding to the input groups by performing a convolution calculation represented by the shared 3D convolutional kernel/filter on all feature values associated with active/valid voxels of each input group to produce corresponding feature values within corresponding output groups; and outputting the output feature map by sequentially stacking the output groups.
1 . An apparatus for sparse three-dimensional (3D) convolution acceleration comprising:
one or more processors including a graphics processor to process data; and
a memory for storage of data, including feature maps;
wherein the one or more processors are to provide for sparse 3D convolution acceleration in a first convolutional layer of a plurality of convolutional layers of a neural network model to facilitate training of the neural network model to perform analysis or processing of image data by applying a shared 3D convolutional kernel/filter to an input feature map derived from the image data to produce an output feature map, including:
linearly increasing sparsity of the input feature map by partitioning the input feature map into a plurality (G) of disjoint input groups having dimensions of magnitudes equal to the input feature map, wherein said partitioning includes programmatically selecting active/valid voxels of the input feature map for inclusion in respective input groups of the plurality of disjoint input groups;
generation of a plurality of disjoint output groups corresponding to the plurality of disjoint input groups by, for each input group of the plurality of disjoint input groups, performing, a convolution operation using the shared 3D convolutional kernel/filter on all input feature values associated with all or a subset of active/valid voxels of the input group to produce corresponding output feature values within a corresponding output group of the plurality of disjoint output groups by sliding the shared 3D convolutional kernel/filter according to a stride, left-to-right and top-to-bottom, within a spatial dimension of the input group; and
creating the output feature map with enhanced feature representation for use by a subsequent convolutional layer of the plurality of convolutional layers, wherein the output feature map merges all output feature values associated with active/valid voxels of each of the plurality of disjoint output groups into the output feature map by sequentially stacking the plurality of disjoint output groups.
2 . The apparatus of claim 1 , wherein a sparsity of the input feature map is s % and a sparsity of an average input group representative of the plurality of disjoint input groups is G*s %.
3 . The apparatus of claim 1 , wherein said partitioning the input feature map into a plurality (G) of disjoint input groups is performed along the spatial dimension of the input feature map and applies to a plurality of input channels.
4 . The apparatus of claim 3 , wherein the active/valid voxels of the input group are efficiently identified with reference to a hash table containing a plurality of entries, wherein each entry of the plurality entries corresponds to a particular active/valid voxel and specifies (i) the spatial dimension of the particular active/valid voxel in terms of w, h and d coordinates and (ii) a group identifier of a particular input group of the plurality of disjoint input groups to which the particular active/valid voxel has been partitioned.
5 . The apparatus of claim 1 , wherein said partitioning the input feature map into a plurality (G) of disjoint input groups is performed independently for each of a plurality of input channel groups.
6 . The apparatus of claim 1 , wherein said partitioning the input feature map into a plurality (G) of disjoint input groups includes dividing the input feature map based on a randomly generated group identifier specifying a group of the plurality of disjoint input groups for each of the active/valid voxels.
7 . The apparatus of claim 1 , wherein said partitioning the input feature map into a plurality (G) of disjoint input groups includes generating a group identifier (GID) specifying a group of the plurality of disjoint input groups to which each of the active/valid voxels is to be partitioned based on a mapping function of a form:
GID=mod( aw+bh+cd;G )
where:
mod is a modulus operation;
(w, h, d) represents a position of a particular active/valid voxel in the input feature map; and
a, b, and c are configurable parameters to control the partitioning.
8 . A method for sparse three-dimensional (3D) convolution acceleration as part of training a neural network model implemented on a data processing system to perform analysis or processing of image data, the method comprising:
receiving, by a first convolutional layer of a plurality of convolutional layers of the neural network model an input feature map derived from the image data;
linearly increasing sparsity of the input feature map, by partitioning, by the first convolutional layer, the input feature map into a plurality (G) of disjoint input groups having dimensions of magnitudes equal to the input feature map;
generating, by the first convolutional layer, a plurality of disjoint output groups corresponding to the plurality of disjoint input groups by, for each input group of the plurality of disjoint input groups, performing, a convolution operation using a shared 3D convolutional kernel/filter associated with the first convolutional layer on all input feature values of all or a subset of active/valid voxels of the input group to produce corresponding output feature values within a corresponding output group of the plurality of disjoint output groups by sliding the convolutional kernel/filter according to a stride, left-to-right and top-to-bottom, within a spatial dimension of the input group; and
creating, by the first convolutional layer, an output feature map with enhanced feature representation for use by a subsequent convolutional layer of the plurality of convolutional layers, wherein the output feature map merges all output feature values associated with active/valid voxels of each of the plurality of disjoint output groups into the output feature map by sequentially stacking the plurality of disjoint output groups.
9 . The method of claim 8 , wherein a sparsity of the input feature map is s % and a sparsity of an average input group representative of the plurality of disjoint input groups is G*s %.
10 . The method of claim 8 , wherein the active/valid voxels of the input group are efficiently identified with reference to a hash table containing a plurality of entries, wherein each entry of the plurality entries corresponds to a particular active/valid voxel and specifies (i) a spatial dimension of the particular active/valid voxel in terms of w, h and d coordinates and (ii) a group identifier of a particular input group of the plurality of disjoint input groups to which the particular active/valid voxel has been partitioned.
11 . The method of claim 8 , wherein said partitioning the input feature map into a plurality (G) of disjoint input groups comprise independently performing said partitioning for each of a plurality of input channel groups.
12 . The method of claim 8 , wherein said partitioning the input feature map into a plurality (G) of disjoint input groups comprises dividing an equal number of voxels of the input feature map among each of the plurality of disjoint input groups based on a randomly generated group identifier specifying a group of the plurality of disjoint input groups for each of the active/valid voxels.
13 . The method of claim 8 , wherein said partitioning the input feature map into a plurality (G) of disjoint input groups comprises generating a group identifier (GID) specifying a group of the plurality of disjoint input groups to which each of the active/valid voxels is to be partitioned based on a mapping function of a form:
GID=mod( aw+bh+cd;G )
where:
mod is a modulus operation;
(w, h, d) represents a position of a particular active/valid voxel in the input feature map; and
a, b, and c are configurable parameters to control the partitioning.
14 . A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processors, including a graphics processor, causes the one or more processors to, as part of training of a neural network model to perform analysis or processing of image data:
receive, by a first convolutional layer of a plurality of convolutional layers of the neural network model, an input feature map derived from the image data;
linearly increase sparsity of the input feature map, by partitioning, by the first convolutional layer, the input feature map into a plurality (G) of disjoint input groups having dimensions of magnitudes equal to the input feature map;
generate, by the first convolutional layer, a plurality of disjoint output groups corresponding to the plurality of disjoint input groups by, for each input group of the plurality of disjoint input groups, performing, a convolution operation using a shared 3D convolutional kernel/filter associated with the convolutional layer on all input feature values of all or a subset of active/valid voxels of the input group to produce corresponding output feature values within a corresponding output group of the plurality of disjoint output groups by sliding the shared 3D convolutional kernel/filter according to a stride, left-to-right and top-to-bottom, within a spatial dimension of the input group; and
create, by the first convolutional layer, an output feature map with enhanced feature representation for use by a subsequent convolutional layer of the plurality of convolutional layers, wherein the output feature map merges all output feature values associated with active/valid voxels of each of the plurality of disjoint output groups into the output feature map by sequentially stacking the plurality of disjoint output groups.
15 . The non-transitory computer-readable storage medium of claim 14 , wherein a sparsity of the input feature map is s % and a sparsity of an average input group representative of the plurality of disjoint input groups is G*s %.
16 . The non-transitory computer-readable storage medium of claim 14 , wherein the active/valid voxels of the input group are efficiently identified with reference to a hash table containing a plurality of entries, wherein each entry of the plurality entries corresponds to a particular active/valid voxel and specifies (i) a spatial dimension of the particular active/valid voxel in terms of w, h and d coordinates and (ii) a group identifier of a particular input group of the plurality of disjoint input groups to which the particular active/valid voxel has been partitioned.
17 . The non-transitory computer-readable storage medium of claim 14 , wherein said partitioning the input feature map into a plurality (G) of disjoint input groups comprises:
independently performing said partitioning for each of a plurality of input channel groups; or
dividing an equal number of voxels of the input feature map among each of the plurality of disjoint input groups based on a randomly generated group identifier specifying a group of the plurality of disjoint input groups for each of the active/valid voxels.
18 . The apparatus of claim 1 , wherein said partitioning the input feature map into a plurality (G) of disjoint input groups is performed in accordance with a first partitioning strategy for a first input channel group of a plurality of input channel groups and in accordance with a second partitioning strategy for a second input channel group of the plurality of input channel groups.
19 . The method of claim 8 , wherein said partitioning the input feature map into a plurality (G) of disjoint input groups is performed in accordance with a first partitioning strategy for a first input channel group of a plurality of input channel groups and in accordance with a second partitioning strategy for a second input channel group of the plurality of input channel groups.
20 . The non-transitory computer-readable storage medium of claim 14 , wherein said partitioning the input feature map into a plurality (G) of disjoint input groups is performed in accordance with a first partitioning strategy for a first input channel group of a plurality of input channel groups and in accordance with a second partitioning strategy for a second input channel group of the plurality of input channel groups.