Multi-domain convolutional neural network
In one embodiment, an apparatus comprises a memory and a processor. The memory is to store visual data associated with a visual representation captured by one or more sensors. The processor is to: obtain the visual data associated with the visual representation captured by the one or more sensors, wherein the visual data comprises uncompressed visual data or compressed visual data; process the visual data using a convolutional neural network (CNN), wherein the CNN comprises a plurality of layers, wherein the plurality of layers comprises a plurality of filters, and wherein the plurality of filters comprises one or more pixel-domain filters to perform processing associated with uncompressed data and one or more compressed-domain filters to perform processing associated with compressed data; and classify the visual data based on an output of the CNN.
1. An apparatus, comprising:
a memory to store visual data associated with a visual representation captured by one or more sensors; and
a processor to:
obtain the visual data associated with the visual representation captured by the one or more sensors, wherein the visual data comprises uncompressed visual data or compressed visual data;
process the visual data using a convolutional neural network (CNN), wherein the CNN comprises a plurality of layers, wherein the plurality of layers comprises a plurality of filters, and wherein the plurality of filters comprises:
one or more pixel-domain filters to perform processing associated with uncompressed data; and
one or more compressed-domain filters to perform processing associated with compressed data; and
classify the visual data based on an output of the CNN.
2. The apparatus of claim 1 , further comprising a communication interface to receive the visual data from one or more devices over a network, wherein the one or more devices are associated with the one or more sensors.
3. The apparatus of claim 1 , wherein:
the CNN is trained to rely on processing associated with the one or more pixel-domain filters when the visual data is uncompressed; and
the CNN is trained to rely on processing associated with the one or more compressed-domain filters when the visual data is compressed.
4. The apparatus of claim 1 , wherein the one or more compressed-domain filters are further to perform processing associated with noncontiguous elements of the visual data, wherein there is a correlation among the noncontiguous elements when the visual data is compressed.
5. The apparatus of claim 1 , wherein the one or more compressed-domain filters comprise one or more butterfly filters, wherein each butterfly filter is to perform a partial inverse transform associated with the visual data.
6. The apparatus of claim 1 , wherein:
the one or more compressed-domain filters comprise one or more three-dimensional (3D) convolution filters to perform processing associated with three-dimensional (3D) visual data;
the 3D visual data is generated based on a three-dimensional (3D) transformation of the visual data; and
the 3D transformation rearranges the visual data into three dimensions such that there is a correlation among one or more adjacent elements within a particular dimension when the visual data used for the 3D transformation is compressed.
7. The apparatus of claim 5 , wherein each butterfly filter is to compute a sum and a difference for a plurality of pairs of elements within the visual data.
8. The apparatus of claim 6 , wherein the processor is further to:
generate the 3D visual data, wherein the 3D visual data is generated by performing the 3D transformation on the visual data; and
provide the 3D visual data as input to the one or more 3D convolution filters.
9. The apparatus of claim 7 , wherein each pair of elements is positioned at locations within the visual data that are a particular distance apart.
10. The apparatus of claim 8 , wherein the processor to generate the 3D visual data is further to:
partition the visual data into a plurality of blocks, wherein each block comprises a plurality of elements;
arrange the plurality of blocks along a first dimension and a second dimension of the 3D visual data; and
arrange the plurality of elements associated with each block along a third dimension of the 3D visual data.
11. The apparatus of claim 9 , wherein the one or more butterfly filters comprise:
a horizontal butterfly filter, wherein the plurality of pairs of elements processed by the horizontal butterfly filter are positioned at different horizontal locations within the visual data; and
a vertical butterfly filter, wherein the plurality of pairs of elements processed by the vertical butterfly filter are positioned at different vertical locations within the visual data.
12. The apparatus of claim 10 , wherein the plurality of elements associated with each block comprises a plurality of transform coefficients when the visual data is compressed.
13. The apparatus of claim 12 , wherein the plurality of transform coefficients comprises:
a plurality of discrete cosine transform (DCT) coefficients;
a plurality of wavelet transform coefficients; or
a plurality of integer transform coefficients.
14. A system, comprising:
one or more sensors to capture a visual representation of an environment; and
one or more processing devices to:
obtain visual data associated with the visual representation captured by the one or more sensors, wherein the visual data comprises uncompressed visual data or compressed visual data;
process the visual data using a convolutional neural network (CNN), wherein the CNN comprises a plurality of layers, the plurality of layers comprises a plurality of filters, and the plurality of filters comprises one or more pixel-domain filters and one or more compressed-domain filters, wherein:
the one or more pixel-domain filters are for performing processing associated with uncompressed data;
the one or more compressed-domain filters are for performing processing associated with compressed data; and
the CNN is trained to rely on processing associated with the one or more pixel-domain filters when the visual data is uncompressed, and the CNN is trained to rely on processing associated with the one or more compressed-domain filters when the visual data is compressed; and
classify the visual data based on an output of the CNN.
15. The system of claim 14 , wherein the one or more compressed-domain filters are further to perform processing associated with noncontiguous elements of the visual data, wherein there is a correlation among the noncontiguous elements when the visual data is compressed.
16. The system of claim 14 , wherein the one or more compressed-domain filters comprise one or more butterfly filters, wherein each butterfly filter is to perform a partial inverse transform associated with the visual data.
17. The system of claim 14 , wherein:
the one or more compressed-domain filters comprise one or more three-dimensional (3D) convolution filters to perform processing associated with three-dimensional (3D) visual data, wherein the 3D visual data is generated based on a three-dimensional (3D) transformation of the visual data, wherein the 3D transformation rearranges the visual data into three dimensions such that there is a correlation among one or more adjacent elements within a particular dimension when the visual data used for the 3D transformation is compressed; and
the one or more processing devices are further to:
generate the 3D visual data, wherein the 3D visual data is generated by performing the 3D transformation on the visual data; and
provide the 3D visual data as input to the one or more 3D convolution filters.
18. The system of claim 16 , wherein each butterfly filter is further to compute a sum and a difference for a plurality of pairs of elements within the visual data, wherein each pair of elements is positioned at locations within the visual data that are a particular distance apart.
19. The system of claim 17 , wherein the one or more processing devices to generate the 3D visual data are further to:
partition the visual data into a plurality of blocks, wherein each block comprises a plurality of elements, and wherein the plurality of elements comprises a plurality of discrete cosine transform (DCT) coefficients when the visual data is compressed;
arrange the plurality of blocks along a first dimension and a second dimension of the 3D visual data; and
arrange the plurality of elements associated with each block along a third dimension of the 3D visual data.
20. At least one non-transitory machine accessible storage medium having instructions stored thereon, wherein the instructions, when executed on a machine, cause the machine to:
obtain visual data associated with a visual representation captured by one or more sensors, wherein the visual data comprises uncompressed visual data or compressed visual data;
process the visual data using a convolutional neural network (CNN), wherein the CNN comprises a plurality of layers, the plurality of layers comprises a plurality of filters, and the plurality of filters comprises one or more pixel-domain filters and one or more compressed-domain filters, wherein:
the one or more pixel-domain filters are for performing processing associated with uncompressed data;
the one or more compressed-domain filters are for performing processing associated with compressed data; and
the CNN is trained to rely on processing associated with the one or more pixel-domain filters when the visual data is uncompressed, and the CNN is trained to rely on processing associated with the one or more compressed-domain filters when the visual data is compressed; and
classify the visual data based on an output of the CNN.
21. The storage medium of claim 20 , wherein the one or more compressed-domain filters comprise one or more butterfly filters to perform one or more partial inverse transforms associated with the visual data.
22. The storage medium of claim 20 , wherein the one or more compressed-domain filters comprise one or more three-dimensional (3D) convolution filters to perform processing associated with three-dimensional (3D) visual data, wherein the 3D visual data is generated based on a three-dimensional (3D) transformation of the visual data, wherein the 3D transformation rearranges the visual data into three dimensions such that there is a correlation among one or more adjacent elements within a particular dimension when the visual data used for the 3D transformation is compressed.
23. A method, comprising:
obtaining visual data associated with a visual representation captured by one or more sensors, wherein the visual data comprises uncompressed visual data or compressed visual data;
processing the visual data using a convolutional neural network (CNN), wherein the CNN comprises a plurality of layers, the plurality of layers comprises a plurality of filters, and the plurality of filters comprises one or more pixel-domain filters and one or more compressed-domain filters, wherein:
the one or more pixel-domain filters are for performing processing associated with uncompressed data;
the one or more compressed-domain filters are for performing processing associated with compressed data; and
the CNN is trained to rely on processing associated with the one or more pixel-domain filters when the visual data is uncompressed, and the CNN is trained to rely on processing associated with the one or more compressed-domain filters when the visual data is compressed; and
classifying the visual data based on an output of the CNN.
24. The method of claim 23 , wherein the one or more compressed-domain filters comprise one or more butterfly filters to perform one or more partial inverse transforms associated with the visual data.
25. The method of claim 23 , wherein the one or more compressed-domain filters comprise one or more three-dimensional (3D) convolution filters to perform processing associated with three-dimensional (3D) visual data, wherein the 3D visual data is generated based on a three-dimensional (3D) transformation of the visual data, wherein the 3D transformation is to rearrange the visual data into three dimensions such that there is a correlation among one or more adjacent elements within a particular dimension when the visual data used for the 3D transformation is compressed.