Multi-domain convolutional neural network
In one embodiment, an apparatus comprises a memory and a processor. The memory is to store visual data associated with a visual representation captured by one or more sensors. The processor is to: obtain the visual data associated with the visual representation captured by the one or more sensors, wherein the visual data comprises uncompressed visual data or compressed visual data; process the visual data using a convolutional neural network (CNN), wherein the CNN comprises a plurality of layers, wherein the plurality of layers comprises a plurality of filters, and wherein the plurality of filters comprises one or more pixel-domain filters to perform processing associated with uncompressed data and one or more compressed-domain filters to perform processing associated with compressed data; and classify the visual data based on an output of the CNN.
1. A computing device to perform content detection on compressed images, comprising:
interface circuitry; and
processing circuitry to:
receive, via the interface circuitry, a compressed image, wherein the compressed image represents an image in a compressed format;
extract a set of transform coefficients from the compressed image, wherein the set of transform coefficients represents a transform performed on a plurality of blocks of the image, wherein the set of transform coefficients comprises a corresponding subset of transform coefficients for each block of the plurality of blocks;
generate a three-dimensional (3D) data structure to represent the set of transform coefficients, wherein a first dimension and a second dimension of the 3D data structure represent the plurality of blocks, and wherein a third dimension of the 3D data structure represents the corresponding subset of transform coefficients for each block; and
detect image content captured in the compressed image, wherein the image content is detected based on analyzing the 3D data structure using a convolutional neural network (CNN).
2. The computing device of claim 1 , wherein the first dimension and the second dimension of the 3D data structure further represent spatial indices of the plurality of blocks within the image.
3. The computing device of claim 1 , wherein the third dimension of the 3D data structure further represents a plurality of channels, wherein each channel represents a corresponding group of correlated transform coefficients among the plurality of blocks, wherein the corresponding group of correlated transform coefficients is from the set of transform coefficients.
4. The computing device of claim 3 , wherein:
the transform is a discrete cosine transform (DCT);
the set of transform coefficients is a set of DCT coefficients; and
the plurality of channels comprises:
a first channel representing a group of DC coefficients among the plurality of blocks, wherein the group of DC coefficients is from the set of DCT coefficients; and
a second channel representing a group of AC coefficients among the plurality of blocks, wherein the group of AC coefficients is from the set of DCT coefficients.
5. The computing device of claim 1 , wherein:
the image is captured by a camera; and
the processing circuitry to receive, via the interface circuitry, the compressed image is further to receive the compressed image over a network from a device associated with the camera.
6. The computing device of claim 1 :
further comprising:
a camera to capture the image; and
image processing circuitry to compress the image into the compressed image; and
wherein the processing circuitry to receive, via the interface circuitry, the compressed image is further to receive the compressed image from the image processing circuitry.
7. The computing device of claim 1 , wherein:
the computing device is an edge processing device for performing visual analytics; or
the computing device is a smart camera.
8. The computing device of claim 1 , wherein the CNN is to detect the image content based on analyzing the 3D data structure using one or more 3D convolution filters, wherein the one or more 3D convolution filters are trained to perform content detection based on transform coefficients from compressed images.
9. The computing device of claim 1 , wherein:
the transform is a discrete cosine transform (DCT); and
the set of transform coefficients is a set of DCT coefficients.
10. The computing device of claim 1 , wherein:
the transform is a wavelet transform; and
the set of transform coefficients is a set of wavelet transform coefficients.
11. The computing device of claim 1 , wherein:
the transform is an integer transform; and
the set of transform coefficients is a set of integer transform coefficients.
12. At least one non-transitory computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed on processing circuitry, cause the processing circuitry to:
receive, via interface circuitry, a compressed image, wherein the compressed image represents an image in a compressed format;
extract a set of transform coefficients from the compressed image, wherein the set of transform coefficients represents a transform performed on a plurality of blocks of the image, wherein the set of transform coefficients comprises a corresponding subset of transform coefficients for each block of the plurality of blocks;
generate a three-dimensional (3D) data structure to represent the set of transform coefficients, wherein a first dimension and a second dimension of the 3D data structure represent the plurality of blocks, and wherein a third dimension of the 3D data structure represents the corresponding subset of transform coefficients for each block; and
detect image content captured in the compressed image, wherein the image content is detected based on analyzing the 3D data structure using a convolutional neural network (CNN).
13. The storage medium of claim 12 , wherein:
the first dimension and the second dimension of the 3D data structure further represent spatial indices of the plurality of blocks within the image; and
the third dimension of the 3D data structure further represents a plurality of channels, wherein each channel represents a corresponding group of correlated transform coefficients among the plurality of blocks, wherein the corresponding group of correlated transform coefficients is from the set of transform coefficients.
14. The storage medium of claim 13 , wherein:
the transform is a discrete cosine transform (DCT);
the set of transform coefficients is a set of DCT coefficients; and
the plurality of channels comprises:
a first channel representing a group of DC coefficients among the plurality of blocks, wherein the group of DC coefficients is from the set of DCT coefficients; and
a second channel representing a group of AC coefficients among the plurality of blocks, wherein the group of AC coefficients is from the set of DCT coefficients.
15. The storage medium of claim 12 , wherein:
the image is captured by a camera; and
the instructions that cause the processing circuitry to receive, via the interface circuitry, the compressed image further cause the processing circuitry to receive the compressed image over a network from a device associated with the camera.
16. The storage medium of claim 12 , wherein the CNN is to detect the image content based on analyzing the 3D data structure using one or more 3D convolution filters, wherein the one or more 3D convolution filters are trained to perform content detection based on transform coefficients from compressed images.
17. The storage medium of claim 12 , wherein:
the transform is a discrete cosine transform (DCT); and
the set of transform coefficients is a set of DCT coefficients.
18. The storage medium of claim 12 , wherein:
the transform is a wavelet transform; and
the set of transform coefficients is a set of wavelet transform coefficients.
19. The storage medium of claim 12 , wherein:
the transform is an integer transform; and
the set of transform coefficients is a set of integer transform coefficients.
20. A method of performing content detection on compressed images, comprising:
receiving, via interface circuitry, a compressed image, wherein the compressed image represents an image in a compressed format;
extracting a set of transform coefficients from the compressed image, wherein the set of transform coefficients represents a transform performed on a plurality of blocks of the image, wherein the set of transform coefficients comprises a corresponding subset of transform coefficients for each block of the plurality of blocks;
generating a three-dimensional (3D) data structure to represent the set of transform coefficients, wherein a first dimension and a second dimension of the 3D data structure represent the plurality of blocks, and wherein a third dimension of the 3D data structure represents the corresponding subset of transform coefficients for each block; and
detecting image content captured in the compressed image, wherein the image content is detected based on analyzing the 3D data structure using a convolutional neural network (CNN).
21. The method of claim 20 , wherein:
the first dimension and the second dimension of the 3D data structure further represent spatial indices of the plurality of blocks within the image; and
the third dimension of the 3D data structure further represents a plurality of channels, wherein each channel represents a corresponding group of correlated transform coefficients among the plurality of blocks, wherein the corresponding group of correlated transform coefficients is from the set of transform coefficients.
22. The method of claim 21 , wherein:
the transform is a discrete cosine transform (DCT);
the set of transform coefficients is a set of DCT coefficients; and
the plurality of channels comprises:
a first channel representing a group of DC coefficients among the plurality of blocks, wherein the group of DC coefficients is from the set of DCT coefficients; and
a second channel representing a group of AC coefficients among the plurality of blocks, wherein the group of AC coefficients is from the set of DCT coefficients.
23. The method of claim 20 , wherein the CNN is to detect the image content based on analyzing the 3D data structure using one or more 3D convolution filters, wherein the one or more 3D convolution filters are trained to perform content detection based on transform coefficients from compressed images.
24. A system for performing content detection on compressed images, comprising:
a camera to capture an image;
image processing circuitry to compress the image into a compressed image; and
processing circuitry to:
receive the compressed image from the image processing circuitry;
extract a set of transform coefficients from the compressed image, wherein the set of transform coefficients represents a transform performed on a plurality of blocks of the image, wherein the set of transform coefficients comprises a corresponding subset of transform coefficients for each block of the plurality of blocks;
generate a three-dimensional (3D) data structure to represent the set of transform coefficients, wherein a first dimension and a second dimension of the 3D data structure represent the plurality of blocks, and wherein a third dimension of the 3D data structure represents the corresponding subset of transform coefficients for each block; and
detect image content captured in the compressed image, wherein the image content is detected based on analyzing the 3D data structure using a convolutional neural network (CNN).
25. The system of claim 24 , wherein the processing circuitry to detect the image content captured in the compressed image comprises:
artificial intelligence (AI) accelerator circuitry, wherein the AI accelerator circuitry is to detect the image content captured in the compressed image based on analyzing the 3D data structure using the CNN.