Compressed video recognition
In one embodiment, an apparatus comprises a communication interface and a processor. The communication interface is to communicate with a plurality of devices. The processor is to: receive compressed data from a first device, wherein the compressed data is associated with visual data captured by sensor(s); perform a current stage of processing on the compressed data using a current CNN, wherein the current stage of processing corresponds to one of a plurality of processing stages associated with the visual data, and wherein the current CNN corresponds to one of a plurality of CNNs associated with the plurality of processing stages; obtain an output associated with the current stage of processing; determine, based on the output, whether processing associated with the visual data is complete; if the processing is complete, output a result associated with the visual data; if the processing is incomplete, transmit the compressed data to a second device.
1 . A device, comprising:
interface circuitry; and
processing circuitry to:
receive, via the interface circuitry, compressed video data, wherein the compressed video data comprises motion vectors and prediction residuals, wherein video data is encoded in the compressed video data using the motion vectors and the prediction residuals;
extract a plurality of features from the compressed video data, wherein at least some of the features are based on the motion vectors and at least some of the features are based on the prediction residuals; and
recognize content in the video data using an artificial neural network, wherein the artificial neural network is trained to recognize the content based on the plurality of features, wherein the artificial neural network comprises a plurality of convolutional neural networks (CNNs), wherein one or more of the CNN are trained to recognize the content in the video data based on at least a subset of the features, and wherein one or more of the CNNs are trained to recognize the content in the video data based on pixel values in the video data,
wherein the plurality of CNNs are configured as a cascaded neural network having multiple decision stages,
wherein a first decision stage uses motion vectors to attempt an early decision,
wherein a subsequent decision stage uses prediction residuals if the first decision stage is unsuccessful, and
wherein a final decision stage uses decompressed video data if the subsequent decision stage is unsuccessful.
2 . The device of claim 1 , wherein:
the compressed video data further comprises transform coefficients, quantization parameters, and macroblock coding modes; and
at least some of the features are based on the transform coefficients, the quantization parameters, or the macroblock coding modes.
3 . The device of claim 1 , wherein the processing circuitry to recognize the content in the video data using the artificial neural network is further to:
recognize the content in the video data using at least some of the plurality of CNNs, wherein the CNNs are used successively until the content in the video data is recognized.
4 . The device of claim 1 , wherein:
the motion vectors represent motion that occurs between frames of the video data; and
the prediction residuals represent differences between the frames of the video data.
5 . The device of claim 1 , wherein the processing circuitry to receive, via the interface circuitry, the compressed video data is further to:
receive the compressed video data over a network; or
receive the compressed video data from a camera.
6 . At least one non-transitory computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed on processing circuitry,
cause the processing circuitry to:
receive compressed video data, wherein the compressed video data comprises motion vectors and prediction residuals, wherein video data is encoded in the compressed video data based at least in part on the motion vectors and the prediction residuals; and
recognize content in the video data using an artificial neural network, wherein the artificial neural network is trained to recognize the content based at least in part on the motion vectors and the prediction residuals,
wherein the artificial neural network comprises a plurality of convolutional neural networks (CNNs), wherein one or more of the CNN are trained to recognize the content in the video data based on at least a subset of the features, and wherein one or more of the CNNs are trained to recognize the content in the video data based on pixel values in the video data,
wherein the plurality of CNNs are configured as a cascaded neural network having multiple decision stages,
wherein a first decision stage uses motion vectors to attempt an early decision,
wherein a subsequent decision stage uses prediction residuals if the first decision stage is unsuccessful, and
wherein a final decision stage uses decompressed video data if the subsequent decision stage is unsuccessful.
7 . The storage medium of claim 6 , wherein the instructions that cause the processing circuitry to recognize the content in the video data using the artificial neural network further cause the processing circuitry to:
extract a plurality of features from the compressed video data, wherein at least some of the features are based on the motion vectors and at least some of the features are based on the prediction residuals; and
recognize the content in the video data using the artificial neural network, wherein the artificial neural network is trained to recognize the content based on the plurality of features.
8 . The storage medium of claim 6 , wherein:
the compressed video data further comprises transform coefficients, quantization parameters, and macroblock coding modes; and
the artificial neural network is trained to recognize the content based at least in part on the transform coefficients, the quantization parameters, or the macroblock coding modes.
9 . The storage medium of claim 8 , wherein the transform coefficients comprise discrete cosine transform (DCT) coefficients.
10 . The storage medium of claim 6 , wherein the artificial neural network comprises one or more convolutional neural networks (CNNs), wherein at least one of the CNNs is trained to recognize the content in the video data based at least in part on the motion vectors or the prediction residuals.
11 . The storage medium of claim 10 , wherein the one or more CNNs comprise a plurality of CNNs, wherein at least one of the CNNs is trained to recognize the content in the video data based on pixel values in the video data.
12 . The storage medium of claim 11 , wherein the instructions that cause the processing circuitry to recognize the content in the video data using the artificial neural network further cause the processing circuitry to:
recognize the content in the video data using at least some of the plurality of CNNs, wherein the CNNs are used successively until the content in the video data is recognized.
13 . The storage medium of claim 6 , wherein:
the motion vectors represent motion that occurs between frames of the video data; and
the prediction residuals represent differences between the frames of the video data.
14 . The storage medium of claim 6 , wherein:
the compressed video data is received over a network; or
the compressed video data is received from a camera.
15 . The storage medium of claim 6 , wherein the video data is encoded in the compressed video data based on a motion-compensated video codec.
16 . The storage medium of claim 15 , wherein the motion-compensated video codec is an H.264 video codec.