IP Library › Granted Patent US 11,166,041
Granted Patent B2
US 11,166,041 · App. 16/457,802 · Granted Nov 2, 2021

Hybrid pixel-domain and compressed-domain video analytics framework

Inventors: Yiting Liao (Sunnyvale, CA); Yen-Kuang Chen (Palo Alto, CA); Shao-Wen Yang (San Jose, CA); Vallabhajosyula S. Somayazulu (Portland, OR); Srenivas Varadarajan (Bangalore, IN); Omesh Tickoo (Portland, OR); Ibrahima J. Ndiour (Portland, OR)
Assignee: Intel Corporation
H04N19/52G06K9/3241G06K9/6256G06N3/04H04N19/172H04N19/523
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,166,041
App. No.
16/457,802
Granted
Nov 2, 2021
Kind
B2
Abstract

In one embodiment, an apparatus comprises processing circuitry to: receive, via a communication interface, a compressed video stream captured by a camera, wherein the compressed video stream comprises: a first compressed frame; and a second compressed frame, wherein the second compressed frame is compressed based at least in part on the first compressed frame, and wherein the second compressed frame comprises a plurality of motion vectors; decompress the first compressed frame into a first decompressed frame; perform pixel-domain object detection to detect an object at a first position in the first decompressed frame; and perform compressed-domain object detection to detect the object at a second position in the second compressed frame, wherein the object is detected at the second position in the second compressed frame based on: the first position of the object in the first decompressed frame; and the plurality of motion vectors from the second compressed frame.

Claims (158)

1. An apparatus, comprising:

a communication interface; and

processing circuitry to:

receive, via the communication interface, a plurality of frames of a compressed video stream, wherein the plurality of frames is represented in a compressed domain, and wherein the plurality of frames comprises:

a first frame; and

a second frame compressed based at least in part on the first frame, wherein the second frame comprises a plurality of motion vectors representing an estimated motion of a plurality of blocks of the second frame relative to the first frame;

decompress the first frame from the compressed domain into a pixel domain;

perform object detection on the first frame in the pixel domain, wherein an object is detected at a first position in the first frame; and

perform object detection on the second frame in the compressed domain, wherein:

the plurality of motion vectors is extracted from the second frame;

a derivative motion vector is computed from the plurality of motion vectors, wherein the derivative motion vector represents an estimated motion of the object from the first frame to the second frame; and

a second position of the object in the second frame is predicted based on the first position of the object in the first frame and the derivative motion vector.

2. The apparatus of claim 1 , wherein the processing circuitry to perform object detection on the second frame in the compressed domain is further to:

extract the plurality of motion vectors from the second frame, wherein the plurality of motion vectors represents the estimated motion of the plurality of blocks in the second frame that are collocated with the first position of the object in the first frame;

compute, based on the plurality of motion vectors, the derivative motion vector representing the estimated motion of the object from the first frame to the second frame; and

predict the second position of the object in the second frame based on:

the first position of the object in the first frame; and

the derivative motion vector representing the estimated motion of the object from the first frame to the second frame.

3. The apparatus of claim 1 , wherein the processing circuitry to perform object detection on the second frame in the compressed domain is further to:

generate a compressed-domain input for an artificial neural network, wherein the compressed-domain input comprises:

a first channel comprising object bounding box information for the first frame, wherein the object bounding box information indicates the first position of the object in the first frame;

a second channel comprising magnitude information for the plurality of motion vectors; and

a third channel comprising angle information for the plurality of motion vectors; and

process the compressed-domain input using the artificial neural network, wherein the artificial neural network is trained to perform object detection based on the compressed-domain input.

4. The apparatus of claim 1 , wherein:

the plurality of frames further comprises a third frame, wherein the third frame is compressed based at least in part on the first frame, and wherein the third frame comprises a second plurality of motion vectors; and

the processing circuitry is further to perform object detection on the third frame in the compressed domain, wherein:

the second plurality of motion vectors is extracted from the third frame;

an estimated motion of the object from the first frame to the third frame is computed using the second plurality of motion vectors; and

a third position of the object in the third frame is predicted based on the first position of the object in the first frame and the estimated motion of the object from the first frame to the third frame.

5. The apparatus of claim 1 , wherein:

the plurality of frames further comprises a third frame, wherein the third frame is compressed based at least in part on the second frame, and wherein the third frame comprises a second plurality of motion vectors; and

the processing circuitry is further to perform object detection on the third frame in the compressed domain, wherein:

the second plurality of motion vectors is extracted from the third frame;

an estimated motion of the object from the second frame to the third frame is computed using the second plurality of motion vectors; and

a third position of the object in the third frame is predicted based on the second position of the object in the second frame and the estimated motion of the object from the second frame to the third frame.

6. The apparatus of claim 1 , wherein:

the first frame is compressed based at least in part on a background frame;

the processing circuitry to decompress the first frame from the compressed domain into the pixel domain is further to decompress the first frame based at least in part on the background frame; and

the processing circuitry to perform object detection on the first frame in the pixel domain is further to:

determine a frame difference between the first frame and the background frame; and

detect the object at the first position in the first frame based on the frame difference.

7. The apparatus of claim 1 , wherein the processing circuitry to perform object detection on the first frame in the pixel domain is further to:

process the first frame in the pixel domain using a convolutional neural network, wherein the convolutional neural network is trained to perform object detection in the pixel domain.

8. The apparatus of claim 1 , wherein the second frame comprises a motion-predicted frame, wherein:

the motion-predicted frame is compressed based at least in part on the first frame;

the motion-predicted frame comprises the plurality of motion vectors; and

the motion-predicted frame does not comprise a prediction residual indicating a difference between the second frame and the first frame.

9. At least one non-transitory machine accessible storage medium having instructions stored thereon, wherein the instructions, when executed on a machine, cause the machine to:

receive, via a communication interface, a plurality of frames of a compressed video stream, wherein the plurality of frames is represented in a compressed domain, and wherein the plurality of frames comprises:

a first frame; and

a second frame compressed based at least in part on the first frame, wherein the second frame comprises a plurality of motion vectors representing an estimated motion of a plurality of blocks of the second frame relative to the first frame;

decompress the first frame from the compressed domain into a pixel domain;

perform object detection on the first frame in the pixel domain, wherein an object is detected at a first position in the first frame; and

perform object detection on the second frame in the compressed domain, wherein:

the plurality of motion vectors is extracted from the second frame;

a derivative motion vector is computed from the plurality of motion vectors, wherein the derivative motion vector represents an estimated motion of the object from the first frame to the second frame; and

a second position of the object in the second frame is predicted based on the first position of the object in the first frame and the derivative motion vector.

10. The storage medium of claim 9 , wherein the instructions that cause the machine to perform object detection on the second frame in the compressed domain further cause the machine to:

extract the plurality of motion vectors from the second frame, wherein the plurality of motion vectors represents the estimated motion of the plurality of blocks in the second frame that are collocated with the first position of the object in the first frame;

compute, based on the plurality of motion vectors, the derivative motion vector representing the estimated motion of the object from the first frame to the second frame; and

predict the second position of the object in the second frame based on:

the first position of the object in the first frame; and

the derivative motion vector representing the estimated motion of the object from the first frame to the second frame.

11. The storage medium of claim 9 , wherein the instructions that cause the machine to perform object detection on the second frame in the compressed domain further cause the machine to:

generate a compressed-domain input for an artificial neural network, wherein the compressed-domain input comprises:

a first channel comprising object bounding box information for the first frame, wherein the object bounding box information indicates the first position of the object in the first frame;

a second channel comprising magnitude information for the plurality of motion vectors; and

a third channel comprising angle information for the plurality of motion vectors; and

process the compressed-domain input using the artificial neural network, wherein the artificial neural network is trained to perform object detection based on the compressed-domain input.

12. The storage medium of claim 9 , wherein:

the plurality of frames further comprises a third frame, wherein the third frame is compressed based at least in part on the first frame, and wherein the third frame comprises a second plurality of motion vectors; and

the instructions further cause the machine to perform object detection on the third frame in the compressed domain, wherein:

the second plurality of motion vectors is extracted from the third frame;

an estimated motion of the object from the first frame to the third frame is computed using the second plurality of motion vectors; and

a third position of the object in the third frame is predicted based on the first position of the object in the first frame and the estimated motion of the object from the first frame to the third frame.

13. The storage medium of claim 9 , wherein:

the plurality of frames further comprises a third frame, wherein the third frame is compressed based at least in part on the second frame, and wherein the third frame comprises a second plurality of motion vectors; and

the instructions further cause the machine to perform object detection on the third frame in the compressed domain, wherein:

the second plurality of motion vectors is extracted from the third frame;

an estimated motion of the object from the second frame to the third frame is computed using the second plurality of motion vectors; and

a third position of the object in the third frame is predicted based on the second position of the object in the second frame and the estimated motion of the object from the second frame to the third frame.

14. The storage medium of claim 9 , wherein:

the first frame is compressed based at least in part on a background frame;

the instructions that cause the machine to decompress the first frame from the compressed domain into the pixel domain further cause the machine to decompress the first frame based at least in part on the background frame; and

the instructions that cause the machine to perform object detection on the first frame in the pixel domain further cause the machine to:

determine a frame difference between the first frame and the background frame; and

detect the object at the first position in the first frame based on the frame difference.

15. The storage medium of claim 9 , wherein the instructions that cause the machine to perform object detection on the first frame in the pixel domain further cause the machine to:

process the first frame in the pixel domain using a convolutional neural network, wherein the convolutional neural network is trained to perform object detection in the pixel domain.

16. The storage medium of claim 9 , wherein the second frame comprises a motion-predicted frame, wherein:

the motion-predicted frame is compressed based at least in part on the first frame;

the motion-predicted frame comprises the plurality of motion vectors; and

the motion-predicted frame does not comprise a prediction residual indicating a difference between the second frame and the first frame.

17. A method, comprising:

receiving, via a communication interface, a plurality of frames of a compressed video stream, wherein the plurality of frames is represented in a compressed domain, and wherein the plurality of frames comprises:

a first frame; and

a second frame compressed based at least in part on the first frame, wherein the second frame comprises a plurality of motion vectors representing an estimated motion of a plurality of blocks of the second frame relative to the first frame;

decompressing the first frame from the compressed domain into a pixel domain;

performing object detection on the first frame in the pixel domain, wherein an object is detected at a first position in the first frame; and

performing object detection on the second frame in the compressed domain, wherein:

the plurality of motion vectors is extracted from the second frame;

a derivative motion vector is computed from the plurality of motion vectors, wherein the derivative motion vector represents an estimated motion of the object from the first frame to the second frame; and

a second position of the object in the second frame is predicted based on the first position of the object in the first frame and the derivative motion vector.

18. The method of claim 17 , wherein performing object detection on the second frame in the compressed domain comprises:

extracting the plurality of motion vectors from the second frame, wherein the plurality of motion vectors represents the estimated motion of the plurality of blocks in the second frame that are collocated with the first position of the object in the first frame;

computing, based on the plurality of motion vectors, the derivative motion vector representing the estimated motion of the object from the first frame to the second frame; and

predicting the second position of the object in the second frame based on:

the first position of the object in the first frame; and

the derivative motion vector representing the estimated motion of the object from the first frame to the second frame.

19. The method of claim 17 , wherein performing object detection on the second frame in the compressed domain comprises:

generating a compressed-domain input for an artificial neural network, wherein the compressed-domain input comprises:

a first channel comprising object bounding box information for the first frame, wherein the object bounding box information indicates the first position of the object in the first frame;

a second channel comprising magnitude information for the plurality of motion vectors; and

a third channel comprising angle information for the plurality of motion vectors; and

processing the compressed-domain input using the artificial neural network, wherein the artificial neural network is trained to perform object detection based on the compressed-domain input.

20. The method of claim 17 , wherein:

the first frame is compressed based at least in part on a background frame;

decompressing the first frame from the compressed domain into the pixel domain comprises decompressing the first frame based at least in part on the background frame; and

performing object detection on the first frame in the pixel domain comprises:

determining a frame difference between the first frame and the background frame; and

detecting the object at the first position in the first frame based on the frame difference.

21. The method of claim 17 , wherein the second frame comprises a motion-predicted frame, wherein:

the motion-predicted frame is compressed based at least in part on the first frame;

the motion-predicted frame comprises the plurality of motion vectors; and

the motion-predicted frame does not comprise a prediction residual indicating a difference between the second frame and the first frame.

22. A system, comprising:

a camera; and

processing circuitry to:

receive a plurality of frames of a compressed video stream captured by the camera, wherein the plurality of frames is represented in a compressed domain, and wherein the plurality of frames comprises:

a first frame; and

a second frame compressed based at least in part on the first frame, wherein the second frame comprises a plurality of motion vectors representing an estimated motion of a plurality of blocks of the second frame relative to the first frame;

decompress the first frame from the compressed domain into a pixel domain;

perform object detection on the first frame in the pixel domain, wherein an object is detected at a first position in the first frame; and

perform object detection on the second frame in the compressed domain, wherein:

the plurality of motion vectors is extracted from the second frame;

a derivative motion vector is computed from the plurality of motion vectors, wherein the derivative motion vector represents an estimated motion of the object from the first frame to the second frame; and

a second position of the object in the second frame is predicted based on the first position of the object in the first frame and the derivative motion vector.

23. The system of claim 22 , wherein the processing circuitry to perform object detection on the second frame in the compressed domain is further to:

extract the plurality of motion vectors from the second frame, wherein the plurality of motion vectors represent the estimated motion of the plurality of blocks in the second frame that are collocated with the first position of the object in the first frame;

compute, based on the plurality of motion vectors, the derivative motion vector representing the estimated motion of the object from the first frame to the second frame; and

predict the second position of the object in the second frame based on:

the first position of the object in the first frame; and

the derivative motion vector representing the estimated motion of the object from the first frame to the second frame.

24. The system of claim 22 , wherein the processing circuitry to perform object detection on the second frame in the compressed domain is further to:

generate a compressed-domain input for an artificial neural network, wherein the compressed-domain input comprises:

a first channel comprising object bounding box information for the first frame, wherein the object bounding box information indicates the first position of the object in the first frame;

a second channel comprising magnitude information for the plurality of motion vectors; and

a third channel comprising angle information for the plurality of motion vectors; and

process the compressed-domain input using the artificial neural network, wherein the artificial neural network is trained to perform object detection based on the compressed-domain input.

25. The system of claim 22 , wherein:

the first frame is compressed based at least in part on a background frame;

the processing circuitry to decompress the first frame from the compressed domain into a pixel domain is further to decompress the first frame based at least in part on the background frame; and

the processing circuitry to perform object detection on the first frame in the pixel domain is further to:

determine a frame difference between the first frame and the background frame; and

detect the object at the first position in the first frame based on the frame difference.

26. The apparatus of claim 2 , wherein the processing circuitry to compute, based on the plurality of motion vectors, the derivative motion vector representing the estimated motion of the object from the first frame to the second frame is further to:

compute the derivative motion vector based on an average of the plurality of motion vectors.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 19, 2019
From: LIAO, YITING; CHEN, YEN-KUANG; YANG, SHAO-WEN; SOMAYAZULU, VALLABHAJOSYULA S.; VARADARAJAN, SRENIVAS; TICKOO, OMESH; NDIOUR, IBRAHIMA J.
To: INTEL CORPORATION
Reel/Frame 050093/0352 →
Continuity (1)
Related Publication 20190327486A1 · Oct 24, 2019