IP Library Granted Patent US 10,291,916
Granted Patent B2
US 10,291,916 · App. 15/784,516 · Granted May 14, 2019

Video content analysis and/or processing using encoding logs

Inventors: Yao-Chung Lin (Sunnyvale, CA); Sam John (Fremont, CA); Thierry Foucu (San Jose, CA); Sasi Inguva (Mountain View, CA)
Assignee: GOOGLE LLC
H04N19/134H04N19/00472H04N19/00587H04N19/00921H04N19/142H04N19/40H04N19/87
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,291,916
App. No.
15/784,516
Granted
May 14, 2019
Kind
B2
Abstract

Systems and methods for processing a video sequence are disclosed. In accordance with some implementations, The method includes determining a first set of shot-change locations associated with a cut transition in a video sequence based on variance data defined in an encoding log associated with at least one previously encoded video frame. The method further includes determining a second set of shot-change locations associated with a fading transition in the video sequence based on the variance data defined in the encoding log associated with the at least one previously encoded video frame. The method also includes processing the video sequence based on at least a portion of the first set of shot-change locations and the second set of shot-change locations.

Claims (75)

1. A system, comprising:

a memory storing computer executable components; and

one or more processors configured to execute the computer executable components stored in the memory, the computer executable components causing the one or more processors to:

determine a first set of shot-change locations associated with a cut transition in a video sequence based on variance data defined in an encoding log associated with at least one previously encoded video frame, wherein to determine the first set of shot-change locations associated with the cut transition comprises to:

determine a ratio of variance data associated with a given video frame and average variance data associated with a plurality of video frames corresponding to the given video frame, wherein the video frames of the plurality of video frames are temporally proximate to the given frame; and

when the ratio is greater than a first threshold, detect an impulse pattern indicative of the cut transition in the variance data associated with the given video frame;

determine a second set of shot-change locations associated with a fading transition in the video sequence based on the variance data defined in the encoding log associated with the at least one previously encoded video frame; and

process the video sequence based on at least a portion of the first set of shot-change locations and the second set of shot-change locations.

2. The system of claim 1 , wherein to determine the first set of shot-change locations associated with the cut transition further comprises to:

detect a step pattern in variance data defined in the encoding log.

3. The system of claim 2 , wherein detecting the step pattern in variance data defined in the encoding log comprises:

for a particular video frame:

determining a first difference value between the particular video frame and a first previous frame immediately preceding the particular video frame;

determining a second difference value between the first previous frame and a second previous frame immediately preceding the first previous frame;

determining a third difference value between the particular video frame and a following frame immediately following the particular video frame;

determining a ratio based on the first difference value and a sum of the second difference value and the third difference value; and

detecting a step when the ratio is greater than a second threshold, wherein the first difference value, the second difference value, and the third difference value are calculated from the variance data.

4. The system of claim 1 , wherein determining the second set of shot-change locations associated with the fading transition comprises:

detecting a ramp pattern in the variance data.

5. The system of claim 4 , wherein detecting the ramp pattern comprises:

identifying a plurality of consecutive video frames;

for each particular video frame in the plurality of consecutive video frames, determining respective variance data corresponding to the particular video frame;

determining a slope value based on the respective variance data corresponding to the particular video frame; and

detecting the ramp pattern corresponding to the consecutive video frames when the slope value is greater than a second threshold.

6. The system of claim 1 , wherein the variance data defined in the encoding log is motion compensated pixel variance data or pixel variance data included in the encoding log.

7. The system of claim 1 , wherein the computer executable components further cause the one or more processors to:

merge the first set of shot-change locations and the second set of shot-change locations to generate a set of potential shot-change locations; and

filter the set of potential shot-change locations based on other data included in the encoding log to generate a set of finalized shot change locations.

8. The system of claim 1 , wherein the encoding log is generated by a different transcoding component.

9. The system of claim 8 , wherein the different transcoding component transmits the video sequence to and the encoding log to the one or more processors.

10. A method, comprising:

determining, by one or more processors, a first set of shot-change locations associated with a cut transition in a video sequence based on variance data defined in an encoding log associated with at least one previously encoded video frame;

determining, by the one or more processors, a second set of shot-change locations associated with a fading transition in the video sequence based on the variance data defined in the encoding log associated with the at least one previously encoded video frame, wherein determining the second set of shot-change locations associated with the fading transition includes:

identifying a plurality of consecutive video frames;

for each given video frame in the plurality of consecutive video frames, determining respective variance data corresponding to the given video frame;

determining a slope value based on the respective variance data corresponding to the given video frame; and

detecting a ramp pattern, indicative of the fading transition, corresponding to the consecutive video frames when the slope value is greater than a first threshold; and

processing, by the one or more processors, the video sequence based on at least a portion of the first set of shot-change locations and the second set of shot-change locations.

11. The method of claim 10 , wherein determining the first set of shot-change locations associated with the cut transition comprises:

detecting an impulse pattern in variance data defined in the encoding log.

12. The method of claim 11 , wherein detecting the impulse pattern comprises:

determining a ratio of variance data associated with a particular video frame and average variance data associated with a plurality of video frames corresponding to the particular video frame, wherein the video frames of the plurality of video frames are temporally proximate to the particular frame; and

when the ratio is greater than a second threshold, detecting the impulse pattern in the variance data associated with the particular video frame.

13. The method of claim 10 , wherein determining the first set of shot-change locations associated with the cut transition comprises detecting a step pattern in variance data defined in the encoding log.

14. The method of claim 13 , wherein detecting the step pattern in variance data defined in the encoding log comprises:

for a particular video frame:

determining a first difference value between the particular video frame and a first previous frame immediately preceding the particular video frame;

determining a second difference value between the first previous frame and a second previous frame immediately preceding the first previous frame;

determining a third difference value between the particular video frame and a following frame immediately following the particular video frame;

determining a ratio based on the first difference value and a sum of the second difference value and the third difference value; and

detecting a step when the ratio is greater than a second threshold, wherein the first difference value, the second difference value, and the third difference value are calculated from the variance data.

15. The method of claim 10 , wherein the variance data defined in the encoding log is motion compensated pixel variance data or pixel variance data included in the encoding log.

16. The method of claim 10 , further comprising:

merging, by the one or more processors, the first set of shot-change locations and the second set of shot-change locations to generate a set of potential shot-change locations; and

filtering, by the one or more processors, the set of potential shot-change locations based on other data included in the encoding log to generate a set of finalized shot change locations.

17. A system, comprising:

a memory storing computer executable components; and

one or more processors configured to execute the computer executable components stored in the memory, the computer executable components causing the one or more processors to:

determine a first set of shot-change locations associated with a cut transition in a video sequence based on variance data defined in an encoding log associated with at least one previously encoded video frame, wherein to determine the first set of shot-change locations associated with the cut transition comprises to:

for a particular video frame:

determine a first difference value between the particular video frame and a first previous frame immediately preceding the particular video frame;

determine a second difference value between the first previous frame and a second previous frame immediately preceding the first previous frame;

determine a third difference value between the particular video frame and a following frame immediately following the particular video frame;

determine a ratio based on the first difference value and a sum of the second difference value and the third difference value; and

detect a step, indicative of the cut transition, when the ratio is greater than a first threshold, wherein the first difference value, the second difference value, and the third difference value are calculated from the variance data;

determine a second set of shot-change locations associated with a fading transition in the video sequence based on the variance data defined in the encoding log associated with the at least one previously encoded video frame; and

process the video sequence based on at least a portion of the first set of shot-change locations and the second set of shot-change locations.

18. The system of claim 17 , wherein to determine the first set of shot-change locations comprises to:

determine a ratio of variance data associated with a given video frame and average variance data associated with a plurality of video frames corresponding to the given video frame, wherein the video frames of the plurality of video frames are temporally proximate to the particular video frame; and

when the ratio is greater than a second threshold, detect an impulse pattern, indicative of a shot-change location, in the variance data associated with the given video frame.

19. The system of claim 17 , wherein to determine the second set of shot-change locations associated with the fading transition comprises to:

identify a plurality of consecutive video frames;

for each given video frame in the plurality of consecutive video frames, determine respective variance data corresponding to the given video frame;

determine a slope value based on the respective variance data corresponding to the video frame; and

detect a ramp pattern, indicative of the fading transition, corresponding to the consecutive video frames when the slope value is greater than a second threshold.

Assignments (2)
MERGER AND CHANGE OF NAME Recorded Jan 19, 2018
From: GOOGLE INC.; GOOGLE LLC
To: GOOGLE LLC
Reel/Frame 045101/0449 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 15, 2018
From: LIN, YAO-CHUNG; JOHN, SAM; FOUCU, THIERRY; INGUVA, SASI
To: GOOGLE INC.
Reel/Frame 044620/0613 →
Continuity (2)
Continuation 14093939 · Dec 2, 2013
Related Publication 20180084254A1 · Mar 22, 2018