IP Library Granted Patent US 10,777,229
Granted Patent B2
US 10,777,229 · App. 16/505,046 · Granted Sep 15, 2020

Generating moving thumbnails for videos

Inventors: Weilong Yang (Fremont, CA); Min-hsuan Tsai (Cupertino, CA); Zheng Sun (Sunnyvale, CA); Pei Cao (Palo Alto, CA); Tomas Izo (Mountain View, CA)
Assignee: Google LLC
G11B27/034G06F16/739G06T7/20G06T7/97G11B27/031G11B27/105G11B27/28G11B27/326H04N17/004H04N21/23418H04N21/8549G06F40/169G06K9/00765G06T2207/10016H04N5/225
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,777,229
App. No.
16/505,046
Granted
Sep 15, 2020
Kind
B2
Abstract

Frame-level quality scores for video frames of a video item is determined. A sliding window is applied to the video frames to identify a plurality of groups of the video frames for scoring on a group-level. A plurality of group-level quality scores for the plurality of groups of video frames of the video item is determined using the frame-level quality scores of the video frames. One of the plurality of groups of video frames of the video item is selected based on the plurality of group-level quality scores. A moving thumbnail is created using the selected group of video frames selected based on the respective group level quality score.

Claims (59)

1. A method comprising:

determining frame-level quality scores for video frames of a video item;

applying a sliding window to the video frames to identify a plurality of groups of video frames for scoring on a group-level, wherein the sliding window groups the video frames into the plurality of groups of video frames, wherein at least two of the plurality of groups of video frames comprise a same video frame, and wherein the plurality of groups of video frames are scored on the group-level to provide an indication of a quality of a respective group of video frames for inclusion in a moving thumbnail;

determining, by a processing device, a plurality of group-level quality scores for the plurality of groups of video frames of the video item using the frame-level quality scores of the video frames;

selecting one of the plurality of groups of video frames of the video item based on the plurality of group-level quality scores; and

creating the moving thumbnail using the selected group of video frames selected based on the respective group level quality score.

2. The method of claim 1 , further comprising:

sampling the video frames of the video item at a fixed sampling rate.

3. The method of claim 2 , wherein the video frames of the video item are sampled from a beginning portion of the video item.

4. The method of claim 1 , wherein determining the frame-level quality scores for the video frames comprises:

generating one or more quality scores for each of the video frames in view of one or more of a plurality of frame-level scorers; and

aggregating the one or more quality scores for each of the video frames to generate the frame-level quality scores for the video frames, wherein the frame-level quality score for a respective one of the video frames is indicative of a quality of the respective video frame of the video item.

5. The method of claim 4 , wherein generating one or more quality scores for each of the video frames in view of one or more of a plurality of frame-level scorers comprises:

generating a first quality score for each of the video frames using a thumbnail quality scorer that measures a visual quality of the video frames;

generating a second quality score for each of the video frames using a face scorer that measures a size of a face relative to a size of the video frames; and

generating a third quality score for each of the video frames using a motion scorer that measures motion between two consecutively sampled video frames.

6. The method of claim 4 , wherein aggregating the quality scores for each the video frames comprises summing the quality scores using a linear weighted summation.

7. The method of claim 1 , wherein applying the sliding window to the video frames to identify the plurality of groups of video frames for scoring on the group-level further comprises:

applying the sliding window to different positions along a time axis to group the video frames into the plurality of groups of video frames, wherein the video frames are ordered with respect to the time axis, and wherein frame-level quality scores of a particular group of video frames are used to determine a group-level quality score for the particular group of video frames.

8. The method of claim 1 , wherein applying the sliding window to the video frames to identify the plurality of groups of video frames for scoring on the group-level further comprises:

applying the sliding window to a first position with respect to a time axis, wherein the sliding window extends for a fixed duration and a first group of the plurality of groups of video frames is within the sliding window at the first position with respect to the time axis; and

applying the sliding window to a second position with respect to the time axis, wherein a second group of the plurality of groups of video frames is within the sliding window at the second position with respect to the time axis; and

wherein determining the plurality of group-level quality scores for the plurality of groups of video frames using the frame-level quality scores of the video frames further comprises:

determining a first group-level quality score of the plurality of group-level quality scores by aggregating frame-level quality scores associated with the first group of video frames; and

determining a second group-level quality score of the plurality of group-level quality scores by aggregating frame-level quality scores associated with the second group of video frames.

9. The method of claim 1 , wherein the selected group of video frames has a highest group-level quality score among the plurality of group-level quality scores.

10. The method of claim 1 , wherein the moving thumbnail comprises a subset of the video frames that have timestamps within a range from a start timestamp to an end timestamp, the method further comprising:

modifying the subset of the video frames of the moving thumbnail to cause at least one of a first video frame of the subset of the video frames or a last video frame of the subset of the video frames to be proximate a shot boundary of the video item.

11. The method of claim 1 , further comprising modifying a playback speed of the moving thumbnail in view of motion measured in a subset of the video frames of the moving thumbnail.

12. The method of claim 1 , wherein the moving thumbnail is stored in a Graphic Interchange Format (GIF).

13. The method of claim 1 , wherein the moving thumbnail is stored in an H.264 format.

14. A non-transitory computer-readable medium storing instructions that, responsive to execution by a processing device, cause the processing device to perform operations comprising:

determining frame-level quality scores for video frames of a video item;

applying a sliding window to the video frames to identify a plurality of groups of video frames for scoring on a group-level, wherein the sliding window groups the video frames into the plurality of groups of video frames, wherein at least two of the plurality of groups of video frames comprise a same video frame, and wherein the plurality of groups of video frames are scored on the group-level to provide an indication of a quality of a respective group of video frames for inclusion in a moving thumbnail;

determining, by a processing device, a plurality of group-level quality scores for the plurality of groups of video frames of the video item using the frame-level quality scores of the video frames;

selecting one of the plurality of groups of video frames of the video item based on the plurality of group-level quality scores; and

creating the moving thumbnail using the selected group of video frames selected based on the respective group level quality score.

15. The non-transitory computer-readable medium of claim 14 , wherein determining the frame-level quality scores for the video frames comprises:

generating one or more quality scores for each of the video frames in view of one or more of a plurality of frame-level scorers; and

aggregating the one or more quality scores for each of the video frames to generate the frame-level quality scores for the video frames, wherein the frame-level quality score for a respective one of the video frames is indicative of a quality of the respective video frame of the video item.

16. The non-transitory computer-readable medium of claim 14 , wherein applying the sliding window to the video frames to identify the plurality of groups of video frames for scoring on the group-level further comprises:

applying the sliding window to different positions along a time axis to group the video frames into the plurality of groups of video frames, wherein the video frames are ordered with respect to the time axis, and wherein frame-level quality scores of a particular group of video frames are used to determine a group-level quality score for the particular group of video frames.

17. The non-transitory computer-readable medium of claim 14 , wherein the moving thumbnail comprises a subset of the video frames that have timestamps within a range from a start timestamp to an end timestamp, the operations further comprising:

modifying the subset of the video frames of the moving thumbnail to cause at least one of a first video frame of the subset of the video frames or a last video frame of the subset of the video frames to be proximate a shot boundary of the video item; and

modifying a playback speed of the moving thumbnail in view of motion measured in the subset of the video frames of the moving thumbnail.

18. A system comprising:

a memory; and

a processing device, coupled to the memory, the processing device to:

determine frame-level quality scores for video frames of a video item;

apply a sliding window to the video frames to identify a plurality of groups of video frames for scoring on a group-level, wherein the sliding window groups the video frames into the plurality of groups of video frames, wherein at least two of the groups of video frames comprise a same video frame, and wherein the plurality of groups of video frames are scored on the group-level to provide an indication of a quality of a respective group of video frames for inclusion in a moving thumbnail;

determine a plurality of group-level quality scores for the plurality of groups of video frames of the video item using the frame-level quality scores of the video frames;

select one of the plurality of groups of video frames of the video item based on the plurality of group-level quality scores; and

create the moving thumbnail using the selected group of video frames selected based on the respective group level quality score.

19. The system of claim 18 , wherein to determine frame-level quality scores for the video frames, the processing device further to:

generate one or more quality scores for each of the video frames in view of one or more of a plurality of frame-level scorers; and

aggregate the one or more quality scores for each of the video frames to generate the frame-level quality scores for the video frames, wherein the frame-level quality score for a respective one of the video frames is indicative of a quality of the respective video frame of the video item.

20. The system of claim 18 , the processing device further to:

modify a subset of the video frames of the moving thumbnail to cause at least one of a first video frame of the subset of the video frames or a last video frame of the subset of the video frames to be proximate a shot boundary of the video item; and

modify a playback speed of the moving thumbnail in view of motion measured in the subset of the video frames of the moving thumbnail.

Assignments (3)
CHANGE OF NAME Recorded Oct 16, 2019
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 050741/0534 →
CHANGE OF NAME Recorded Oct 16, 2019
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 050741/0751 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 8, 2019
From: YANG, WEILONG; TSAI, MIN-HSUAN; SUNG, ZHENG; CAO, PEI; IZO, TOMAS
To: GOOGLE INC.
Reel/Frame 049691/0839 →
Continuity (2)
Continuation 15199269 · Jun 30, 2016
Related Publication 20190333538A1 · Oct 31, 2019