IP Library Granted Patent US 7,212,666
Granted Patent B2
US 7,212,666 · App. 10/405,971 · Granted May 1, 2007

Generating visually representative video thumbnails

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 7,212,666
App. No.
10/405,971
Granted
May 1, 2007
Kind
B2
Abstract

An algorithm identifies a salient video frame from a video sequence for use as a video thumbnail. The identification of a video thumbnail is based on a frame goodness measure. The algorithm calculates a color histogram of a frame, and then calculates the entropy and standard deviation of the color histogram. The frame goodness measure is a weighted combination of the entropy and the standard deviation. A video frame having the highest value of frame goodness measure for a video sequence is determined as the video thumbnail for a video sequence.

Claims (134)

1. A tangible computer-readable medium comprising computer-program instructions executable by a processor, the computer-program instructions when executed by the processor per for performing operations comprising:

receiving a video sequence; and

calculating frame goodness measures for frames within the video sequence from histogram entropies and histogram standard deviations; and

presenting a visually representative thumbnail based on the frame goodness measures for a user to browse video data.

2. The computer-readable medium of claim 1 , comprising further processor-executable instructions configured for determining a maximum goodness measure from the frame goodness measures.

3. The computer-readable medium of claim 2 , wherein the computer-program instructions further comprise instructions for generating a video thumbnail from a frame that corresponds with the maximum goodness measure.

4. The computer-readable medium of claim 1 , wherein each frame goodness measure is a weighted combination of a histogram entropy and a standard deviation entropy.

5. The computer-readable medium of claim 1 , wherein the computer-program instructions further comprise instructions for calculating each histogram entropy.

6. The computer-readable medium of claim 5 , wherein the computer-program instructions for calculating each histogram entropy comprise instructions for calculating a color histogram.

7. A processor-readable medium as recited in claim 1 , wherein the computer-program instructions further comprise instructions for calculating each histogram standard deviation.

8. The computer-readable medium of claim 7 , wherein the computer-program instructions for calculating each histogram standard deviation further comprise instructions for calculating a color histogram.

9. The computer-readable medium of claim 1 , wherein the computer-program instructions for calculating a frame goodness measure comprise instructions for:

subdividing a frame into a plurality of blocks;

calculating a block goodness measure for each block;

determining a minimum block goodness measure; and

selecting the minimum block goodness measure as a frame goodness measure for the frame.

10. The computer-readable medium of claim 1 , wherein the computer-program instructions for calculating a frame goodness measure comprise instructions for:

calculating a color histogram;

calculating an entropy of the color histogram;

calculating a standard deviation of the color histogram; and

calculating the goodness measure as a weighted combination of the entropy and the standard deviation.

11. A tangible computer-readable medium comprising computer-program instructions executable by a processor, the computer-program instructions when executed by the processor for performing operations comprising:

temporally segmenting a video sequence;

calculating frame goodness measures within a particular temporal segment using histogram entropies and histogram standard deviations;

determining from the frame goodness measures, a local maximum goodness measure for the particular temporal segment; and

presenting a visually representative thumbnail based on the local maximum goodness measure for a user to browse video data.

12. The computer-readable medium of claim 11 , wherein the computer-program instructions further comprise instructions for generating a video thumbnail from a frame associated with the local maximum goodness measure.

13. The computer-readable medium of claim 11 wherein a frame goodness measure is calculated for every nth frame within the particular temporal segment and n is equal to or greater than 1.

14. The computer-readable medium of claim 11 , wherein each goodness measure is a weighted combination of a histogram entropy and a standard deviation entropy.

15. The computer-readable medium of claim 11 , wherein the particular temporal segment is selected from:

a beginning temporal segment of the video sequence;

a middle temporal segment of the video sequence; and

an ending temporal segment of the video sequence.

16. The computer-readable medium of claim 11 , wherein the computer-program instructions for calculating a frame goodness measure further comprise instructions for:

subdividing a frame into a plurality of blocks;

calculating a block goodness measure for each block;

determining a minimum block goodness measure; and

selecting the minimum block goodness measure as a frame goodness measure for the frame.

17. A tangible computer-readable medium comprising computer-program instructions executable by a processor, the computer-program instructions when executed by the processor for performing operations comprising:

segmenting a video sequence into a plurality of video shots;

calculating a color histogram of a video frame in a first video shot;

calculating an entropy of the color histogram;

calculating a standard deviation of the color histogram;

calculating a frame goodness measure as a weighted combination of the entropy and the standard deviation; and

presenting a visually representative thumbnail based on the frame goodness measure for a user to browse video data.

18. The computer-readable medium of claim 17 , wherein the computer-program instructions further comprise instructions for:

calculating additional frame goodness measures for additional video frames in the first video shot; and

determining a local maximum frame goodness measure for the first video shot by comparing frame goodness measures.

19. The computer-readable medium of claim 18 , wherein the computer-program instructions further comprise instructions for:

calculating additional local maximum frame goodness measures for additional video shots in the video sequence; and

determining a thumbnail video frame for the video sequence by comparing local maximum frame goodness measures.

20. The computer-readable medium of claim 19 , wherein the computer-program instructions for determining further comprise instructions for:

identifying a highest valued local maximum frame goodness measure; and

selecting a video frame corresponding to the highest valued local maximum frame goodness measure as the thumbnail video frame.

21. A tangible computer-readable medium comprising computer-program instructions executable by a processor, the computer-program instructions when executed by the processor for performing operations comprising:

segmenting a video sequence into a plurality of shots;

selecting a frame from each of the plurality of shots;

for each selected frame, calculating a frame goodness measure as a function of a histogram entropy and a histogram standard deviation;

determining a thumbnail frame from the selected frames as a function of the frame goodness measures; and

wherein the thumbnail frame is for presentation to a user to facilitate user browsing of video data.

22. The computer-readable medium of claim 21 , wherein the computer-program instructions for determining further comprise instructions for:

identifying a maximum frame goodness measure from the frame goodness measures; and

determining the thumbnail frame as a frame associated with the maximum frame goodness measure.

23. The computer-readable medium of claim 21 , wherein the computer-program instructions for calculating a goodness measure further comprise instructions for:

calculating a color histogram;

calculating an entropy of the color histogram;

calculating a standard deviation of the color histogram; and

calculating the goodness measure as a weighted combination of the entropy and the standard deviation.

24. The computer-readable medium of claim 21 , wherein the computer-program instructions for segmenting further comprise instructions for performing a shot boundary detection.

25. A computer-implemented method comprising:

calculating a color histogram of a video frame in a video sequence;

calculating an entropy of the color histogram;

calculating a standard deviation of the color histogram;

calculating a frame goodness measure as a weighted combination of the entropy and the standard; and

display in a visually representative video thumbnail based on the frame goodness measure for user browsing of video data.

26. The method of claim 25 , further comprising:

calculating additional frame goodness measures for additional video frames in the video sequence; and

determining a thumbnail video frame for the video sequence by comparing frame goodness measures.

27. The method of claim 26 , wherein the determining further comprises:

identifying a maximum frame goodness measure; and

selecting a video frame corresponding to the maximum frame goodness measure as the thumbnail video frame.

28. A computer-implemented method comprising:

temporally segmenting a video sequence;

calculating frame goodness measures within a particular temporal segment using histogram entropies and histogram standard deviations;

determining from the frame goodness measures, a local maximum goodness measure for the particular temporal segment; and

displaying a visually representative video thumbnail based on the local maximum goodness measure for user browsing of video data.

29. The method of claim 28 , further comprising generating a video thumbnail from a frame associated with the local maximum goodness measure.

30. A computer-implemented method comprising:

segmenting a video sequence into a plurality of shots;

selecting a frame from each of the plurality of shots;

for each selected frame, calculating a frame goodness measure as a function

of a histogram entropy and a histogram standard deviation;

determining a thumbnail frame from the selected frames as a function of the frame goodness measures; and

displaying the thumbnail frame for user browsing of video data.

31. The method of claim 30 , wherein the determining further comprises:

identifying a maximum frame goodness measure from the frame goodness measures; and

determining the thumbnail frame as a frame associated with the maximum frame goodness measure.

32. A computing device comprising:

a processor; and

a memory coupled to the processor, the memory comprising video content and computer-program instructions executable by the processor, the computer-program instructions being configured such that when executed by the processor to implement a video thumbnail algorithm the video thumbnail algorithm being configured to determine a visually representative video thumbnail frame by calculating a frame goodness measure for each of a plurality of video frames within the video content, each frame goodness measure comprising a weighted combination of a frame histogram entropy and a frame histogram standard deviation.

33. A video device comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program instructions executable by the processor for performing operations comprising:

recording video content; and

determining a video thumbnail; and

wherein the determining comprises calculating a frame goodness measure for each of a plurality of video frames within the video content, each frame goodness measure being based on a weighted combination of a frame histogram entropy and a frame histogram standard deviation.

34. The video device of claim 33 , further comprising a hardware circuit block configured to implement the video thumbnail algorithm.

35. A video recording device comprising:

means for calculating a color histogram of a video frame in a video sequence;

means for calculating an entropy of the color histogram;

means for calculating a standard deviation of the color histogram;

means for calculating a frame goodness measure as a weighted combination of the entropy and the standard deviation; and

means for displaying in a visually representative video thumbnail based on the frame goodness measure for user browsing of video data.

36. A computer comprising:

means for receiving a video sequence;

means for calculating frame goodness measures for frames within the video sequence from histogram entropies and histogram standard deviations; and

means for displaying a visually representative video thumbnail based on the frame goodness measure for user browsing of video data.

37. A computer comprising:

means for calculating a color histogram of a video frame in a video sequence;

means for calculating an entropy of the color histogram;

means for calculating a standard deviation of the color histogram;

means for calculating a frame goodness measure as a weighted combination of the entropy and the standard; and

means for displaying a visually representative video thumbnail based on the frame goodness measure for user browsing of video data.

38. A computer comprising:

means for temporally segmenting a video sequence;

means for calculating frame goodness measures within a particular temporal segment using histogram entropies and histogram standard deviations;

means for determining from the frame goodness measures, a local maximum goodness measure for the particular temporal segment; an

means for displaying a visually representative video thumbnail based on the maximum goodness measure for user browsing of video data.

39. A computer comprising:

means for segmenting a video sequence into a plurality of shots;

means for selecting a frame from each of the plurality of shots;

means for calculating a frame goodness measure as a function of a histogram entropy and a histogram standard deviation for each selected frame;

means for determining a thumbnail frame from the selected frames as a function of the frame goodness; and

means for displaying the thumbnail frame for user browsing of video data.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 22, 2018
From: MICROSOFT TECHNOLOGY LICENSING, LLC
To: SZ DJI TECHNOLOGY CO., LTD.
Reel/Frame 047259/0160 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 9, 2014
From: MICROSOFT CORPORATION
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 034541/0477 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 15, 2003
From: ZHANG, DONG; WANG, YIJIN; ZHANG, HONG-JIANG
To: MICROSOFT CORPORATION
Reel/Frame 014389/0108 →