IP Library › Granted Patent US 10,998,007
Granted Patent B2
US 10,998,007 · App. 16/588,662 · Granted May 4, 2021

Providing context aware video searching

Inventors: Ajay Bedi (Hamirpur, IN); Amol Jindal (Patiala, IN)
Assignee: ADOBE INC.
G11B27/34G06K9/00744H04N21/47217H04N21/845
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,998,007
App. No.
16/588,662
Granted
May 4, 2021
Kind
B2
Abstract

This disclosure relates to methods, non-transitory computer readable media, and systems that can generate a context-aware-video-progress bar including a video-scene-proportionate timeline with time-interval sections sized according to relative scene proportions within time intervals of a video. In some implementations, for instance, the disclosed systems determine relative proportions of scenes within a video across time intervals of the video and generate a video-scene-proportionate timeline comprising time-interval sections sized proportionate to the relative proportions of scenes across the time intervals. By integrating the video-scene-proportionate timeline within a video-progress bar, the disclosed systems generate a context-aware-video-progress bar for a video. Such a context-aware-video-progress bar can facilitate more precise and intelligent scrubbing through a video, a dynamic graphical user interface for navigating within and identifying frames of the video, and a flexible user-friendly tool for quickly identifying scenes.

Claims (81)

1. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computing device to:

detect reference frames comprising image features similar to other frames of scenes within a video by:

determining, utilizing a feature-detection algorithm, a numerical value of image-feature points that are shared in common between particular frames of same scenes and are associated with objects from the particular frames;

generating feature-matching scores for frames corresponding to the scenes whereby a feature-matching score indicates a degree to which particular image-feature points within a particular frame matches image-feature points within the other frames of a same scene based on the numerical value of image-feature points that are shared in common between the particular frame and the other frames of the same scene; and

selecting particular reference frames for particular scenes based on a highest feature-matching score from among the feature-matching scores for frames corresponding to a particular scene;

determine relative proportions of the reference frames across a set of time intervals within the video;

generate a video-scene-proportionate timeline for the video comprising time-interval sections for the set of time intervals sized according to the relative proportions of the reference frames across the set of time intervals; and

provide, for display within a graphical user interface, the video and a context-aware-video-progress bar comprising the video-scene-proportionate timeline.

2. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to detect the reference frames comprising the image features similar to the other frames of the scenes within the video in part by:

detecting, for a first scene, a first reference frame comprising a subset of image-feature points depicting a first object; and

detecting, for a second scene, a second reference frame comprising a subset of image-feature points depicting a second object.

3. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to identify a subset of frames corresponding to a scene within the video in part by:

detecting a set of image-feature points within a set of frames corresponding to the video;

comparing a selected subset of image-feature points from a selected frame to a subset of image-feature points from each frame of a contiguous set of frames within a threshold number of frames from the selected frame;

determining that the selected subset of image-feature points from the selected frame does not satisfy a minimum number or a minimum percentage of matching image-feature points shared in common with the subset of image-feature points from each frame of the contiguous set of frames; and

identifying the selected frame as a frame for the subset of frames corresponding to the scene.

4. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate a feature-matching score for a frame from a subset of feature-matching scores for a subset of frames by determining a mean number of matching image-feature points for the frame in common with each other frame within the subset of frames.

5. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to determine a normalized congestion score for a time interval from the set of time intervals by determining a ratio of a number of the reference frames within the time interval to a total number of the reference frames within the video.

6. The non-transitory computer readable medium of claim 5 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the video-scene-proportionate timeline for the video by:

generating a first time-interval section sized in proportion to a first normalized congestion score for a first time interval of the video; and

generating a second time-interval section sized in proportion to a second normalized congestion score for a second time interval of the video.

7. The non-transitory computer readable medium of claim 6 , further comprising instructions that, when executed by the at least one processor, cause the computing device to determine the numerical value of image-feature points that are shared in common between the particular frames of the same scenes in part by:

resizing a subset of frames corresponding to a scene to generate a subset of smaller frames;

detecting, utilizing the feature-detection algorithm, subsets of image-feature points within particular smaller frames from the subset of smaller frames; and

determining a particular numerical value of image-feature points shared in common between the particular smaller frames.

8. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:

detect a user selection of a time location within the context-aware-video-progress bar corresponding to a time-interval section sized in proportion to a congestion score for a time interval within the video-scene-proportionate timeline; and

based on detecting the user selection of the time location:

select a portion of the video from a time within the time interval corresponding to the time-interval section of the video-scene-proportionate timeline; and

render a thumbnail image of the portion of the video overlaid on a display of the video.

9. A system comprising:

one or more memory devices comprising a video having multiple frames; and

one or more server devices that cause the system to:

detect reference frames comprising image features similar to other frames of scenes within a video by:

determining, utilizing a feature-detection algorithm, a numerical value of image-feature points that are shared in common between particular frames of same scenes and are associated with objects from the particular frames;

generating feature-matching scores for frames corresponding to the scenes whereby a feature-matching score indicates a degree to which particular image-feature points within a particular frame matches image-feature points within the other frames of a same scene based on the numerical value of image-feature points that are shared in common between the particular frame and the other frames of the same scene; and

selecting particular reference frames for particular scenes from among the frames based on a highest feature-matching score from among the feature-matching scores for frames corresponding to a particular scene;

determine relative proportions of the reference frames across a set of time intervals within the video by:

determining a number of the reference frames within each time interval from the set of time intervals;

generating a set of congestion scores for the set of time intervals indicating relative proportions of the reference frames within particular time intervals;

generate a video-scene-proportionate timeline for the video comprising time-interval sections in proportion to the set of congestion scores for the set of time intervals; and

provide, for display within a graphical user interface, the video and a context-aware-video-progress bar comprising the video-scene-proportionate timeline.

10. The system of claim 9 , wherein the one or more server devices further cause the system to detect the reference frames comprising the image features similar to the other frames of the scenes within the video in part by:

detecting, for a first scene, a first reference frame comprising a subset of image-feature points depicting a first object; and

detecting, for a second scene, a second reference frame comprising a subset of image-feature points depicting a second object.

11. The system of claim 9 , wherein the one or more server devices further cause the system to identify a subset of frames corresponding to a scene within the video in part by:

detecting a set of image-feature points within a set of frames corresponding to the video;

comparing a selected subset of image-feature points from a selected frame to a subset of image-feature points from each frame of a contiguous set of frames within a threshold number of frames from the selected frame;

determining that the selected subset of image-feature points from the selected frame does not satisfy a minimum number or a minimum percentage of matching image-feature points shared in common with the subset of image-feature points from each frame of the contiguous set of frames; and

identifying the selected frame as a frame for the first subset of frames corresponding to the scene.

12. The system of claim 9 , wherein the one or more server devices further cause the system to generate a feature-matching score for a frame from a subset of feature-matching scores for a subset of frames by determining a mean number of matching image-feature points for the frame in common with each other frame within the subset of frames.

13. The system of claim 9 , wherein the one or more server devices further cause the system to determine a congestion score for a time interval from the set of time intervals by determining, as a normalized congestion score for the time interval, a ratio of a number of the reference frames within the time interval to a total number of the reference frames within the video.

14. The system of claim 9 , wherein the one or more server devices further cause the system to generate the video-scene-proportionate timeline for the video by:

generating a larger time-interval section sized in proportion to a first congestion score indicating a relatively larger proportion of the reference frames within a first time interval; and

generating a smaller time-interval section sized in proportion to a second congestion score indicating a relatively smaller proportion of the reference frames within a second time interval.

15. The system of claim 9 , wherein the one or more server devices further cause the system to determine the numerical value of image-feature points that are shared in common between the particular frames of the same scenes in part by:

resizing a subset of frames corresponding to a scene to generate a subset of smaller frames;

detecting, utilizing the feature-detection algorithm, subsets of image-feature points within particular smaller frames from the subset of smaller frames; and

determining a particular numerical value of image-feature points shared in common between the particular smaller frames.

16. A computer-implemented method comprising:

detecting reference frames comprising image features similar to other frames of scenes within a video by:

determining, utilizing a feature-detection algorithm, a numerical value of image-feature points that are shared in common between particular frames of same scenes and are associated with objects from the particular frames;

generating feature-matching scores for frames corresponding to the scenes whereby a feature-matching score indicates a degree to which particular image-feature points within a particular frame matches image-feature points within the other frames of a same scene based on the numerical value of image-feature points that are shared in common between the particular frame and the other frames of the same scene; and

selecting particular reference frames for particular scenes based on a highest feature-matching score from among the feature-matching scores for frames corresponding to a particular scene;

determining relative proportions of the reference frames across a set of time intervals within the video;

generating a video-scene-proportionate timeline for the video comprising time-interval sections for the set of time intervals sized according to the relative proportions of the reference frames across the set of time intervals; and

providing, for display within a graphical user interface, the video and a context-aware-video-progress bar comprising the video-scene-proportionate timeline.

17. The computer-implemented method of claim 16 , wherein generating a feature-matching score for a frame from a subset of feature-matching scores for a subset of frames comprises determining a mean number of matching image-feature points for the frame in common with each other frame within the subset of frames.

18. The computer-implemented method of claim 16 , wherein detecting the reference frames comprising the image features similar to the other frames of the scenes within the video comprises:

detecting, for a first scene, a first reference frame comprising a subset of image-feature points depicting a first object; and

detecting, for a second scene, a second reference frame comprising a subset of image-feature points depicting a second object.

19. The computer-implemented method of claim 16 , further comprising identifying a subset of frames corresponding to a scene within the video in part by:

detecting a set of image-feature points within a set of frames corresponding to the video;

comparing a selected subset of image-feature points from a selected frame to a subset of image-feature points from each frame of a contiguous set of frames within a threshold number of frames from the selected frame;

determining that the selected subset of image-feature points from the selected frame does not satisfy a minimum number or a minimum percentage of matching image-feature points shared in common with the subset of image-feature points from each frame of the contiguous set of frames; and

identifying the selected frame as a frame for the subset of frames corresponding to the scene.

20. The computer-implemented method of claim 16 , further comprising:

detecting a user selection of a time location within the context-aware-video-progress bar corresponding to a time-interval section sized in proportion to a congestion score for a time interval within the video-scene-proportionate timeline; and

based on detecting the user selection of the time location:

selecting a portion of the video from a time within the time interval corresponding to the time-interval section of the video-scene-proportionate timeline; and

rendering a thumbnail image of the portion of the video overlaid on a display of the video.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 30, 2019
From: BEDI, AJAY; JINDAL, AMOL
To: ADOBE INC.
Reel/Frame 050570/0402 →
Continuity (1)
Related Publication 20210098026A1 · Apr 1, 2021