IP Library Granted Patent US 11,941,049
Granted Patent B2
US 11,941,049 · App. 17/902,457 · Granted Mar 26, 2024

Adaptive search results for multimedia search queries

Inventors: Amol Jindal (Punjab, IN); Subham Gupta (Uttarakhand, IN); Poonam Bhalla (New Delhi, IN); Krishna Singh Karki (New Delhi, IN); Ajay Bedi (Pradesh, IN)
Assignee: Adobe Inc.
G06F16/738G06F16/7328G06F16/735G06F16/783G06F16/7837G06F16/7867
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,941,049
App. No.
17/902,457
Granted
Mar 26, 2024
Kind
B2
Abstract

A system identifies a video comprising frames associated with content tags. The system detects features for each frame of the video. The system identifies, based on the detected features, scenes of the video. The system determines, for each frame for each scene, a frame score that indicates a number of content tags that match the other frames within the scene. The system selects, for each scene, a set of key frames that represent the scene based on the determined frame scores. The system receives a search query comprising a keyword. The system generates, for display, search results responsive to the search query including a dynamic preview of the video. The dynamic preview comprises an arrangement of frames of the video corresponding to each scene of the video. Each of the arrangement of frames is selected from the selected set of key frames representing the respective scene of the video.

Claims (72)

1. A method performed by one or more computing devices, comprising:

identifying a video comprising frames, the frames associated with content tags;

detecting features for each frame of the video;

identifying, based on the detected features for each frame of the video, scenes of the video;

determining, for each frame for each scene of the identified scenes of the video, a frame score that indicates a number of content tags in the respective frame of the respective scene that match content tags associated with the other frames within the respective scene;

determining, for each scene of the identified scenes of the video, a mean frame score based on a total number of content tags within the frames of the respective scene divided by a total number of the frames of the respective scene;

selecting, for each scene of the identified scenes of the video, a subset of frames of the respective scene based on the determined frame scores, wherein each frame of the selected subset of frames has a determined frame score that is greater than the mean frame score;

receiving a search query comprising a keyword; and

generating, for display, search results responsive to the search query, the search results including a dynamic preview of the video, wherein the dynamic preview comprises an arrangement of frames of the video, each frame of the arrangement of frames corresponding to a respective identified scene of the identified scenes of the video, wherein each frame of the arrangement of frames is selected from the selected subset of frames.

2. The method of claim 1 , further comprising:

extracting content attributes from each frame of the video frames; and

generating, for each frame of the video, the content tags based on the respective content attributes.

3. The method of claim 1 , wherein the dynamic preview comprises a video clip, the method further comprising:

displaying, responsive to the search query, a search result comprising the dynamic preview;

detecting a user input comprising a location within a predefined distance of the dynamic preview; and

playing back the video clip in response to the user input.

4. The method of claim 1 , wherein generating the search results comprising the dynamic preview comprises:

identifying a time duration from a user profile; and

combining the arrangement of frames into a video clip having the time duration.

5. The method of claim 1 , wherein the dynamic preview comprises a collage, a GIF, or a playback loop.

6. The method of claim 1 , further comprising:

determining, for each frame of the video, an aesthetic score; and

wherein the arrangement of frames is based in part on the aesthetic scores of each of the arrangement of frames.

7. The method of claim 6 , wherein the aesthetic score comprises a parameter indicating one or more of a quality, a balancing, a harmony, a sharpness, a lighting, or a symmetry.

8. The method of claim 6 , wherein generating, for display, the dynamic preview comprises:

computing a respective total aesthetic score for each scene of the video, wherein the total aesthetic score comprising a sum of the aesthetic scores for a subset of frames within the scene,

wherein selecting the subset of frames is based in part on the subset of frames being included in respective scenes having greater total aesthetic scores than other scenes of the video.

9. The method of claim 1 , wherein each frame of the arrangement of frames respectively comprise a timestamp, and wherein generating, for display, the dynamic preview comprises:

determining an order of the subset of frames based on a chronological order of the timestamps; and

arranging the subset of frames based at least in part on the chronological order.

10. A system comprising

a processing device; and

a non-transitory computer-readable medium communicatively coupled to the processing device and storing program code,

wherein the processing device is configured for executing the program code and thereby performing operations comprising:

identifying a video comprising frames, the frames associated with content tags;

detecting features for each frame of the video;

identifying, based on the detected features for each frame of the video, scenes of the video;

determining, for each frame for each scene of the identified scenes of the video, a frame score that indicates a number of content tags in the respective frame of the respective scene that match content tags associated with the other frames within the respective scene;

determining, for each scene of the identified scenes of the video, a mean frame score based on a total number of content tags within the frames of the respective scene divided by a total number of the frames of the respective scene;

selecting, for each scene of the identified scenes of the video, a subset of frames of the respective scene based on the determined frame scores, wherein each frame of the selected subset of frames has a determined frame score that is greater than the mean frame score;

receiving a search query comprising a keyword; and

generating, for display, search results responsive to the search query, the search results including a dynamic preview of the video, wherein the dynamic preview comprises an arrangement of frames of the video, each frame of the arrangement of frames corresponding to a respective identified scene of the identified scenes of the video, wherein each frame of the arrangement of frames is selected from the selected subset of frames.

11. The system of claim 10 , the operations further comprising:

extracting content attributes from each frame of the video; and

generating, for each frame of the video, the content tags based on the respective content attributes.

12. The system of claim 10 , wherein the dynamic preview comprises a video clip, the operations further comprising:

displaying, responsive to the search query, a search result comprising the dynamic preview;

detecting a user input comprising a location within a predefined distance of the dynamic preview; and

playing back the video clip in response to the user input.

13. The system of claim 10 , wherein generating the search results comprising the dynamic preview comprises:

identifying a time duration from a user profile; and

combining the arrangement of frames into a video clip having the time duration.

14. The system of claim 10 , wherein the dynamic preview comprises a collage, a GIF, or a playback loop.

15. The system of claim 10 , the operations further comprising:

determining, for each frame of the video, an aesthetic score; and

wherein the arrangement of frames is based in part on the aesthetic scores of each frame of the arrangement of frames.

16. The system of claim 15 , wherein the aesthetic score comprises a parameter indicating one or more of a quality, a balancing, a harmony, a sharpness, a lighting, or a symmetry.

17. The system of claim 15 , wherein generating, for display, the dynamic preview comprises:

computing a respective total aesthetic score for each scene of the video, wherein the total aesthetic score comprising a sum of the aesthetic scores for a subset of frames within the scene,

wherein selecting the subset of frames is based in part on the subset of frames being included in respective scenes having greater total aesthetic scores than other scenes of the video.

18. The system of claim 10 , wherein each frame of the arrangement of frames respectively comprise a timestamp, and wherein generating, for display, the dynamic preview comprises:

determining an order of the subset of frames based on a chronological order of the timestamps; and

arranging the subset of frames based at least in part on the chronological order.

19. A non-transitory computer-readable medium having program code stored thereon, wherein the program code, when executed by one or more processing devices, performs operations comprising:

identifying a video comprising frames, the frames associated with content tags;

detecting features for each frame of the video;

identifying, based on the detected features for each frame of the video, scenes of the video;

determining, for each frame for each scene of the identified scenes of the video, a frame score that indicates a number of content tags in the respective frame of the respective scene that match content tags associated with the other frames within the respective scene;

determining, for each scene of the identified scenes of the video, a mean frame score based on a total number of content tags within the frames of the respective scene divided by a total number of the frames of the respective scene;

selecting, for each scene of the identified scenes of the video, a subset of frames of the respective scene based on the determined frame scores, wherein each frame of the selected subset of frames has a determined frame score that is greater than the mean frame score;

receiving a search query comprising a keyword; and

generating, for display, search results responsive to the search query, the search results including a dynamic preview of the video, wherein the dynamic preview comprises an arrangement of frames of the video, each frame of the arrangement of frames corresponding to a respective identified scene of the identified scenes of the video, wherein each frame of the arrangement of frames is selected from the selected subset of frames.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 3, 2022
From: JINDAL, AMOL; GUPTA, SUBHAM; BHALLA, POONAM; KARKI, KRISHNA SINGH; BEDI, AJAY
To: ADOBE INC.
Reel/Frame 060986/0086 →
Continuity (2)
Continuation 16591847 · Oct 3, 2019
Related Publication 20220414149A1 · Dec 29, 2022