IP Library Granted Patent US 10,645,142
Granted Patent B2
US 10,645,142 · App. 15/270,960 · Granted May 5, 2020

Video keyframes display on online social networks

Inventors: Dirk John Stoop (Menlo Park, CA); Adam Eugene Bussing (San Francisco, CA); Oliver Scholz (Menlo Park, CA); Balmanohar Paluri (Mountain View, CA)
Assignee: Facebook, Inc.
H04L67/02H04L65/4084H04L65/602H04L67/306
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,645,142
App. No.
15/270,960
Granted
May 5, 2020
Kind
B2
Abstract

In one embodiment, a method includes receiving a query from a user for videos; identifying videos matching the query; retrieving, for each identified video, a set of keyframes that are associated with one or more concepts; calculating, for each keyframe of each identified video, a keyframe-score based on a prevalence of the concepts associated with the keyframe, determined with reference to the concepts associated with each other keyframe in the set of retrieved keyframes for the identified video; and sending, to the first user, a search-results interface including search results corresponding to one or more of the identified videos, each search result comprising keyframes for the corresponding identified video having keyframe-scores greater than a threshold keyframe-score.

Claims (47)

1. A method comprising, by one or more computing devices:

receiving, from a client system of a first user, a search query inputted by the first user, wherein the search query is associated with one or more concepts;

identifying one or more videos that match the search query;

retrieving, for each identified video, a set of keyframes for the identified video, each keyframe being a frame of the identified video, wherein each keyframe is associated with the one or more concepts associated with the search query;

calculating, for each keyframe of each identified video, a keyframe-score based on a prevalence of the one or more concepts associated with the keyframe, wherein the prevalence is determined with reference to the one or more concepts associated with each other keyframe in the set of retrieved keyframes for the identified video; and

sending, to the client system of the first user, instructions for displaying a search-results interface comprising one or more search results corresponding to one or more of the identified videos, wherein each search result is displayed with one or more optimal keyframes for the corresponding identified video, wherein the optimal keyframes for the corresponding identified video are keyframes associated with the one or more concepts associated with the search query and having keyframe-scores greater than a threshold keyframe-score,

wherein each of the one or more of the search results comprises a preview region, and

wherein the preview region displays the optimal keyframes of the respective search result, the display of the optimal keyframes being based on one or more inputs from the first user.

2. The method of claim 1 , further comprising:

accessing a social graph comprising a plurality of nodes and a plurality of edges connecting the nodes, each of the edges between two of the nodes representing a single degree of separation between them, the nodes comprising:

a first node corresponding to the first user; and

a plurality of second nodes corresponding to a plurality of objects, respectively.

3. The method of claim 1 , wherein the set of keyframes for a respective identified video is determined based on a keyframe-extraction process that comprises:

detecting one or more scene-changes in the respective identified video; and

extracting, for each of one or more of the scene-changes, a first frame that occurs in the respective identified video during the scene-change as a first keyframe.

4. The method of claim 3 , wherein one or more of the scene-changes are detected based on one or more changes in one or more visual features of a plurality of frames in the respective identified video.

5. The method of claim 3 , wherein one or more of the scene-changes are detected based on one or more changes in one or more audio features of the respective identified video.

6. The method of claim 1 , wherein the keyframe-score is further based on the one or more concepts associated with the search query matching one or more of the concepts associated with the respective keyframe.

7. The method of claim 1 , wherein the respective keyframe-score of each keyframe is further based on user engagement associated with a portion of the respective identified video that contains the respective keyframe.

8. The method of claim 1 , wherein the respective keyframe-score of each keyframe is further based on information associated with the first user.

9. The method of claim 1 , wherein the respective keyframe-score of each keyframe is further based on a current date or time, the current date or time being associated with the one or more concepts associated with the search query that match one or more of the concepts associated with the respective keyframe.

10. The method of claim 1 , wherein each of one or more of the search results comprises a preview region, wherein the preview region displays the optimal keyframes of the search result in a slideshow that automatically proceeds through each of the optimal keyframes.

11. The method of claim 10 , wherein each of the optimal keyframes is displayed for a duration that is based on the respective keyframe-score of the optimal keyframe.

12. The method of claim 1 , wherein the inputs from the first user correspond to inputs for navigating forward or backward through the optimal keyframes of the search result.

13. The method of claim 1 , wherein the search-results interface further comprises, for each of one or more of the search results, a timeline-scrubber element, wherein the timeline-scrubber element comprises a timeline of the corresponding identified video of the search result and a visual depiction of points in the timeline that correspond to occurrences in the corresponding identified video of the optimal keyframes, and wherein the timeline-scrubber element further comprises a moveable scrubber component positioned proximate to the timeline, the position of the moveable scrubber component corresponding to a currently displayed optimal keyframe.

14. The method of claim 1 , further comprising, for each of one or more of the search results, ordering the respective optimal keyframes for display based on the relative keyframe-scores of the respective optimal keyframes.

15. The method of claim 1 , further comprising, for a particular search result:

receiving, from the client system of the first user, an indication of a trigger event associated with a particular optimal keyframe of the particular search result; and

causing, in response to the detection of the trigger event, the client system of the first user to play the corresponding identified video from a time-point of the particular optimal keyframe.

16. The method of claim 15 , wherein the trigger event comprises an input by the first user to cause the particular optimal keyframe to be displayed for a threshold period of time.

17. The method of claim 1 , wherein the search-results interface further comprises, for each of one or more of the search results, a display of a description of one or more of the respective optimal keyframes, the description of each respective optimal keyframe being based on one or more n-grams or media items extracted from one or more communications associated with a portion of the corresponding identified video that contains the respective optimal keyframe.

18. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

receive, from a client system of a first user, a search query inputted by the first user, wherein the search query is associated with one or more concepts;

identify one or more videos that match the search query;

retrieve, for each identified video, a set of keyframes for the identified video, each keyframe being a frame of the identified video, wherein each keyframe is associated with the one or more concepts associated with the search query;

calculate, for each keyframe of each identified video, a keyframe-score based on a prevalence of the one or more concepts associated with the keyframe, wherein the prevalence is determined with reference to the one or more concepts associated with each other keyframe in the set of retrieved keyframes for the identified video; and

send, to the client system of the first user, instructions for displaying a search-results interface comprising one or more search results corresponding to one or more of the identified videos, wherein each search result is displayed with one or more optimal keyframes for the corresponding identified video, wherein the optimal keyframes for the corresponding identified video are keyframes associated with the one or more concepts associated with the search query and having keyframe-scores greater than a threshold keyframe-score,

wherein each of one or more of the search results comprises a preview region, and

wherein the preview region displays the optimal keyframes of the search result, the display of the optimal keyframes being based on one or more inputs from the first user.

19. A system comprising: one or more processors; and a non-transitory memory coupled to the one or more processors comprising instructions executable by the one or more processors, the one or more processors operable when executing the instructions to:

receive, from a client system of a first user, a search query inputted by the first user, wherein the search query is associated with one or more concepts;

identify one or more videos that match the search query;

retrieve, for each identified video, a set of keyframes for the identified video, each keyframe being a frame of the identified video, wherein each keyframe is associated with the one or more concepts associated with the search query;

calculate, for each keyframe of each identified video, a keyframe-score based on a prevalence of the one or more concepts associated with the keyframe, wherein the prevalence is determined with reference to the one or more concepts associated with each other keyframe in the set of retrieved keyframes for the identified video; and

send, to the client system of the first user, instructions for displaying a search-results interface comprising one or more search results corresponding to one or more of the identified videos, wherein each search result is displayed with one or more optimal keyframes for the corresponding identified video, wherein the optimal keyframes for the corresponding identified video are keyframes associated with the one or more concepts associated with the search query and having keyframe-scores greater than a threshold keyframe-score,

wherein each of one or more of the search results comprises a preview region, and

wherein the preview region displays the optimal keyframes of the search result, the display of the optimal keyframes being based on one or more inputs from the first user.

Assignments (4)
CHANGE OF NAME Recorded Jan 23, 2022
From: FACEBOOK, INC.
To: META PLATFORMS, INC.
Reel/Frame 058819/0711 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 16, 2017
From: SCHOLZ, OLIVER
To: FACEBOOK, INC.
Reel/Frame 042035/0549 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 24, 2017
From: STOOP, DIRK JOHN
To: FACEBOOK, INC.
Reel/Frame 041068/0417 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 28, 2016
From: BUSSING, ADAM EUGENE; PALURI, BALMANOHAR
To: FACEBOOK, INC.
Reel/Frame 040510/0065 →
Continuity (1)
Related Publication 20180084023A1 · Mar 22, 2018