IP Library Granted Patent US 11,354,900
Granted Patent B1
US 11,354,900 · App. 16/548,686 · Granted Jun 7, 2022

Classifiers for media content

Inventors: Shangwen Li (San Jose, CA); Jiantao Wu (Freemont, CA)
Assignee: Meta Platforms, Inc.
G06V20/41G06K9/6215G06K9/6227G06T7/0002G06V10/56G06V20/46G06V20/49G06T2207/10016G06T2207/20081
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,354,900
App. No.
16/548,686
Filed
Aug 22, 2019
Granted
Jun 7, 2022
Kind
B1
Art Unit
2661
USPC
382/159
Abstract

Techniques are described that classify content, and control whether and how the content is shared based on the classification(s). In some examples, video content may be classified based on sequential image frames of the video, and time between the sequential image frames. Audio content may be classified based on combining classifications of multiple sound events in the audio signal. The classifications may be used to control how the content is shared, such as by preventing offensive content from being shared and/or outputting recommendations or search results based on the classifications.

Claims (57)

1. A method comprising:

receiving a video comprising multiple image frames;

selecting a subset of the multiple image frames, the subset of the multiple image frames being sequential frames of the video;

inputting the subset of the multiple image frames into a machine-learned model trained to detect and classify objects in videos;

receiving, from the machine-learned model, an indication of a first object detected in the subset of the multiple image frames and a classification of the first object across the subset of the multiple image frames;

determining a first score based at least in part on the classification of the first object across the subset of the multiple image frames;

receiving, from the machine-learned model, an indication of a second object detected in the subset of the multiple image frames and a classification of the second object across the subset of the multiple frames;

determining a second score based at least at least in part on the classification of the second object across the subset of the multiple image frames; and

controlling sharing of the video with one or more user accounts based at least in part on the first score and the second score.

2. The method of claim 1 , wherein the subset of the multiple image frames comprise consecutive frames of the multiple image frames.

3. The method of claim 1 , wherein the machine-learned model is trained to detect and classify the objects based at least in part on a time between individual frames in the videos.

4. The method of claim 1 , wherein the machine-learned model is trained to detect a person or an animal engaged in fighting.

5. The method of claim 1 , wherein the machine-learned model is trained to detect animal cruelty, nudity, or sexually explicit acts.

6. The method of claim 1 , wherein a number of frames in the subset of the multiple image frames is selected based at least in part on an action that the machine-learned model is trained to detect.

7. The method of claim 1 , wherein the machine-learned model is further configured to detect and classify the objects in the videos based at least in part on one or more colors in one or more pixels of frames of the videos.

8. The method of claim 1 , wherein the machine-learned model is a first machine-learned model, the method further comprising:

inputting the first score for the first object into a second machine-learned model trained to determine offensiveness of content;

receiving, from the second machine-learned model, an offensiveness level of an item of content that includes the video; and

controlling sharing of the content based at least in part on the offensiveness level.

9. The method of claim 8 , wherein the item of content that includes the video further comprises one or more components, the one or more components including:

text;

information associated with a user account that is sharing the content;

audio; or

a comment associated with the content; and

wherein the offensiveness level is further based at least in part on the one or more components.

10. A system comprising:

one or more processors; and

computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:

receiving a video comprising multiple image frames;

selecting a subset of the multiple image frames, the subset of the multiple image frames being sequential frames of the video;

inputting the subset of the multiple image frames into a machine-learned model trained to detect and classify objects in videos;

receiving, from the machine-learned model, an indication of a first object detected in the subset of the multiple image frames and a classification of the first object across the subset of the multiple image frames;

determining a first score based at least in part on the classification of the first object across the subset of the multiple image frames;

receiving, from the machine-learned model, a second object detected in the subset of the multiple image frames and a classification of the second object across the subset of the multiple frames;

determining a second score based at least at least in part on the classification of the second object across the subset of the multiple image frames; and

controlling sharing of the video with one or more user accounts based at least in part on the first score and the second score.

11. The system of claim 10 , wherein the subset of the multiple image frames comprise consecutive frames of the multiple image frames.

12. The system of claim 10 , wherein the machine-learned model is trained to detect and classify the objects based at least in part on a time between individual frames in the videos.

13. The system of claim 10 , wherein a number of frames in the subset of the multiple image frames is selected based at least in part on an action that the machine-learned model is trained to detect.

14. The system of claim 10 , wherein the machine-learned model is further configured to detect and classify the objects in the videos based at least in part on one or more colors in one or more pixels of the images.

15. The system of claim 10 , wherein the first score corresponds to an offensiveness level associated with the video, and wherein controlling sharing of the video comprises preventing the video from being shared responsive to the first score being higher than a threshold offensiveness score.

16. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a server computing device, cause the server computing device to perform operations comprising:

receiving a video comprising multiple image frames;

selecting a subset of the multiple image frames, the subset of the multiple image frames being sequential frames of the video;

inputting the subset of the multiple image frames into a machine-learned model trained to detect and classify objects in videos;

receiving, from the machine-learned model, an indication of a first object detected in the subset of the multiple image frames and a classification of the first object across the subset of the multiple image frames;

determining a first score based at least in part on the classification of the first object;

receiving, from the machine-learned model, a second object detected in the subset of multiple image frames and a classification of the second object across the subset of the multiple image frames;

determining a second score based at least in part on the classification of the second object; and

controlling sharing of the video with one or more user accounts based at least in part on the first score and the second score.

17. The one or more non-transitory computer-readable media of claim 16 , wherein controlling sharing of the video comprises sharing the video in response to a search by the one or more user accounts, the search including a search term associated with the classification of the first object.

18. The one or more non-transitory computer-readable media of claim 16 , wherein controlling sharing of the video comprises:

determining an interest of the one or more user accounts based on prior content viewed by the one or more user accounts; and

sharing the video with the one or more user accounts responsive to the interest being associated with the classification of the first object.

19. The one or more non-transitory computer-readable media of claim 16 , wherein controlling sharing of the video comprises sharing the video with the one or more user accounts based on a similarity of the classification of the first object to the second object in one or more other videos viewed by the one or more user accounts.

20. The one or more non-transitory computer-readable media of claim 16 , wherein the subset of the multiple image frames comprise consecutive frames of the multiple image frames.

21. The one or more non-transitory computer-readable media of claim 16 , wherein the machine-learned model is trained to detect and classify the objects based at least in part on a time between individual frames in the videos.

Assignments (2)
CHANGE OF NAME Recorded May 5, 2022
From: FACEBOOK, INC.
To: META PLATFORMS, INC.
Reel/Frame 059855/0899 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 30, 2019
From: LI, SHANGWEN; WU, JIANTAO
To: FACEBOOK, INC.
Reel/Frame 050869/0401 →
Cited By (7)
US 12,489,953 US 12,513,365 US 12,567,256 US 12,568,290 US 12,585,719 US 12,639,921 US 12,647,650