IP Library Granted Patent US 11,372,917
Granted Patent B2
US 11,372,917 · App. 15/855,521 · Granted Jun 28, 2022

Labeling video files using acoustic vectors

Inventors: Ying Zhang (Palo Alto, CA); Yun Lei (San Francisco, CA)
Assignee: META PLATFORMS, INC.
G06F16/71G06N3/08G06Q50/01
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,372,917
App. No.
15/855,521
Granted
Jun 28, 2022
Kind
B2
Abstract

In one embodiment, a method includes receiving a video file. The video file includes a corresponding audio stream. The method further includes accessing the audio stream, and generating, based on the audio stream, a representative vector. The vector has a particular number of dimensions. The method further includes accessing a label-embedding space, which has the same particular number of dimensions, and includes a number of regions that each correspond to a respective label. The method further includes determining a region of the label-embedding space that corresponds to the vector, the determined region corresponding to a particular label. The method further includes associating the particular label with the video file.

Claims (68)

1. A method comprising:

by a computing device, receiving a video file, the video file comprising an audio stream;

by the computing device, accessing the audio stream of the video file;

by the computing device, generating, based on the audio stream, a vector representing the audio stream, wherein the vector has a particular number of dimensions (N);

by the computing device, accessing a label-embedding space, wherein the label-embedding space has:

the particular number of dimensions (N); and

a plurality of regions, wherein each region from the plurality of regions corresponds to a label from a plurality of labels;

by the computing device, determining a region of the label-embedding space that corresponds to the vector representing the audio stream, wherein the region is associated with a particular label; and

by the computing device, associating the particular label with the video file.

2. The method of claim 1 , wherein the particular label comprises:

a language in the video;

a genre of the video;

a maturity rating of the video;

a topic associated with the video; or

whether the video should be flagged for objectionable content.

3. The method of claim 2 , wherein the label-embedding space is selected from one of a plurality of label-embedding spaces based on a type of the label.

4. The method of claim 1 , wherein the label-embedding space is trained using a neural network and a plurality of labeled videos.

5. The method of claim 4 , wherein training the label-embedding space comprises:

for each of a plurality of labeled videos, determining a vector corresponding only to an audio file of the labeled video; and

assigning one or more regions corresponding to the vectors in the label-embedding space with the respective labels.

6. The method of claim 1 , further comprising:

making the labeled video file accessible to one or more users of a social-networking system;

receiving an indication that one or more of the users have viewed or interacted with the video file; and

based on the indication and information associated with the one or more users, determine whether the particular label should be changed.

7. The method of claim 6 , further comprising: based on the indication and information associated with the one or more users, determining whether the regions of the label-embedded space should be adjusted so that the video file no longer corresponds to the particular label.

8. The method of claim 1 , wherein the regions of the label-embedding space are further adjusted based on an image recognition method performed on the video file.

9. The method of claim 1 , further comprising:

dividing the received video file into a plurality of segments; and

for each segment, determining a particular label.

10. The method of claim 9 , wherein dividing the received video file into a plurality of segments is based on determining one or more scene transitions in the video file.

11. The method of claim 9 , wherein dividing the received video file into a plurality of segments is based on one or more timestamps in the video file.

12. The method of claim 9 , wherein dividing the received video file into a plurality of segments is based on using only audio recognition to determine segments based on separating one or more speakers.

13. The method of claim 1 , further comprising:

determining a new type of label to assign to a video file;

accessing a general embedding space based on a plurality of label-embedding spaces; and

creating a new label-embedding space based on the general embedding space and one or more labeled video files.

14. The method of claim 1 , wherein generating the vector comprises:

dividing the video file into a plurality of segments, wherein each particular segment overlaps the subsequent segment;

determining, for each segment, a vector having the particular number of dimensions; and

pooling the individual vectors for the plurality of segments into a single vector that is representative of the video file.

15. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

receive a video file, the video file comprising an audio stream;

access the audio stream of the video file;

generate, based on the audio stream, a vector representing the audio stream, wherein the vector has a particular number of dimensions (N);

access a label-embedding space, wherein the label-embedding space has:

the particular number of dimensions (N); and

a plurality of regions, wherein each region from the plurality of regions corresponds to a label from a plurality of labels;

determine a region of the label-embedding space that corresponds to the vector representing the audio stream, wherein the region is associated with a particular label; and

associate the particular label with the video file.

16. The media of claim 15 , wherein the particular label comprises:

a language in the video;

a genre of the video;

a maturity rating of the video;

a topic associated with the video; or

whether the video should be flagged for objectionable content.

17. The media of claim 16 , wherein the label-embedding space is selected from one of a plurality of label-embedding spaces based on a type of the label.

18. The media of claim 15 , wherein the label-embedding space is trained using a neural network and a plurality of labeled videos.

19. The media of claim 18 , wherein training the label-embedding space comprises:

for each of a plurality of labeled videos, determining a vector corresponding only to an audio file of the labeled video; and

assigning one or more regions corresponding to the vectors in the label-embedding space with the respective labels.

20. A system comprising: one or more processors; and a memory coupled to the processors comprising instructions executable by the processors, the processors being operable when executing the instructions to:

receive a video file, the video file comprising an audio stream;

access the audio stream of the video file;

generate, based on the audio stream, a vector representing the audio stream, wherein the vector has a particular number of dimensions (N);

access a label-embedding space, wherein the label-embedding space has:

the particular number of dimensions (N); and

a plurality of regions, wherein each region from the plurality of regions corresponds to a label from a plurality of labels;

determine a region of the label-embedding space that corresponds to the vector representing the audio stream, wherein the region is associated with a particular label; and associate the particular label with the video file.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 21, 2022
From: LEI, YUN
To: META PLATFORMS, INC.
Reel/Frame 059670/0137 →
CHANGE OF NAME Recorded Jan 24, 2022
From: FACEBOOK, INC.
To: META PLATFORMS, INC.
Reel/Frame 058827/0681 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 2, 2018
From: ZHANG, YING
To: FACEBOOK, INC.
Reel/Frame 046255/0940 →
Continuity (1)
Related Publication 20190197187A1 · Jun 27, 2019
Cited By (1)
US 12,627,885