Granted Patent B1

US 12,651,457 · App. 17/852,124 · Granted Jun 9, 2026

Inventors: Mohamed Kamal Omar (Seattle, WA); Ashutosh Sanan (Seattle, WA); Xiaohang Sun (Bellevue, WA); Han-Kai Hsu (Seattle, WA); Wentao Zhu (Redmond, WA); Xiang Hao (Redmond, WA); Ahmed Aly Saad Ahmed (Bothell, WA)

Assignee: Amazon Technologies, Inc.

G06V20/41G06F16/738G06V10/774G06V10/776G06V10/82

Abstract

Claims (78)

1 . A method, comprising:

receiving, from a user computing device comprising one or more processors and memory, user input data indicative of a new attribute of video content for adding labels to video data stored in a database of video data, wherein the video data stored in the database is not currently labelled for the new attribute, wherein the input data is further indicative of a type of machine learning model to be trained to label video data having the new attribute;

determining embedding values for a machine learning model of the type of machine learning model based on the user input data indicative of the new attribute;

determining a first subset of data having a similarity to the new attribute from a database of video data based on the user input data and the embedding values, wherein the video data of the database is labelled with previously determined labels according to one or more attributes different from the new attribute and is not currently labelled for the new attribute;

receiving indications of matching for the first subset of data, the indications of matching associated with whether an instance of the first subset of data is associated with the new attribute;

determining a training dataset, from the first subset of data, based on the indications of matching;

generating a trained machine learning model to label video data having the new attribute by training the machine learning model using the training dataset;

determining labels for a second subset of data using the trained machine learning model;

determining a precision of the trained machine learning model based on the labels for the second subset of data and label verification data that indicates whether the labels are accurately applied to the second subset of data;

performing at least one of:

further training the trained machine learning model using the label verification data in response to the precision being below a threshold; or

generating an alert that the trained machine learning model is trained in response to the precision being equal to or above the threshold;

using the trained machine learning model to identify video data in the database having the new attribute; and

storing indications of labels for the new attribute associated with the identified video data within the database such that labelling of the video data of the database is updated according to the new attribute and continues to provide the previously determined labels for the one or more attributes different from the new attribute.

2 . The method of claim 1 , further comprising:

searching a second database of existing machine learning models for the type of machine learning model, and wherein determining the embedding values comprises one of:

generating the embedding values in response to the database of existing machine learning models not including the type of machine learning model; or

selecting the embedding values of an existing machine learning model in response to the database including the type of machine learning model.

3 . The method of claim 1 , wherein:

the machine learning model comprises a convolutional neural network and a linear classifier, with the linear classifier configured to receive an output of the convolutional neural network;

the embedding values comprise the embedding values of the convolutional neural network; and

training the machine learning model comprises training the linear classifier using the training dataset.

4 . The method of claim 1 , wherein:

the labels comprise attributes of the video data; and

the user input data comprises two or more of text data, audio data, or video data.

5 . A method, comprising:

receiving a user query associated with a new attribute of video content for adding labels to video data having the new attribute stored in a database different from labels for any attributes currently associated with the video data stored in the database;

determining a first subset of the video data having a similarity to the new attribute based at least in part on the user query;

determining metadata indicating a correlation between the first subset of data and the new attribute;

determining a second subset of the video data based at least in part on the first subset of data and the metadata;

determining a training dataset based at least in part on the second subset of data;

training a machine learning model using the training dataset and the metadata;

determining, using the trained machine learning model, one or more instances of the new attribute within the video data stored in the database; and

storing indications of labels for the new attribute associated with the database identifying one or more locations of the determined one or more instances of the new attribute within the video data stored in the database.

6 . The method of claim 5 , wherein the new attribute is a first attribute and further comprising:

receiving a second user query for a second attribute;

determining a third subset of the video data based at least in part on the second user query;

determining second metadata indicating a second correlation between the third subset of data and the second attribute;

determining a second training dataset for the user query based at least in part on the third subset of data and the second metadata; and

training the machine learning model to determine instances of the second attribute in addition to the first attribute.

7 . The method of claim 5 , wherein determining the first subset of data comprises searching indexed data of the database of video data for keywords associated with the user query.

8 . The method of claim 5 , wherein determining the training dataset comprises iteratively:

receiving second metadata for a subset of data indicating a second correlation between the subset of data and the new attribute; and

determining a subsequent subset of data from the video data based at least in part on the subset of data and the second metadata.

9 . The method of claim 5 , further comprising determining a type of machine learning model by searching a database of existing machine learning models based at least in part on the user query or the new attribute.

10 . The method of claim 5 , further comprising determining an expected accuracy and precision of the machine learning model based at least in part on the training dataset prior to training the machine learning model.

11 . The method of claim 5 , further comprising:

determining one or more additional instances for one or more additional attributes within the video data using the machine learning model; and

determining a content summary for individual videos of the video data based at least in part on the one or more additional instances of the one or more additional attributes.

12 . The method of claim 11 , wherein the machine learning model comprises a feature extraction backbone and a linear classifier, and wherein training the machine learning model comprises training the linear classifier to identify instances of the one or more additional attributes.

13 . The method of claim 5 , wherein determining the metadata comprises presenting the first subset of data on a user interface and receiving indications with respect to positive instances and negative instances of the new attribute within the first subset of data.

14 . The method of claim 5 , wherein the user query comprises at least one of:

image data;

audio data; or

text data.

15 . A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving a user query associated with a new attribute of video content for adding labels to video data having the new attribute stored in a database that is different from any attributes currently associated with the video data stored in the database;

determining a first subset of the data having a similarity to the new attribute based at least in part on the user query;

determining metadata indicating a correlation between the first subset of data and the new attribute;

determining a second subset of the video data based at least in part on the first subset of data and the metadata;

determining a training dataset based at least in part on the second subset of data;

training a machine learning model using the training dataset and the metadata;

determining, using the trained machine learning model, one or more instances of the new attribute within the video data stored in the database of video data; and

storing indications of labels for the new attribute associated with the database identifying one or more locations of the determined one or more instances of the new attribute within the video data stored in the database.

16 . The non-transitory computer-readable medium of claim 15 , wherein determining the training dataset comprises iteratively:

receiving second metadata for a subset of data indicating a second correlation between the subset of data and the new attribute; and

determining a subsequent subset of data from the video data based at least in part on the subset of data and the second metadata.

17 . The non-transitory computer-readable medium of claim 15 , further comprising:

determining one or more additional instances for one or more additional attributes within the video data using the machine learning model; and

determining a content summary for individual videos of the video data based at least in part on the one or more additional instances of the one or more additional attributes.

18 . The non-transitory computer-readable medium of claim 17 , wherein the machine learning model comprises a feature extraction backbone and a linear classifier, and wherein training the machine learning model comprises training the linear classifier to identify instances of the one or more additional attributes.

19 . The non-transitory computer-readable medium of claim 15 , wherein determining the metadata comprises presenting the first subset of data on a user interface and receiving indications with respect to positive instances and negative instances of the new attribute within the first subset of data.

20 . The non-transitory computer-readable medium of claim 15 , wherein the new attribute is a first attribute and further comprising:

receiving a second user query for a second attribute;

determining a third subset of the video data based at least in part on the second user query;

determining second metadata indicating a second correlation between the third subset of data and the second attribute;

determining a second training dataset for the user query based at least in part on the third subset of data and the second metadata; and

training the machine learning model to determine instances of the second attribute in addition to the first attribute.

Assignments (1)

References Cited (21)