Support model and orchestration procedure for manual labeling processes
Labelling in machine learning is disclosed. Assistant models are trained to generate the labelling behavior of corresponding labelers. When an observation is received, the label generated by the assistant model is compared to the user-generated label. If the labels are different, an ensemble operation is performed wherein multiple assistant models, each configured to mimic the labelling behavior of a different user, are used to resolve the conflict and determine a final label for the observation. Labels generated by the ensemble operation may be incorporated into retraining operations performed to retrain the assistant models.
1 . A method comprising:
receiving a first label for an observation, wherein the first label is generated by an assistant model, the observation having a content type;
when a user determines to review the observation when the content type is sensitive:
receiving a second label for the observation, wherein the second label is generated by the user associated with the assistant model;
comparing the first label with the second label; and
performing an ensemble operation when the first label and the second label are different, wherein the first label is stored in a database when the first label and the second label are equal as a final label;
when the user declines to review the observation, performing the ensemble operation; and
wherein the ensemble operation includes determining the final label for the observation based on labels generated by a plurality of assistant models for the observation; and
storing the final label determined by the ensemble operation in a unified database; and
training the plurality of assistant models with the database and/or the unified database.
2 . The method of claim 1 , further comprising training the assistant model to mimic a labeling behavior of the user.
3 . The method of claim 1 , further comprising storing the first label and the observation in the database.
4 . The method of claim 1 , wherein the assistant model is trained with data labeled by the user such that the assistant model mimics labeling of the user.
5 . The method of claim 1 , wherein the ensemble operation includes inputting the observation to each of the plurality of assistant models, the plurality of assistant models including the assistant model, wherein each of the plurality of assistant models is configured to mimic labeling behavior of a different user.
6 . The method of claim 5 , wherein a set of labels is generated by the plurality of assistant models.
7 . The method of claim 6 , further comprising selecting a most common label from the set of labels as the final label.
8 . The method of claim 7 , further comprising storing the most common label and the observation in the unified database, wherein labels in the unified database are only labeled by the plurality of assistant models.
9 . The method of claim 8 , further comprising retraining the assistant model using the unified database and the database that includes labels generated by the assistant model that match labels generated by the user.
10 . A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
receiving a first label for an observation, wherein the first label is generated by an assistant model, the observation having a content type;
when a user determines to review the observation when the content type is sensitive:
receiving a second label for the observation, wherein the second label is generated by the user associated with the assistant model;
comparing the first label with the second label; and
performing an ensemble operation when the first label and the second label are different, wherein the first label is stored in a database when the first label and the second label are equal as a final label;
when the user declines to review the observation, performing the ensemble operation; and
wherein the ensemble operation includes determining the final label for the observation based on labels generated by a plurality of assistant models for the observation; and
storing the final label determined by the ensemble operation in a unified database; and
training the plurality of assistant models with the database and/or the unified database.
11 . The non-transitory storage medium of claim 10 , further comprising training the assistant model to mimic a labeling behavior of the user.
12 . The non-transitory storage medium of claim 10 , further comprising storing the first label and the observation in the database.
13 . The non-transitory storage medium of claim 10 , wherein the assistant model is trained with data labeled by the user such that the assistant model mimics labeling of the user.
14 . The non-transitory storage medium of claim 10 , wherein the ensemble operation includes inputting the observation to each of the plurality of assistant models, the plurality of assistant models including the assistant model, wherein each of the plurality of assistant models is configured to mimic labeling behavior of a different user.
15 . The non-transitory storage medium of claim 14 , wherein a set of labels is generated by the plurality of assistant models.
16 . The non-transitory storage medium of claim 15 , further comprising selecting a most common label from the set of labels as the final label.
17 . The non-transitory storage medium of claim 16 , further comprising storing the most common label and the observation in the unified database, wherein labels in the unified database are only labeled by the plurality of assistant models.
18 . The non-transitory storage medium of claim 17 , further comprising retraining the assistant model using the unified database and the database that includes labels generated by the assistant model that match labels generated by the user.