Lexical dropout for natural language processing
Techniques are provided for improved training of a machine learning model using lexical dropout. A machine learning model and a training data set are accessed. The training data set can include sample utterances and corresponding labels. A dropout parameter is identified. The dropout parameter can indicate a likelihood for dropping out one or more feature vectors for tokens associated with respective entities during training of the machine learning model. The dropout parameter is applied to feature vectors for tokens associated with respective entities. The machine learning model is trained using the training data set and the dropout parameter to generate a trained machine learning model. The use of the trained the machine learning model is facilitated.
1 . A computer-implemented method for training a machine learning model to process audio or textual language input, the method comprising:
accessing a machine learning model;
accessing a training data set that includes sample utterances and corresponding labels;
identifying a dropout parameter that indicates a likelihood for dropping out one or more feature vectors for tokens labeled as respective entities during training of the machine learning model;
training the machine learning model using the training data set and the dropout parameter to generate a trained machine learning model, including selectively applying the dropout parameter to the one or more feature vectors for the tokens labeled as the respective entities, wherein, when the dropout parameter is applied, the machine learning model is caused to learn contextual information of the training data set, and wherein the contextual information includes a subset of the training data that is not labeled as an entity, and wherein the contextual information includes at least one or more verbs or at least one or more prepositions;
and
facilitating use of the trained the machine learning model to identify a named entity based upon an input utterance.
2 . The method of claim 1 , wherein the dropout parameter is a first dropout parameter, the method further comprising:
processing the training data set using a first model to generate a first feature vector; and
processing the training data set using a second model to generate a second feature vector,
wherein selectively applying the dropout parameter comprises applying the first dropout parameter to the first feature vector and applying a second dropout parameter to the second feature vector.
3 . The method of claim 1 , wherein the dropout parameter is a hyperparameter of the machine learning model, the method further comprising performing hypertuning to identify the dropout parameter.
4 . The method of claim 1 , wherein selectively applying the dropout parameter to the one or more feature vectors includes selectively applying the dropout parameter to one or more named entity tokens generated from the one or more feature vectors.
5 . The method of claim 1 , wherein selectively applying the dropout parameter to the one or more feature vectors includes selectively applying the dropout parameter to one or more tokens generated from the one or more feature vectors and matched via a gazetteer.
6 . The method of claim 1 , wherein:
the dropout parameter is a dropout rate;
applying the dropout parameter comprises dropping out the one or more feature vectors for the tokens labeled as the respective entities according to the dropout rate; and
the machine learning model includes a plurality of self-attention layers.
7 . A system comprising:
one or more processors; and
a non-transitory computer-readable memory coupled to the one or more processors, the memory comprising a plurality of instructions executable by the one or more processors to cause the one or more processors to perform operations comprising:
accessing a machine learning model;
accessing a training data set that includes sample utterances and corresponding labels;
identifying a dropout parameter that indicates a likelihood for dropping out one or more feature vectors for tokens labeled as respective entities during training of the machine learning model;
training the machine learning model using the training data set and the dropout parameter to generate a trained machine learning model, including selectively applying the dropout parameter to the one or more feature vectors for the tokens labeled as the respective entities, wherein, when the dropout parameter is applied, the machine learning model is caused to learn contextual information of the training data set, and wherein the contextual information includes a subset of the training data that is not labeled as an entity, and wherein the contextual information includes at least one or more verbs or at least one or more prepositions;
and
facilitating use of the trained the machine learning model to identify a named entity based upon an input utterance.
8 . The system of claim 7 , wherein the dropout parameter is a first dropout parameter, and wherein the operations further comprise:
processing the training data set using a first model to generate a first feature vector; and
processing the training data set using a second model to generate a second feature vector,
wherein selectively applying the dropout parameter comprises applying the first dropout parameter to the first feature vector and applying a second dropout parameter to the second feature vector.
9 . The system of claim 7 , wherein the dropout parameter is a hyperparameter of the machine learning model, and wherein the operations further comprise performing hypertuning to identify the dropout parameter.
10 . The system of claim 7 , wherein the operation of selectively applying the dropout parameter to the one or more feature vectors includes selectively applying the dropout parameter to one or more named entity tokens generated from the one or more feature vectors.
11 . The system of claim 7 , wherein the operation of selectively applying the dropout parameter to the one or more feature vectors includes selectively applying the dropout parameter to one or more tokens generated from the one or more feature vectors and matched via a gazetteer.
12 . The system of claim 7 , wherein:
the dropout parameter is a dropout rate;
the operation of applying the dropout parameter comprises dropping out the one or more feature vectors for the tokens labeled as the respective entities according to the dropout rate; and
the machine learning model includes a plurality of self-attention layers.
13 . A non-transitory computer-readable memory comprising a plurality of instructions executable by one or more processors to cause the one or more processors to perform operations comprising:
accessing a machine learning model and a training data set that includes sample utterances and corresponding labels;
identifying a dropout parameter that indicates a likelihood for dropping out one or more feature vectors for tokens labeled as respective entities during training of the machine learning model;
training the machine learning model using the training data set and the dropout parameter to generate a trained machine learning model, including selectively applying the dropout parameter to the one or more feature vectors for the tokens labeled as the respective entities, wherein, when the dropout parameter is applied, the machine learning model is caused to learn contextual information of the training data set, and wherein the contextual information includes a subset of the training data that is not labeled as an entity, and wherein the contextual information includes at least one or more verbs or at least one or more prepositions;
and
facilitating use of the trained the machine learning model to identify a named entity based upon an input utterance.
14 . The non-transitory computer-readable memory of claim 13 , wherein the dropout parameter is a first dropout parameter, and wherein the operations further comprise:
processing the training data set using a first model to generate a first feature vector; and
processing the training data set using a second model to generate a second feature vector,
wherein selectively applying the dropout parameter comprises applying the first dropout parameter to the first feature vector and applying a second dropout parameter to the second feature vector.
15 . The non-transitory computer-readable memory of claim 13 , wherein the dropout parameter is a hyperparameter of the machine learning model, and wherein the operations further comprise performing hypertuning to identify the dropout parameter.
16 . The non-transitory computer-readable memory of claim 13 , wherein the operation of selectively applying the dropout parameter to the one or more feature vectors includes selectively applying the dropout parameter to one or more named entity tokens generated from the one or more feature vectors.
17 . The non-transitory computer-readable memory of claim 13 , wherein the operation of selectively applying the dropout parameter to the one or more feature vectors includes selectively applying the dropout parameter to one or more tokens generated from the one or more feature vectors and matched via a gazetteer.