Methods and systems for generating mobile enabled extraction models
A computing system generates a plurality of training data sets for generating the NLP model. The computing system trains a teacher network to extract and classify tokens from a document. The training includes a pre-training stage where the teacher network is trained to classify generic data in the plurality of training data sets and a fine-tuning stage where the teacher network is trained to classify targeted data in the plurality of training data sets. The computing system trains a student network to extract and classify tokens from a document by distilling knowledge learned by the teacher network during the fine-tuning stage from the teacher network to the student network. The computing system outputs the NLP model based on the training. The computing system causes the NLP model to be deployed in a remote computing environment.
1. A computer-implemented method comprising:
generating a plurality of training data sets for generating a machine learning model;
training a teacher network, the training comprising a first stage where the teacher network is trained to classify a first data in the plurality of training data sets and a second stage where the teacher network is trained to classify a second data in the plurality of training data sets;
training a student network by distilling knowledge learned by the teacher network during the second stage from the teacher network to the student network, wherein distilling knowledge comprises implementing a distillation loss using weights that are same; and
outputting the machine learning model based on the training of the student network.
2. The computer-implemented method of claim 1 , wherein the teacher network is a bidirectional encoder representations for transformer (BERT) model and wherein the student network is a MobileBERT model.
3. The computer-implemented method of claim 1 , wherein generating the plurality of training data sets for generating the machine learning model, comprises:
generating a first plurality of training data sets using a text corpus for a generic training process; and
generating a second plurality of training data sets using targeted data for which the machine learning model is optimized.
4. The computer-implemented method of claim 1 , wherein training the student network from a document by distilling knowledge learned by the teacher network during the second stage from the teacher network to the student network comprises:
training the student network to approximate an original function learned by the teacher network during training of the teacher network.
5. The computer-implemented method of claim 1 , wherein training the student network from a document by distilling knowledge learned by the teacher network during the second stage from the teacher network to the student network comprises:
classifying tokens for subsequent mapping of tokens to one or more of a plurality of input data fields.
6. The computer-implemented method of claim 1 , wherein training a student network from a document by distilling knowledge learned by the teacher network during the second stage from the teacher network to the student network comprises:
inputting soft labels generated by the teacher network during training of the teacher network into the student network.
7. The computer-implemented method of claim 1 , wherein outputting the machine learning model based on the training, comprises:
compressing a trained student model into the machine learning model using dynamic quantization.
8. A non-transitory computer readable medium having one or more sequences of instructions, which, when executed by a processor, causes operations comprising:
generating a plurality of training data sets for generating a machine learning model;
training a teacher network, the training comprising a first stage where the teacher network is trained to classify a first data in the plurality of training data sets and a second stage where the teacher network is trained to classify a second data in the plurality of training data sets;
training a student network by distilling knowledge learned by the teacher network during the second stage from the teacher network to the student network, wherein distilling knowledge comprises implementing a distillation loss using weights that are same; and
outputting the machine learning model based on the training of the student network.
9. The non-transitory computer readable medium of claim 8 , wherein the teacher network is a bidirectional encoder representations for transformer (BERT) model and wherein the student network is a MobileBERT model.
10. The non-transitory computer readable medium of claim 8 , wherein generating the plurality of training data sets for generating the machine learning model, comprises:
generating a first plurality of training data sets using a text corpus for a generic training process; and
generating a second plurality of training data sets using targeted data for which the machine learning model is optimized.
11. The non-transitory computer readable medium of claim 8 , wherein training the student network from a document by distilling knowledge learned by the teacher network during the second stage from the teacher network to the student network comprises:
training the student network to approximate an original function learned by the teacher network during training of the teacher network.
12. The non-transitory computer readable medium of claim 8 , wherein training the student network from a document by distilling knowledge learned by the teacher network during the second stage from the teacher network to the student network comprises:
classifying tokens for subsequent mapping of tokens to one or more of a plurality of input data fields.
13. The non-transitory computer readable medium of claim 8 , wherein training a student network from a document by distilling knowledge learned by the teacher network during the second stage from the teacher network to the student network comprises:
inputting soft labels generated by the teacher network during training of the teacher network into the student network.
14. The non-transitory computer readable medium of claim 8 , wherein outputting the machine learning model based on the training, comprises:
compressing a trained student model into the machine learning model using dynamic quantization.
15. A system comprising:
a processor; and
a memory having one or more instructions stored thereon, which, when executed by the processor, causes the system to perform operations comprising:
generating a plurality of training data sets for generating a machine learning model;
training a teacher network, the training comprising a first stage where the teacher network is trained to classify a first data in the plurality of training data sets and a second stage where the teacher network is trained to classify a second data in the plurality of training data sets;
training a student network by distilling knowledge learned by the teacher network during the second stage from the teacher network to the student network, wherein distilling knowledge comprises implementing a distillation loss using weights that are same; and
outputting the machine learning model based on the training of the student network.
16. The system of claim 15 , wherein the teacher network is a bidirectional encoder representations for transformer (BERT) model and wherein the student network is a MobileBERT model.
17. The system of claim 15 , wherein generating the plurality of training data sets for generating the machine learning model, comprises:
generating a first plurality of training data sets using a text corpus for a generic training process; and
generating a second plurality of training data sets using targeted data for which the machine learning model is optimized.
18. The system of claim 15 , wherein training the student network from a document by distilling knowledge learned by the teacher network during the second stage from the teacher network to the student network comprises:
training the student network to approximate an original function learned by the teacher network during training of the teacher network.
19. The system of claim 15 , wherein training the student network from a document by distilling knowledge learned by the teacher network during the second stage from the teacher network to the student network comprises:
classifying tokens for subsequent mapping of tokens to one or more of a plurality of input data fields.
20. The system of claim 15 , wherein training a student network from a document by distilling knowledge learned by the teacher network during the second stage from the teacher network to the student network comprises:
inputting soft labels generated by the teacher network during training of the teacher network into the student network.