Compression of models for natural language processing
An example electronic computing device can include: a processor; and a system memory, the system memory including instructions which, when executed by the processor, cause the electronic computing device to: receive a model for natural language processing of data, the model including a plurality of self-attention heads; prune the model by removing one or more of the plurality of self-attention heads of the model to create a pruned model; and evaluate a classification accuracy of the pruned model to maintain a performance level.
1. An electronic computing device, comprising:
a processor; and
a system memory, the system memory including instructions which, when executed by the processor, cause the electronic computing device to:
define a budget that quantifies a maximum amount of a classification accuracy that is sacrificed;
receive a model for natural language processing of data, the model including a plurality of self-attention heads;
prune the model by removing one or more of the plurality of self-attention heads of the model to create a pruned model based on a computed cost of pruning the one or more of the plurality of self-attention heads of the model;
evaluate a classification accuracy of the pruned model to maintain a performance level;
determine whether remaining self-attention heads of the plurality of self-attention heads should be pruned based on the budget and the classification accuracy of the pruned model; and
when the remaining self-attention heads of the plurality of self-attention heads should be pruned:
determine, using a heuristic, an estimated cost of pruning the remaining self-attention heads in a next iteration of pruning, wherein the heuristic determines the estimated cost of pruning the remaining self-attention heads in the next iteration of pruning based on the computed cost in a current iteration; and
update the computed cost of pruning for the remaining self-attention heads for which the estimated cost of pruning is less than a remaining amount of the budget.
2. The electronic computing device of claim 1 , wherein the model is a Bidirectional Encoder Representations from Transformers model or a Generative Pre-trained Transformer model.
3. The electronic computing device of claim 1 , further comprising instructions which, when executed by the processor, cause the electronic computing device to train the model using a given training dataset.
4. The electronic computing device of claim 1 , further comprising instructions which, when executed by the processor, cause the electronic computing device to use an A* algorithm to prune the model.
5. The electronic computing device of claim 4 , wherein the A* algorithm is a search heuristic algorithm.
6. The electronic computing device of claim 1 , further comprising instructions which, when executed by the processor, cause the electronic computing device to iterate through each of the plurality of self-attention heads of the model when pruning the model.
7. The electronic computing device of claim 6 , further comprising instructions which, when executed by the processor, cause the electronic computing device to calculate a performance cost associated with pruning the model.
8. The electronic computing device of claim 1 , further comprising instructions which, when executed by the processor, cause the electronic computing device to compare a pruned classification accuracy of the model after pruning to an original classification accuracy of the model before pruning, to determine the performance level of the model.
9. The electronic computing device of claim 1 , wherein the budget defines a boundary for the classification accuracy.
10. A method for compressing a model, the method comprising:
defining a budget that quantifies a maximum amount of a classification accuracy that is sacrificed,
receiving a model for natural language processing of data, the model including a plurality of self-attention heads;
pruning the model by removing one or more of the plurality of self-attention heads of the model to create a pruned model based on a computed cost of pruning for said one or more of the plurality of self-attention heads of the model;
evaluating a classification accuracy of the pruned model to maintain a performance level; and
determining whether remaining self-attention heads of the plurality of self-attention heads should be pruned based on the budget and the classification accuracy of the pruned model;
when the remaining self-attention heads of the plurality of self-attention heads should be pruned:
determining, using a heuristic, an estimated cost of pruning the remaining self-attention heads in a next iteration of pruning, wherein the heuristic determines the estimated cost of pruning the remaining self-attention heads in the next iteration of pruning based on the computed cost in a current iteration; and
updating the computed cost of pruning for the remaining self-attention heads for which the estimated cost of pruning is less than a remaining amount of the budget.
11. The method of claim 10 , wherein the model is a Bidirectional Encoder Representations from Transformers model or a Generative Pre-trained Transformer model.
12. The method of claim 10 , further comprising training the model using a given training dataset.
13. The method of claim 10 , further comprising using an A* algorithm to prune the model.
14. The method of claim 13 , wherein the A* algorithm is a search heuristic algorithm.
15. The method of claim 10 , further comprising iterating through each of the plurality of self-attention heads of the model when pruning.
16. The method of claim 15 , further comprising calculating a performance cost associated with pruning the model.
17. The method of claim 10 , further comprising comparing a pruned classification accuracy of the model after pruning to an original classification accuracy of the model before pruning to determine the performance level.
18. The method of claim 10 , wherein the budget defines a boundary for the classification accuracy.