Domain specific model compression

1. A computer implemented method for domain specific model compression, the method comprising:

providing, by one or more computer processors, a pre-trained neural network model;

providing, by the one or more computer processors, an additional weighting parameter for a candidate operation of the pre-trained neural network model;

applying, by the one or more computer processors, the weighting parameter to an output vector of the candidate operation;

performing, by the one or more computer processors, a regularization of the application of the weighting parameter to the output vector, wherein the regularization comprises:

adding a penalty term to a loss function, the penalty term associated with the weighting parameter, wherein the penalty term includes one of an absolute value of the additional weighting parameter and the square of the additional weighting parameter;

compressing, by the one or more computer processors, the neural network model according to a result of the regularization, by removing candidate operations having a zero-value weighting parameter; and

providing, by the one or more computer processors, the neural network model after compression.

2. The computer implemented method according to claim 1 , wherein the neural network model comprises a language processing model.

3. The computer implemented method according to claim 1 , further comprising training, by the one or more computer processors, the neural network using unlabeled domain data.

4. The computer implemented method according to claim 1 , further comprising training, by the one or more computer processors, the neural network with two objectives, wherein one objective comprises a domain classification task.

5. The computer implemented method according to claim 1 , further comprising training, by the one or more computer processors, the neural network using labeled domain data.

6. The computer implemented method according to claim 1 , further comprising reducing, by the one or more computer processors, a neural network model attention head weighting value to zero.

7. The computer implemented method according to claim 1 , further comprising:

training, by the one or more computer processors, the neural network with two objectives, wherein one objective comprises a domain classification task; and

training, by the one or more computer processors, the neural network using labeled data.

8. A computer program product for domain specific model compression, the computer program product comprising one or more computer readable storage devices and stored program instructions on the one or more computer readable storage devices, the stored program instructions comprising:

program instructions to provide a pre-trained neural network model;

program instructions to provide an additional weighting parameter for a candidate operation of the neural network model;

program instructions to apply the weighting parameter to an output vector of the candidate operation;

program instructions to perform a regularization of the application of the weighting parameter to the output vector wherein the regularization comprises:

adding a penalty term to a loss function, the penalty term associated with the weighting parameter, wherein the penalty term includes one of an absolute value of the additional weighting parameter and the square of the additional weighting parameter;

program instructions to compress the neural network model according to a result of the regularization, by removing candidate operations having a zero-value weighting parameter; and

program instructions to provide the neural network model after compression.

9. The computer program product according to claim 8 , wherein the neural network model comprises a language processing model.

10. The computer program product according to claim 8 , further comprising program instructions to train the neural network using unlabeled domain data.

11. The computer program product according to claim 8 , further comprising program instructions to train the neural network with two objectives, wherein one objective comprises a domain classification task.

12. The computer program product according to claim 8 , further comprising program instructions to train the neural network using labeled domain data.

13. The computer program product according to claim 8 , further comprising program instructions to reduce a neural network model attention head weighting value to zero.

14. The computer program product according to claim 8 , further comprising program instructions to:

train the neural network with two objectives, wherein one objective comprises a domain classification task; and

train the neural network using labeled data.

15. A computer system for domain specific model compression, the computer system comprising:

one or more computer processors;

one or more computer readable storage devices; and

stored program instructions on the one or more computer readable storage devices for execution by the one or more computer processors, the stored program instructions comprising:

program instructions to provide a pre-trained neural network model;

program instructions to provide an additional weighting parameter for a candidate operation of the neural network model;

program instructions to apply the weighting parameter to an output vector of the candidate operation;

program instructions to perform a regularization of the application of the weighting parameter to the output vector wherein the regularization comprises:

adding a penalty term to a loss function, the penalty term associated with the weighting parameter, wherein the penalty term includes one of an absolute value of the additional weighting parameter and the square of the additional weighting parameter;

program instructions to compress the neural network model according to a result of the regularization, by removing candidate operations having a zero-value weighting parameter; and

program instructions to provide the neural network model after compression.

16. The computer system according to claim 15 , wherein the neural network model comprises a language processing model.

17. The computer system according to claim 15 , further comprising program instructions to train the neural network using unlabeled domain data.

18. The computer system according to claim 15 , further comprising program instructions to train the neural network with two objectives, wherein one objective comprises a domain classification task.

19. The computer system according to claim 15 , further comprising program instructions to train the neural network using labeled domain data.

20. The computer system according to claim 15 , further comprising program instructions to reduce a neural network model attention head weighting value to zero.