Data processing method, apparatus, device, storage medium and program product
This application discloses a data processing method performed by a computer device. Obtained service data is inputted into a pre-trained model and an initial service model to obtain first fully-connected layer outputs of the pre-trained model and second fully-connected layer outputs of the initial service model. The pre-trained model and the initial service model construct modeling units with different granularities, first modeling units of the pre-trained model are matched with second modeling units of the initial service model, fully-connected layer outputs of the second modeling units have the same sequence length as fully-connected layer outputs of matched target first modeling units so that with the assistance of the pre-trained model, the target service model can be obtained through the initial service model, the knowledge distillation of the models including modeling units with different granularities is realized, thereby effectively improving the service performance of the service model.
1 . A method performed by a computer device, the method comprising:
obtaining service data;
inputting the service data into a pre-trained model and an initial service model to obtain first fully-connected layer outputs of the pre-trained model and second fully-connected layer outputs of the initial service model, wherein:
each of first modeling units of the pre-trained model is a single character,
each of second modeling units of the initial service model is a phrase,
the pre-trained model is obtained by training based on both pre-training data and the service data, and
the initial service model is obtained by training based on the service data;
determining, from the first modeling units, target first modeling units matched with the second modeling units respectively, wherein fully-connected layer outputs of the second modeling units and fully-connected layer outputs of the matched target first modeling units have a same sequence length; and
performing knowledge distillation by utilizing the second fully-connected layer outputs of the initial service model and the fully-connected layer outputs corresponding to the matched target first modeling units in the first fully-connected layer outputs of the pre-trained model to obtain a target service model.
2 . The method according to claim 1 , wherein the service data is a string of text, the first modeling units of the pre-trained model have a granularity of a single character and the second modeling units of the initial service model have a granularity of a phrase.
3 . The method according to claim 1 , wherein the service data is a string of text without punctuations, the method further comprises:
adding punctuations to the string of text using the target service model.
4 . The method according to claim 1 , where the determining, from the first modeling units, the target first modeling unit matched with the second modeling units respectively comprises:
matching, according to a target phrase corresponding to a target second modeling unit, a target character set constituting the target phrase from the first modeling units, the target character set containing a plurality of characters; and
determine, from the target character set, a first modeling unit corresponding to the last character as a target first modeling unit matched with the target second modeling unit, based on a word order.
5 . The method according to claim 1 , wherein the service data is voice data and the method further comprises:
obtaining voice data;
recognizing the voice data to obtain a target text corresponding to the voice data;
adding punctuations to the target text using the target service model;
outputting the target text added with the punctuations.
6 . The method according to claim 1 , wherein the pre-trained model is obtained by self-supervised training based on pre-training data, and a data volume of the pre-training data is greater than a data volume of service data required for training to obtain the target service model.
7 . A computer device comprising:
a processor, a communication interface, a memory and a communication bus;
the processor, the communication interface and the memory communicating with each other through the communication bus; the communication interface being an interface of a communication module;
the memory being configured to store a program code and transmit the program code to the processor;
the processor being configured to invoke instructions in the memory to perform a data processing method including:
obtaining service data;
inputting the service data into a pre-trained model and an initial service model to obtain first fully-connected layer outputs of the pre-trained model and second fully-connected layer outputs of the initial service model, wherein:
each of first modeling units of the pre-trained model is a single character,
each of second modeling units of the initial service model is a phrase,
the pre-trained model is obtained by training based on both pre-training data and the service data, and
the initial service model is obtained by training based on the service data;
determining, from the first modeling units, target first modeling units matched with the second modeling units respectively, wherein fully-connected layer outputs of the second modeling units and fully-connected layer outputs of the matched target first modeling units have a same sequence length; and
performing knowledge distillation by utilizing the second fully-connected layer outputs of the initial service model and the fully-connected layer outputs corresponding to the matched target first modeling units in the first fully-connected layer outputs of the pre-trained model to obtain a target service model.
8 . The computer device according to claim 7 , wherein the service data is a string of text, the first modeling units of the pre-trained model have a granularity of a single character and the second modeling units of the initial service model have a granularity of a phrase.
9 . The computer device according to claim 7 , wherein the service data is a string of text without punctuations, the data processing method further comprises:
adding punctuations to the string of text using the target service model.
10 . The computer device according to claim 7 , where the determining, from the first modeling units, the target first modeling unit matched with the second modeling units respectively comprises:
matching, according to a target phrase corresponding to a target second modeling unit, a target character set constituting the target phrase from the first modeling units, the target character set containing a plurality of characters; and
determine, from the target character set, a first modeling unit corresponding to the last character as a target first modeling unit matched with the target second modeling unit, based on a word order.
11 . The computer device according to claim 7 , wherein the service data is voice data and the method further comprises:
obtaining voice data;
recognizing the voice data to obtain a target text corresponding to the voice data;
adding punctuations to the target text using the target service model;
outputting the target text added with the punctuations.
12 . The computer device according to claim 7 , wherein the pre-trained model is obtained by self-supervised training based on pre-training data, and a data volume of the pre-training data is greater than a data volume of service data required for training to obtain the target service model.
13 . A non-transitory computer readable storage medium, having stored thereon a computer program that, when executed by a processor of a computer device, causes the computer device to perform a data processing method including:
obtaining service data;
inputting the service data into a pre-trained model and an initial service model to obtain first fully-connected layer outputs of the pre-trained model and second fully-connected layer outputs of the initial service model, wherein:
each of first modeling units of the pre-trained model is a single character,
each of second modeling units of the initial service model is a phrase,
the pre-trained model is obtained by training based on both pre-training data and the service data, and
the initial service model is obtained by training based on the service data;
determining, from the first modeling units, target first modeling units matched with the second modeling units respectively, wherein fully-connected layer outputs of the second modeling units and fully-connected layer outputs of the matched target first modeling units have a same sequence length; and
performing knowledge distillation by utilizing the second fully-connected layer outputs of the initial service model and the fully-connected layer outputs corresponding to the matched target first modeling units in the first fully-connected layer outputs of the pre-trained model to obtain a target service model.
14 . The non-transitory computer readable storage medium according to claim 13 , wherein the service data is a string of text, the first modeling units of the pre-trained model have a granularity of a single character and the second modeling units of the initial service model have a granularity of a phrase.
15 . The non-transitory computer readable storage medium according to claim 13 , wherein the service data is a string of text without punctuations, the data processing method further comprises:
adding punctuations to the string of text using the target service model.
16 . The non-transitory computer readable storage medium according to claim 13 , where the determining, from the first modeling units, the target first modeling unit matched with the second modeling units respectively comprises:
matching, according to a target phrase corresponding to a target second modeling unit, a target character set constituting the target phrase from the first modeling units, the target character set containing a plurality of characters; and
determine, from the target character set, a first modeling unit corresponding to the last character as a target first modeling unit matched with the target second modeling unit, based on a word order.
17 . The non-transitory computer readable storage medium according to claim 13 , wherein the service data is voice data and the method further comprises:
obtaining voice data;
recognizing the voice data to obtain a target text corresponding to the voice data;
adding punctuations to the target text using the target service model;
outputting the target text added with the punctuations.
18 . The non-transitory computer readable storage medium according to claim 13 , wherein the pre-trained model is obtained by self-supervised training based on pre-training data, and a data volume of the pre-training data is greater than a data volume of service data required for training to obtain the target service model.