Method of knowledge transferring, information processing apparatus and storage medium
There are provided a method of knowledge transferring, an information processing apparatus and a storage medium. The method of knowledge transferring includes: obtaining a first model which has been trained in advance with respect to a predetermined task; and training a second model with respect to the predetermined task by utilizing a comprehensive loss function, such that the second model has knowledge of the first model. The comprehensive loss function is based on a first loss function weighted by accuracy of an output result of the first model for a training sample in regard to the predetermined task, and a second loss function. The first loss function represents a difference between processing results of the second model and the first model for the training sample. The second loss function represents accuracy of an output result of the second model for the training sample in regard to the predetermined task.
1. A method of knowledge transferring, including:
obtaining a first model which has been trained in advance with respect to a predetermined task; and
training a second model with respect to the predetermined task by utilizing a comprehensive loss function, such that the second model has knowledge of the first model,
wherein the comprehensive loss function is based on a first loss function and a second loss function, the first loss function representing differences between respective processing results of the second model and the first model for a plurality of training samples, the differences being weighted by weights each of which is accuracy of an output result of the first model for a corresponding one in the plurality of training samples in regard to the predetermined task, and the second loss function representing accuracy of each output result of the second model for each one of the plurality of training samples in regard to the predetermined task, and
wherein, for each training sample of the plurality of training samples, when a training sample is correctly classified by the first model, accuracy of an output result of the first model for the training sample in regard to the predetermined task that is used as the weight in the first loss function reaches a maximum value.
2. The method of knowledge transferring according to claim 1 , wherein the first loss function includes a first sub-loss function representing a difference between a predetermined feature extracted from one of the training samples by the second model and a predetermined feature extracted from the training sample by the first model, the first sub-loss function being weighted by the weight which is accuracy of the output result of the first model for the training sample in regard to the predetermined task.
3. The method of knowledge transferring according to claim 2 , wherein the first loss function further includes a second sub-loss function representing a difference between output results of the second model and the first model for the training sample in regard to the predetermined task, the second sub-loss function being weighted by the weight which is accuracy of the output result of the first model for the training sample in regard to the predetermined task.
4. The method of knowledge transferring according to claim 2 , wherein the training sample includes one of an image sample, a speech sample, or a document sample.
5. The method of knowledge transferring according to claim 2 , wherein when the accuracy of the output result of the first model for the training sample in regard to the predetermined task is higher, the weight in the first loss function is greater.
6. The method of knowledge transferring according to claim 2 , wherein the predetermined task includes a classification task.
7. The method of knowledge transferring according to claim 6 , wherein the first sub-loss function is based on a cosine distance between the predetermined feature extracted from the training sample by the second model and the predetermined feature extracted from the training sample by the first model.
8. An information processing apparatus, comprising:
a processor configured to:
input data for process into the second model which is obtained by the method of knowledge transferring according to claim 1 ; and
obtain a processing result of the second model for the data for process.
9. The information processing apparatus according to claim 8 , wherein the data for process includes one of image data, speech data, or document data.
10. The information processing apparatus according to claim 8 , wherein the first loss function includes a first sub-loss function representing a difference between a predetermined feature extracted from one of the training samples by the second model and a predetermined feature extracted from the training sample by the first model, the first sub-loss function being weighted by the weight which is accuracy of the output result of the first model for the training sample in regard to the predetermined task.
11. The information processing apparatus according to claim 9 , wherein the first loss function further includes a second sub-loss function representing a difference between output results of the second model and the first model for the training sample in regard to the predetermined task, the second sub-loss function being weighted by the weight which is accuracy of the output result of the first model for the training sample in regard to the predetermined task.
12. The information processing apparatus according to claim 8 , wherein when the accuracy of the output result of the first model for the training sample in regard to the predetermined task is higher, the weight in the first loss function is greater.
13. The information processing apparatus according to claim 8 , wherein the predetermined task includes a classification task, the first loss function is based on cosine distances between predetermined features extracted from the training samples by the second model and a predetermined features extracted from the training samples by the first model, and
the processor is configured to extract a predetermined feature from the data for process by the second model, as the processing result for the data for process, and is further configured to calculate a cosine similarity between the predetermined feature extracted from the data for process and a predetermined feature extracted from target data by the second model, to determine a similarity between the data for process and the target data based on the cosine similarity.
14. A non-transitory storage medium having stored thereon machine readable instruction code that, when read and executed by a machine, causes the machine to implement a method of knowledge transferring, comprising:
obtaining a first model which has been trained in advance with respect to a predetermined task; and
training a second model with respect to the predetermined task by utilizing a comprehensive loss function, such that the second model has knowledge of the first model,
wherein the comprehensive loss function is based on a first loss function and a second loss function, the first loss function representing differences between respective processing results of the second model and the first model for a plurality of training samples, the differences being weighted by weights each of which is accuracy of an output result of the first model for a corresponding one in the plurality of training samples in regard to the predetermined task, and the second loss function representing accuracy of each output result of the second model for each one of the plurality of training samples in regard to the predetermined task, and
wherein, for each training sample of the plurality of training samples, when a training sample is correctly classified by the first model, accuracy of an output result of the first model for the training sample in regard to the predetermined task that is used as the weight in the first loss function reaches a maximum value.
15. The non-transitory storage medium according to claim 14 , wherein the first loss function includes a first sub-loss function representing a difference between a predetermined feature extracted from one of the training samples by the second model and a predetermined feature extracted from the training sample by the first model, the first sub-loss function being weighted by the weight which is accuracy of the output result of the first model for the training sample in regard to the predetermined task.
16. The non-transitory storage medium according to claim 15 , wherein the first loss function further includes a second sub-loss function representing a difference between output results of the second model and the first model for the training sample in regard to the predetermined task, the second sub-loss function being weighted by the weight which is accuracy of the output result of the first model for the training sample in regard to the predetermined task.
17. The non-transitory storage medium according to claim 15 , wherein the training sample includes one of an image sample, a speech sample, or a document sample.
18. The non-transitory storage medium according to claim 15 , wherein when the accuracy of the output result of the first model for the training sample in regard to the predetermined task is higher, the weight in the first loss function is greater.
19. The non-transitory storage medium according to claim 15 , wherein the predetermined task includes a classification task.
20. The non-transitory storage medium according to claim 19 , wherein the first sub-loss function is based on a cosine distance between the predetermined feature extracted from the training sample by the second model and the predetermined feature extracted from the training sample by the first model.