Classification and prediction method and apparatus, device, storage medium, and computer program product
The present subject matter relates to the field of machine learning and provides a classification and prediction method and apparatus, a device, a storage medium, and a computer program product. At least two pieces of data are acquired for a specified classification task. Each piece of data corresponds to one modality. A confidence corresponding to each of at least two modalities is acquired. The confidence indicates a classification and prediction probability of the modality in the specified classification task. Weighted fusion on data features of the at least two pieces of data is performed based on the confidence corresponding to each of the at least two modalities to obtain a fused feature. A prediction is performed according to the fused feature to obtain a classification and prediction result corresponding to the specified classification task.
1. A classification and prediction method performed by a computer device, comprising:
acquiring at least two pieces of data for a specified classification task, wherein
the at least two pieces of data correspond to at least two modalities, and
each piece of data corresponds to one modality;
acquiring a confidence corresponding to each of the at least two modalities, wherein
the confidence indicates a classification and prediction probability of the modality in the specified classification task;
performing weighted fusion on data features of the at least two pieces of data based on the confidence corresponding to each of the at least two modalities to obtain a fused feature; and
performing prediction according to the fused feature to obtain a classification and prediction result corresponding to the specified classification task.
2. The method according to claim 1 , wherein the acquiring the confidence corresponding to each of the at least two modalities comprises:
inputting data corresponding to an i th modality of the at least two modalities into a confidence prediction network corresponding to the i th modality, wherein
the confidence prediction network is a pre-trained network configured to perform confidence analysis, and
i is a positive integer; and
performing the confidence analysis on the data corresponding to the i th modality through the confidence prediction network corresponding to the i th modality to obtain a confidence corresponding to the i th modality.
3. The method according to claim 2 , wherein before the inputting, the method further comprises:
acquiring a sample data set in the i th modality, the sample data set comprising sample data marked with a confidence label; and
training the confidence prediction network corresponding to the i th modality based on the sample data marked with the confidence label.
4. The method according to claim 3 , wherein the sample data is marked with a classification label; and the method further comprises:
performing initial classification and prediction on the sample data to obtain an initial classification and prediction result corresponding to the sample data;
expressing the classification label in a one-hot encoding form to obtain a one-hot encoding vector, wherein
the classification label corresponds to the sample data having a first value and other classification labels having a second value; and
determining the confidence label corresponding to the sample data based on the initial prediction classification result and the one-hot encoding vector.
5. The method according to claim 1 , wherein before the performing weighted fusion on data features of the at least two pieces of data to obtain a fused feature, the method further comprises:
extracting candidate data features of the at least two pieces of data; and
respectively performing significance analysis on the candidate data features of the at least two pieces of data based on a feature significance screening condition to obtain the data features corresponding to the at least two pieces of data.
6. The method according to claim 5 , wherein the respectively performing comprises:
inputting a candidate data feature of the data corresponding to the i th modality of the at least two modalities into a significance prediction network corresponding to the i th modality, wherein
the significance prediction network is a pre-trained network configured to perform significance analysis, and
i is a positive integer; and
performing the significance analysis on the candidate data feature of the data corresponding to the i th modality through the significance prediction network corresponding to the i th modality to obtain a data feature of the data corresponding to the i th modality.
7. The method according to claim 6 , wherein the performing the significance analysis on the candidate data feature of the data corresponding to the i th modality through the significance prediction network corresponding to the i th modality to obtain a data feature of the data corresponding to the i th modality comprises:
performing the significance analysis on the candidate data feature of the data corresponding to the i th modality through the significance prediction network corresponding to the i th modality to obtain a significance score of the candidate data feature of the data corresponding to the i th modality; and
obtaining the data feature of the data corresponding to the i th modality based on the significance score and the candidate data feature of the data corresponding to the i th modality.
8. The method according to claim 7 , wherein
the candidate data feature of the data corresponding to the i th modality comprises at least two feature elements, and
the significance score comprises significance sub-scores respectively corresponding to the at least two feature elements; and
the obtaining comprises:
determining one of the two feature elements whose significance sub-score reaches a preset gating threshold as the data feature of the data corresponding to the i th modality.
9. The method according to claim 1 , wherein the performing prediction according to the fused feature to obtain a classification and prediction result corresponding to the specified classification task comprises:
performing prediction according to the fused feature through a classification and prediction network to obtain the classification and prediction result corresponding to the specified classification task.
10. The method according to claim 9 , wherein before the performing prediction according to the fused feature through a classification and prediction network to obtain the classification and prediction result corresponding to the specified classification task, the method further comprises:
acquiring n pieces of sample data marked with classification labels, wherein
the n pieces of sample data corresponding to the at least two modalities,
each piece of sample data corresponding to one modality, and
n is a positive integer;
acquiring the confidence corresponding to each of the at least two modalities, wherein the confidence indicates the classification and prediction probability of the modality;
fusing data features of the n pieces of sample data based on the confidence corresponding to each of the at least two modalities, to obtain a fused feature; and
training a candidate classification model based on the fused feature and the classification labels to obtain the classification and prediction network.
11. The method according to claim 10 , wherein the training a candidate classification model based on the fused feature and the classification labels to obtain the classification and prediction network comprises:
performing classification and prediction on the fused feature through the candidate classification model to obtain the classification and prediction result;
determining a loss value corresponding to the candidate classification model based on the classification and prediction result and the classification labels; and
adjusting a model parameter of the candidate classification model based on the loss value to obtain the classification and prediction network.
12. A classification and prediction apparatus, comprising:
an acquisition module configured to acquire at least two pieces of data for a specified classification task, wherein
the at least two pieces of data corresponding to at least two modalities,
each piece of data corresponding to one modality, and
the acquisition module is further configured to acquire a confidence corresponding to each of the at least two modalities, wherein
the confidence indicates a classification and prediction probability of the modality in the specified classification task;
a fusion module configured to perform weighted fusion on data features of the at least two pieces of data based on the confidence corresponding to each of the at least two modalities to obtain a fused feature; and
a prediction module configured to perform prediction according to the fused feature to obtain a classification and prediction result corresponding to the specified classification task.
13. The apparatus according to claim 12 , wherein the acquisition module comprises:
an input unit configured to input data corresponding to an i th modality of the at least two modalities into a confidence prediction network corresponding to the i th modality, wherein
the confidence prediction network is a pre-trained network configured to perform confidence analysis, and
i is a positive integer; and
an analysis unit configured to perform the confidence analysis on the data corresponding to the i th modality through the confidence prediction network corresponding to the i th modality to obtain a confidence corresponding to the i th modality.
14. The apparatus according to claim 13 , wherein the acquisition module further comprises:
an acquisition unit configured to acquire a sample data set in the i th modality, wherein the sample data set comprises sample data marked with a confidence label; and
a prediction unit configured to train the confidence prediction network corresponding to the i th modality based on the sample data marked with the confidence label.
15. The apparatus according to claim 14 , wherein
the prediction module is further configured to perform initial classification and prediction on the sample data to obtain an initial classification and prediction result corresponding to the sample data; and the apparatus further comprises:
an expression module configured to express the classification label in a one-hot encoding form to obtain a one-hot encoding vector, wherein
the classification label corresponding to the sample data having a first value and other classification labels having a second value; and
a determination module configured to determine the confidence label corresponding to the sample data based on the initial prediction classification result and the one-hot encoding vector.
16. The apparatus according to claim 12 , further comprising:
an extraction module configured to extract candidate data features of the at least two pieces of data; and
an analysis module configured to respectively perform significance analysis on the candidate data features of the at least two pieces of data based on a feature significance screening condition to obtain the data features corresponding to the at least two pieces of data.
17. The apparatus according to claim 16 , wherein the analysis module is further configured to:
input a candidate data feature of the data corresponding to the i th modality of the at least two modalities into a significance prediction network corresponding to the i th modality, wherein
the significance prediction network is a pre-trained network configured to perform significance analysis, and
i is a positive integer; and
perform the significance analysis on the candidate data feature of the data corresponding to the i th modality through the significance prediction network corresponding to the i th modality to obtain a data feature of the data corresponding to the i th modality.
18. A computer device comprising a processor and a memory storing program instructions loadable and executable by the processor to implement the classification and prediction method according to claim 1 .
19. A non-transitory computer-readable storage medium storing program instructions executable by a processor to implement the classification and prediction method according to claim 1 .