Image processing apparatus and image processing method
An image processing apparatus has a first image acquisitor that acquires a source image, a second image acquisitor that acquires a first target image, a label acquisitor that acquires a label, a feature extractor including a first neural network that extracts a feature of the source image and a feature of the first target image, a class classifier including a second neural network that performs a class classification of the source image and the first target image, a domain classifier including a third neural network that performs a domain classification of the source image and the first target image, a processor that assigns a pseudo label to the first target image, a self-learner that performs a self-learning of the first neural network, the second neural network, and the third neural network, and a learner that learns the first, second and third neural networks, by performing a back propagation process.
1. An image processing apparatus comprising:
a first image acquisitor configured to acquire a source image to which a label is assigned;
a second image acquisitor configured to acquire a first target image to which no label is assigned;
a label acquisitor configured to acquire a label;
a feature extractor including a first neural network configured to extract a feature of the source image and a feature of the first target image;
a class classifier including a second neural network configured to perform a class classification of the source image and the first target image based on a plurality of the features extracted by the feature extractor;
a domain classifier including a third neural network configured to perform a domain classification of the source image and the first target image based on the feature extracted by the feature extractor;
a processor configured to assign a pseudo label to the first target image using the class classifier including the second neural network in a middle of learning;
a self-learner configured to perform a self-learning of the first neural network, the second neural network, and the third neural network based on a feature obtained by inputting the first target image to which the pseudo label is assigned to the feature extractor; and
a learner configured to learn the first neural network, the second neural network, and the third neural network by performing a back propagation process based on a classification result by the class classifier, a classification result by the domain classifier, and a self-learning result by the self-learner.
2. The image processing apparatus according to claim 1 , further comprising:
a first loss calculator configured to calculate a first loss representing reliability of the class classification of the source image by the class classifier;
a second loss calculator configured to calculate a second loss representing reliability of the domain classification of the source image and the first target image by the domain classifier; and
a third loss calculator configured to calculate a third loss representing reliability of the class classification of the first target image to which the pseudo label is assigned, wherein
the self-learner is configured to input the first target image to which the pseudo label is assigned to the feature extractor and is configured to cause the third loss calculator to calculate the third loss, and
the learner is configured to learn the first neural network, the second neural network, and the third neural network by performing the back propagation process based on the first loss, the second loss, and the third loss.
3. The image processing apparatus according to claim 1 , further comprising:
a first loss calculator configured to calculate a first loss representing reliability of the class classification of the source image by the class classifier;
a second loss calculator configured to calculate a second loss representing reliability of the domain classification of the source image and the first target image by the domain classifier; and
a third loss calculator configured to calculate a third loss representing reliability of the class classification of the first target image by the class classifier, wherein
the self-learner is configured to input the first target image to which the pseudo label is assigned to the feature extractor and is configured to cause the third loss calculator to calculate the third loss, and
the learner comprises a first learner configured to perform the back propagation process based on the first loss and the second loss to learn the first neural network, the second neural network, and the third neural network, and a second learner configured to perform the back propagation process based on the third loss to learn the first neural network and the second neural network.
4. The image processing apparatus according to claim 2 , wherein
the self-learner is configured to stop the self-learning when the second loss is equal to or greater than a threshold value after starting the self-learning.
5. The image processing apparatus according to claim 2 , further comprising:
a weight loss generator configured to adjust weights of the first loss, the second loss, and the third loss when the back propagation process is performed to generate a weight loss, wherein
the learner is further configured to perform the back propagation process based on the weight loss to learn the first neural network, the second neural network, and the third neural network.
6. The image processing apparatus according to claim 5 , wherein
the weight loss generator is configured to generate the weight loss by lowering weights of the first loss and the second loss and raising a weight of the third loss as the self-learning by the self-learner progresses.
7. The image processing apparatus according to claim 2 , wherein
with an image group including a plurality of the source images and a plurality of the first target images being set as one epoch, the class classifier and the domain classifier are configured to perform class classification and domain classification for a plurality of epochs, and
the self-learner is configured to start labeling the first target image and the self-learning in a case where the first loss is equal to or less than a threshold value or in a case where the number of the processed epochs exceeds a predetermined ratio with respect to a total number of the epochs.
8. The image processing apparatus according to claim 1 , further comprising:
a third image acquisitor configured to acquire a second target image to which a label is assigned, the number of the second target images being smaller than the number of the first target images, wherein
the feature extractor is configured to extract a feature of the second target image,
the class classifier is configured to perform the class classification of the second target image based on the feature of the second target image extracted by the feature extractor,
the domain classifier is configured to perform the domain classification of the second target image based on the feature of the second target image extracted by the feature extractor, and
the learner is configured to learn the first neural network, the second neural network, and the third neural network based on classification results of classes and domains of the source image, the first target image, and the second target image.
9. The image processing apparatus according to claim 8 , further comprising:
a padder configured to increase the number of the second target images so that the number of the second target images acquired by the third image acquisitor is equal between classes.
10. The image processing apparatus according to claim 1 , wherein
the self-learner is configured to perform the self-learning by inputting, to the feature extractor, the first target image, as teacher data, in which a certainty factor of the pseudo label is equal to or greater than a threshold value among the first target images to each of which the pseudo label is assigned by the class classifier.
11. The image processing apparatus according to claim 1 , wherein
the class classifier and the domain classifier are configured to perform class classification and domain classification for each image group including a plurality of the source images and a plurality of the first target images, and
the self-learner is configured to determine whether to assign the pseudo label to the first target image every time one of the first target images in the image group is input to the second image acquisitor, and is configured to perform the self-learning based on the first target image to which the pseudo label is assigned.
12. The image processing apparatus according to claim 1 , wherein
the class classifier and the domain classifier are configured to perform class classification and domain classification for each image group including a plurality of the source images and a plurality of the first target images, and
the self-learner is configured to assign, without referring to a pseudo label assigned to the first target image in the one image group, a pseudo label to the first target image in the another image group.
13. The image processing apparatus according to claim 1 , wherein
the class classifier is configured to output a Softmax value obtained by quantifying a certainty factor with which the first target image is classified into each of a plurality of classes, and
in a case where a maximum value of the Softmax value exceeds a threshold value, the self-learner is configured to assign the pseudo label associated with a class having the maximum value.
14. The image processing apparatus according to claim 1 , wherein
the class classifier is configured to output a Softmax value obtained by quantifying a certainty factor with which the first target image is classified into each of a plurality of classes, and
in a case where a maximum value of the Softmax value is equal to or less than a threshold value, the self-learner is configured to perform the self-learning by masking a value related to the maximum value of the Softmax value.
15. The image processing apparatus according to claim 1 , wherein
the class classifier is configured to output a Softmax value obtained by quantifying a certainty factor with which the first target image is classified into each of a plurality of classes, and
in a case where a maximum value of the Softmax value is equal to or less than a threshold value, the self-learner is configured to perform the self-learning by masking values related to all the Softmax values associated with the plurality of classes each having the Softmax value.
16. The image processing apparatus according to claim 1 , wherein
the class classifier and the domain classifier are configured to perform class classification and domain classification for each image group including a plurality of the source images and a plurality of the first target images, and
the self-learner is configured to adjust, among a plurality of the first target images included in the image group, the number of first target images, for each class, to each of which the pseudo label is assigned.
17. An image processing apparatus comprising:
a first image acquisitor configured to acquire a source image to which a label is assigned;
a second image acquisitor configured to acquire a first target image to which no label is assigned;
a third image acquisitor configured to acquire a second target image to which a label is assigned, the number of the second target images being smaller than the number of the first target images;
a label acquisitor configured to acquire a label;
a feature extractor including a first neural network configured to extract a feature of the source image, a feature of the first target image, and a feature of the second target image;
a class classifier including a second neural network configured to perform class classification of the source image and the second target image based on a plurality of the features extracted by the feature extractor;
a domain classifier including a third neural network configured to perform domain classification of the source image, the first target image, and the second target image; and
a learner configured to learn the first neural network, the second neural network, and the third neural network by performing a back propagation process based on a classification result by the class classifier and a classification result by the domain classifier.
18. The image processing apparatus according to claim 17 , further comprising:
a first loss calculator configured to calculate a first loss representing reliability of the class classification of the source image and the second target image by the class classifier; and
a second loss calculator configured to calculate a second loss representing reliability of the domain classification of the source image, the first target image, and the second target image by the domain classifier, wherein
the learner is configured to learn the first neural network, the second neural network, and the third neural network by performing a back propagation process based on the first loss and the second loss.
19. The image processing apparatus according to claim 18 , further comprising:
a processor configured to assign a pseudo label to the first target image using the class classifier including the second neural network in a middle of learning; and
a self-learner configured to perform self-learning of the first neural network, the second neural network, and the third neural network by inputting the first target image to which the pseudo label is assigned to the feature extractor.
20. The image processing apparatus according to claim 19 , further comprising:
a third loss calculator configured to calculate a third loss representing reliability of the class classification of the first target image by the class classifier, wherein
the self-learner is configured to input the first target image to which the pseudo label is assigned to the feature extractor and is configured to cause the third loss calculator to calculate the third loss, and
the learner is configured to learn the first neural network, the second neural network, and the third neural network by performing a back propagation process based on the first loss, the second loss, and the third loss.