Image recognition method and electronic apparatus thereof
An image recognition method and an electronic apparatus configured for image recognition are provided. A training sample set is provided to train a recognition model including neural networks to recognize a classification label to which an image to be tested belongs through the trained recognition model. The training sample set includes image sets respectively belonging to users. During the training process, training images corresponding to classification labels are obtained from a first image set in the training sample set as reference images for training; a training image is obtained from a second image set different from the first image set as an input image for training; the reference images for training and the input image for training are obtained as inputs to the neural networks for training. The input to each neural network includes at least one of the reference images for training and the input image for training.
1 . An electronic apparatus configured for image recognition and comprising:
a storage apparatus, storing a training sample set, the training sample set comprising a plurality of image sets respectively belonging to a plurality of users, each of the image sets comprising a plurality of training images, each of the training images being labeled by one of a plurality of classification labels, each of the training images included in each of the image sets having a face area of one of the users corresponding to each of the image sets; and
a processor, coupled to the storage apparatus and configured to train a recognition model to recognize which one of the classification labels to which an image to be tested belongs through the trained recognition model,
wherein the recognition model comprises a plurality of neural networks,
the processor is configured to train the recognition model, and steps of training the recognition model comprises:
obtaining the training images respectively corresponding to the classification labels from a first image set as a plurality of reference images for training, wherein the first image set belonging to a first user is one of the image sets, each of the training images included in the first image set has a face area of the first user;
obtaining one of the training images from a second image set belonging to a second user as an input image for training, wherein the second image set is another of the image sets different from the first image set, the first user is different from the second user, each of the training images included in the second image set has a face area of the second user; and
taking the reference images for training belonging to the first user and the input image for training belonging to the second user as inputs to the neural networks, wherein the input to each of the neural networks comprises at least one of the reference images for training and the input image for training,
wherein in each of a plurality of iteration training processes, the processor is configured to use both of at least one of the reference images for training from the first image set and the input image for training from the second image set as inputs to each of the neural networks,
the recognition model adopts one of a first architecture or a second architecture,
the first architecture further comprises a plurality of first fusion layers, a plurality of fully connected layers and a voting module, each of the neural networks corresponds to one of the first fusion layers and one of the fully connected layers, an output of each of the neural networks is used as an input of a corresponding first fusion layer, an output of the corresponding first fusion layer is used as an input of a corresponding fully connected layer, outputs of the fully connected layers are used as inputs of the voting module, and a first final predicted result is obtained through the voting module from the outputs of the fully connected layers,
the second architecture further comprises a second fusion layer and a timing neural network, outputs of the neural networks are used as inputs of the second fusion layer, and an output of the second fusion layer is used as input of the timing neural network, and a second final prediction result is obtained through the timing neural network based on the output of the second fusion layer.
2 . The electronic apparatus according to claim 1 , wherein the processor is configured to:
perform a cropping process on each of the training images to crop a mask portion and keep a recognizable portion.
3 . The electronic apparatus according to claim 1 , wherein in response to the recognition model adopting the first architecture, the processor is configured to:
extract a reference feature and an input feature from one of the reference images for training and the input image for training in the input to each of the neural networks, respectively;
combine the reference feature and the input feature to obtain a combined feature through one of the first fusion layers corresponding to each of the neural networks;
obtain a predicted result from the combined feature through one of the fully connected layers corresponding to each of the neural networks; and
obtain the final predicted result from all of the predicted results of the neural networks by applying a voting method through the voting module.
4 . The electronic apparatus according to claim 3 , wherein the recognition model comprises a first quantity of the neural networks, and in each of the iteration training processes, the processor is configured to:
obtain the first quantity of the training images from the first image set as the reference images for training and individually input the reference images for training together with the same input image for training to the neural networks.
5 . The electronic apparatus according to claim 1 , wherein in response to the recognition model adopting the second architecture, the processor is configured to:
extract a reference feature and an input feature from one of the reference images for training and the input image for training in the input to each of the neural networks, respectively;
combine all of the reference features and all of the input features obtained from the neural networks through the second fusion layer to obtain a combined feature; and
obtain a final predicted result from the combined feature through the timing neural network.
6 . The electronic apparatus according to claim 5 , wherein the recognition model comprises a second quantity of the neural networks, and in each of the iteration training processes, the processor is configured to:
obtain the second quantity of the training images as the reference images for training from a plurality of consecutive training images which are all labeled by one of the classification labels and are not yet obtained in the previous iteration training processes in the first image set and individually input the reference images for training together with the same input image for training to the neural networks.
7 . The electronic apparatus according to claim 1 , wherein
in a process of recognizing the image to be tested through the trained recognition model, the inputs to the recognition model comprise the image to be tested or comprise the image to be tested and at least one reference image for testing, wherein the at least one reference image for testing and the image to be tested are labeled by the same one of the classification labels belonging to the same one of the users.
8 . An image recognition method carried out by a processor and comprising:
training a recognition model to recognize which one of a plurality of classification labels to which an image to be tested belongs through the trained recognition model, wherein the recognition model comprises a plurality of neural networks, and
steps of training the recognition model comprise:
providing a training sample set, the training sample set comprising a plurality of image sets respectively belonging to a plurality of users, each of the image sets comprising a plurality of training images, each of the training images being labeled by one of a plurality of classification labels, each of the training images included in each of the image sets having a face area of one of the users corresponding to each of the image sets;
obtaining the training images respectively corresponding to the classification labels from a first image set as a plurality of reference images for training, wherein the first image set belonging to a first user is one of the image sets, each of the training images included in the first image set has a face area of the first user;
obtaining one of the training images from a second image set belonging to a second user as an input image for training, wherein the second image set is another of the image sets different from the first image set, the first user is different from the second user, each of the training images included in the second image set has a face area of the second user; and
taking the reference images for training belonging to the first user and the input image for training belonging to the second user as inputs to the neural networks, so as to perform the training, wherein the input to each of the neural networks comprises at least one of the reference images for training and the input image for training,
wherein in each of a plurality of iteration training processes, both of at least one of the reference images for training from the first image set and the input image for training from the second image set are used as inputs to each of the neural networks,
the recognition model adopts one of a first architecture or a second architecture,
the first architecture further comprises a plurality of first fusion layers, a plurality of fully connected layers and a voting module, each of the neural networks corresponds to one of the first fusion layers and one of the fully connected layers, an output of each of the neural networks is used as an input of a corresponding first fusion layer, an output of the corresponding first fusion layer is used as an input of a corresponding fully connected layer, outputs of the fully connected layers are used as inputs of the voting module, and a first final predicted result is obtained through the voting module from the outputs of the fully connected layers,
the second architecture further comprises a second fusion layer and a timing neural network, outputs of the neural networks are used as inputs of the second fusion layer, and an output of the second fusion layer is used as input of the timing neural network, and a second final prediction result is obtained through the timing neural network based on the output of the second fusion layer.
9 . The image recognition method according to claim 8 , wherein the step of training the recognition model further comprises:
performing a cropping process on each of the training images to crop a mask portion and keep a recognizable portion.
10 . The image recognition method according to claim 8 , wherein in response to the recognition model adopting the first architecture, the step of training by taking the reference images for training belonging to the first user and the input image for training belonging to the second user as the inputs to the neural networks comprises:
extracting a reference feature and an input feature from one of the reference images for training and the input image for training in the input to each of the neural networks, respectively;
combining the reference feature and the input feature to obtain a combined feature through one of the first fusion layers corresponding to each of the neural networks;
obtaining a predicted result from the combined feature through one of the fully connected layers corresponding to each of the neural networks; and
obtaining the final predicted result from all of the predicted results of the neural networks by applying a voting method through the voting module.
11 . The image recognition method according to claim 10 , wherein the recognition model comprises a first quantity of the neural networks, and each of the iteration training processes, the image recognition method further comprises:
obtaining the first quantity of the training images from the first image set as the reference images for training and individually inputting the reference images for training together with the same input image for training to the neural networks.
12 . The image recognition method according to claim 8 , wherein in response to the recognition model adopting the second architecture, the step of taking the reference images for training belonging to the first user and the input image for training belonging to the second user as the inputs to the neural networks comprises:
extracting a reference feature and an input feature from one of the reference images for training and the input image for training in the input to each of the neural networks, respectively;
combining all of the reference features and all of the input features obtained from the neural networks through the second fusion layer to obtain a combined feature; and
obtaining a final predicted result from the combined feature through the timing neural network.
13 . The image recognition method according to claim 12 , wherein the recognition model comprises a second quantity of the neural networks, and in each of the iteration training processes, the image recognition method further comprises:
obtaining the second quantity of the training images as the reference images for training from a plurality of consecutive training images which are all labeled by one of the classification labels and are not yet obtained in the previous iteration training processes in the first image set, and individually inputting the reference images for training together with the same input image for training to the neural networks.
14 . The image recognition method according to claim 8 , wherein in a process of recognizing the image to be tested through the trained recognition model, the image recognition method comprises:
inputting the image to be tested to the recognition model; or
inputting the image to be tested and at least one reference image for testing to the recognition model, wherein the image to be tested and the at least one test reference image are labeled by the same one of the classification labels belonging to the same one of the users.