Learning apparatus and method
According to one embodiment, a learning apparatus includes a processor. The processor performs, on a neural network model, an adaptation processing that includes at least either insertion of an activation function, or correction of the activation function. The processor generates a trained model by training the neural network model on which the adaptation processing has been performed. The processor performs pruning on the trained model to generate a reconstructed model from which a parameter has been reduced.
1 . A learning apparatus comprising:
a storage storing (i) a neural network model for classifying an image and (ii) training data, the training data comprising image data of a plurality of images, and each of the plurality of images being associated with a ground truth; and
a hardware processor operably coupled to the storage, the hardware processor being configured to:
perform, on the neural network model, an adaptation processing that includes at least either insertion of an activation function, or correction of the activation function;
generate a trained model for classification by training the neural network model on which the adaptation processing has been performed using the training data stored in the storage as input; and
perform pruning on the trained model to generate a reconstructed model from which a parameter has been reduced,
wherein the reconstructed model is iteratively processed by the adaptation processing, the training, and the pruning, until a predetermined determination index is equal to or smaller than a threshold value, whereby pruning of the trained model is completed, and
wherein, in response to completion of pruning of the trained mode, the hardware processor stores the reconstructed model in the storage.
2 . The apparatus according to claim 1 , wherein the processor performs the insertion of the activation function by connecting the activation function next to a hidden layer to be pruned, from among a plurality of hidden layers that constitute the neural network model.
3 . The apparatus according to claim 1 , wherein the processor performs the correction of the activation function, by replacing a first activation function contained in the neural network model, with a second activation function which is different from the first activation function.
4 . The apparatus according to claim 3 , wherein the second activation function has a smaller gradient of a parameter relative to loss calculated during training of a model, as compared with the first activation function.
5 . The apparatus according to claim 1 , wherein the processor performs the correction of the activation function, by connecting a first activation function contained in the neural network model, to a point different from a current point of connection.
6 . The apparatus according to claim 1 , wherein the processor restores the activation function inserted or corrected by the adaptation processing, to a state before the adaptation processing.
7 . The apparatus according to claim 1 , wherein the processor is further configured to:
determine whether performance of the reconstructed model after the adaptation processing is equivalent or superior to performance of the model before the adaptation processing; and
maintain the adaptation processing for the reconstructed model, if the performance of the reconstructed model after the adaptation processing is equivalent or superior to the performance of the model before the adaptation processing.
8 . A learning method executed under control of a hardware processor operably coupled to a storage, the storage storing (i) a neural network model for classifying an image and (ii) training data, the training data comprising image data of a plurality of images, each of the plurality of images being associated with a ground truth, and the method comprising:
performing, on the neural network model, an adaptation processing that includes at least either insertion of an activation function, or correction of the activation function;
generating a trained model for classification by training the neural network model on which the adaptation processing has been performed using the training data stored in the storage as input; and
performing pruning on the trained model to generate a reconstructed model from which a parameter has been reduced,
wherein the reconstructed model is iteratively processed by the adaptation processing, the training, and the pruning, until a predetermined determination index is equal to or smaller than a threshold value, whereby pruning of the trained model is completed, and
wherein the method further comprises, in response to completion of pruning of the trained mode, storing the reconstructed model in the storage.
9 . The method according to claim 8 , wherein the performing the adaptation process performs the insertion of the activation function by connecting the activation function next to a hidden layer to be pruned, from among a plurality of hidden layers that constitute the neural network model.
10 . The method according to claim 8 , wherein the performing the adaptation process performs the correction of the activation function, by replacing a first activation function contained in the neural network model, with a second activation function which is different from the first activation function.
11 . The method according to claim 10 , wherein the second activation function has a smaller gradient of a parameter relative to loss calculated during training of a model, as compared with the first activation function.
12 . The method according to claim 8 , wherein the performing the adaptation process performs the correction of the activation function, by connecting a first activation function contained in the neural network model, to a point different from a current point of connection.
13 . The method according to claim 8 , wherein the performing the adaptation process restores the activation function inserted or corrected by the adaptation processing, to a state before the adaptation processing.
14 . The method according to claim 1 , further comprising:
determining whether performance of the reconstructed model after the adaptation processing is equivalent or superior to performance of the model before the adaptation processing; and
maintaining the adaptation processing for the reconstructed model, if the performance of the reconstructed model after the adaptation processing is equivalent or superior to the performance of the model before the adaptation processing.