Machine learning method for continual learning and electronic device
A machine learning method for continual learning is provided, and the method includes following steps. Capturing an input image. Performing feature extraction on the input image by a plurality of sub-models to obtain a plurality of feature maps, where the sub-models correspond to a plurality of tasks, and the sub-models are determined by a neural network model and a plurality of channel-wise masks. Converting the feature maps into a plurality of energy scores. Selecting a target sub-model corresponding to a target task of the tasks from the sub-models according to the energy scores. Outputting a prediction result corresponding to the target task by the target sub-model.
1. A machine learning method for continual learning, comprising:
capturing an input image;
performing feature extraction on the input image by a plurality of sub-models to obtain a plurality of feature maps, wherein the sub-models correspond to a plurality of tasks, and the sub-models are determined by a neural network model and a plurality of channel-wise masks;
converting the feature maps into a plurality of energy scores;
selecting a target sub-model corresponding to a target task of the tasks from the sub-models according to the energy scores; and
outputting a prediction result corresponding to the target task by the target sub-model,
wherein the machine learning method further comprises:
receiving training data associated with the tasks, wherein the training data comprise a training image and a class label corresponding to the training image;
performing the feature extraction on the training image according to the neural network model and the channel-wise masks to obtain a training feature map;
converting the training feature map into a feature energy score;
determining a loss function according to the class label, the channel-wise masks, and the feature energy score;
determining a backward propagation gradient according to the loss function; and
updating the neural network model and the channel-wise masks according to the backward propagation gradient, wherein the channel-wise masks correspond to the tasks, and a channel-wise mask of the channel-wise masks corresponding to at least one training task of the tasks is updated.
2. The machine learning method according to claim 1 , wherein the neural network model comprises at least one convolutional layer and at least one batch normalization layer.
3. The machine learning method according to claim 2 , wherein each of the channel-wise masks comprises a plurality of scaling factors of the at least one batch normalization layer, and the scaling factors are associated with a plurality of convolutional kernels of the at least one convolutional layer, wherein the step of performing the feature extraction on the input image by the sub-models to obtain the feature maps comprises:
putting the input image through the at least one convolutional layer to obtain an input feature map; and
performing normalization calculation on the input feature map and the scaling factors of each of the channel-wise masks through at least one batch normalization layer to obtain the feature maps.
4. The machine learning method according to claim 3 , wherein the normalization calculation is performed by a following formula:
{dot over (X)}=γ l t ×(( X*−μ l t )/σ l t )+β l t
wherein {dot over (X)} is the feature maps, X* is the input feature map, γ l t is the scaling factors, μ l t is an average value, σ l t is a standard deviation, and β l t is a shifting factor.
5. The machine learning method according to claim 1 , wherein the step of converting the feature maps into the energy scores is calculated by a following formula:
E
=
-
log
∑
c
=
1
C
′
exp
{
f
(
X
)
c
}
wherein E is the energy scores, f(X) c is a value of the feature maps in a class c, and C′ is a number of classes.
6. The machine learning method according to claim 1 , wherein the step of selecting the target sub-model corresponding to the target task of the tasks from the sub-models according to the energy scores comprises:
selecting a sub-model corresponding to a minimum value of the energy scores as the target sub-model.
7. The machine learning method according to claim 1 , wherein each of the channel-wise masks comprises a plurality of importance factors, and the step of performing the feature extraction on the input image by the sub-models to obtain the feature maps comprises:
putting the input image through the neural network model to obtain an output feature map;
performing an element-wise multiplication on the output feature map and the importance factors of each of the channel-wise masks to obtain the feature maps.
8. The machine learning method according to claim 7 , wherein the importance factors are calculated by a following formula:
IF
l
t
=
1
1
+
e
kx
l
t
wherein IF l t is the importance factors, x l t is the output feature map, and k is a parameter.
9. The machine learning method according to claim 1 , wherein the neural network model comprises at least one convolutional layer and at least one batch normalization layer, each of the channel-wise masks comprises a plurality of scaling factors of the at least one batch normalization layer, and the scaling factors are associated with a plurality of convolutional kernels of the at least one convolutional layer, wherein the step of performing the feature extraction on the training image according to the neural network model and the channel-wise masks to obtain the training feature map comprises:
putting the training image through the at least one convolutional layer to obtain an input feature map; and
performing normalization calculation on the input feature map and the scaling factors through the at least one batch normalization layer to obtain the training feature map.
10. The machine learning method according to claim 9 , wherein the loss function comprises a linear combination of:
a cross entropy loss;
a channel-wise mask polarization regularization term; and
an energy loss, wherein
the channel-wise mask polarization regularization term is associated with the scaling factors, and
the energy loss is associated with the feature energy score.
11. The machine learning method according to claim 10 , wherein the channel-wise mask polarization regularization term is associated with the scaling factors.
12. The machine learning method according to claim 11 , wherein the channel-wise mask polarization regularization term is calculated by a following formula:
R s ( r )=Σ l=1 L Σ c=1 C l k|r l,c t |−|r l,c t − r |, r =Σ l=1 L Σ c=1 C l r l,c t /Σ l=1 L C l
wherein R s (r) is the channel-wise mask polarization regularization term, r l,c t is the scaling factors, L a number of layers, C l is a number of channels, and k is a parameter.
13. The machine learning method according to claim 10 , wherein the energy loss is calculated by a following formula:
Loss E =max(0, E same −EL same )+max(0, EL diff −E diff )
wherein Loss E is the energy loss, E same is the energy score of samples of the same task, E diff is the energy score of samples of different tasks, and EL same and EL diff are parameters.
14. The machine learning method according to claim 9 , wherein the step of updating the neural network model and the channel-wise masks according to the backward propagation gradient comprises:
determining a plurality of activation parameters according to the channel-wise masks;
adjusting the backward propagation gradient as an adjusted backward propagation gradient according to the activation parameters; and
updating the neural network model and the channel-wise masks according to the adjusted backward propagation gradient.
15. The machine learning method according to claim 14 , wherein the adjusted backward propagation gradient is calculated by a following formula:
g
l
′
=
(
1
-
act
(
γ
l
≤
t
)
)
g
l
,
{
act
(
x
)
=
1
if
x
≠
0
act
(
x
)
=
0
if
x
=
0
wherein g′ l is the adjusted backward propagation gradient, g l is the backward propagation gradient, γ l ≤t is a cumulative maximum value of the scaling factors of task 1 to task t corresponding to convolutional parameters at a same location, and act(γ l ≤t ) corresponds to the activation parameters determined by the cumulative maximum value γ l ≤t .
16. The machine learning method according to claim 1 , wherein each of the channel-wise masks comprises a plurality of importance factors, the loss function comprises an importance factor regularization term, and the importance factor regularization term is associated with the importance factors.
17. The machine learning method according to claim 16 , wherein the step of updating the neural network model and the channel-wise masks according to the backward propagation gradient comprises:
determining a plurality of activation parameters according to the channel-wise masks;
adjusting the backward propagation gradient as an adjusted backward propagation gradient according to the activation parameters; and
updating the neural network model and the channel-wise masks according to the adjusted backward propagation gradient, wherein the adjusted backward propagation gradient is calculated by a following formula:
g′ l =(1− IF l ≤t ) g l
wherein g′ l is the adjusted backward propagation gradient, g l is the backward propagation gradient, and IF l ≤t is a cumulative maximum value of the importance factors of task 1 to task t corresponding to convolutional parameters at a same position.
18. An electronic device for continual learning, comprising:
a storage medium, storing a plurality of modules; and
a processor, coupled to the storage medium and executes the modules, wherein the modules comprise:
an image capturing module, capturing an input image;
an inference module, performing feature extraction on the input image by a plurality of sub-models to obtain a plurality of feature maps, wherein the sub-models correspond to a plurality of tasks, the sub-models are determined by a neural network model and a plurality of channel-wise masks, the feature maps are converted into a plurality of energy scores, a target sub-model corresponding to a target task of the tasks is selected from the sub-models according to the energy scores, and a prediction result corresponding to the target task is output by the target sub-model; and
a training module, receiving training data associated with the tasks, wherein the training data comprise a training image and a class label corresponding to the training image; performing the feature extraction on the training image according to the neural network model and the channel-wise masks to obtain a training feature map; converting the training feature map into a feature energy score; determining a loss function according to the class label, the channel-wise masks, and the feature energy score; determining a backward propagation gradient according to the loss function; and updating the neural network model and the channel-wise masks according to the backward propagation gradient, wherein the channel-wise masks correspond to the tasks, and a channel-wise mask of the channel-wise masks corresponding to at least one training task of the tasks is updated.