Online adaptation of neural networks
Described herein are neural network-based systems, methods and instrumentalities associated with imagery data processing. The neural networks may be pre-trained to learn parameters or models for processing the imagery data and upon deployment the neural networks may automatically perform further optimization of the learned parameters or models based on a small set of online data samples. The online optimization may be facilitated via offline meta-learning so that the optimization may be accomplished quickly in a few optimization steps.
1. A system comprising at least one processor, wherein the at least one processor is configured to:
implement one or more artificial neural networks, wherein the one or more artificial neural networks are configured with a plurality of predetermined parameters for processing images of an anatomical structure;
bring the one or more artificial neural networks online to process the images; and
while the one or more artificial neural networks are online:
perform online adjustments to the plurality of predetermined parameters of the one or more artificial neural networks based on a first set of online images of the anatomical structure, wherein the online adjustments are performed by at least determining a loss associated with processing the first set of online images using the plurality of predetermined parameters and adjusting the plurality of predetermined parameters based on a gradient descent associated with the loss, and wherein the plurality of predetermined parameters are acquired via offline meta-learning that facilitates the online adjustments; and
process a second set of online images of the anatomical structure using the adjusted plurality of predetermined parameters of the one or more artificial neural networks.
2. The system of claim 1 , wherein the online adjustments to the plurality of predetermined parameters are performed by backpropagating the gradient descent through the one or more artificial neural networks.
3. The system of claim 1 , wherein the meta-learning is performed using multiple training videos of the anatomical structure, the meta-learning comprising:
obtaining baseline parameters for processing the images of the anatomical structure;
optimizing a respective copy of the baseline parameters based on a respective first loss associated with processing each of the multiple training videos using the respective copy of the baseline parameters;
determining a respective second loss associated with processing each of the multiple training videos using the respective optimized copy of the baseline parameters; and
updating the baseline parameters based on the respective second losses associated with the multiple training videos.
4. The system of claim 3 , wherein the meta-learning comprises:
for each of the multiple training videos:
deriving the respective copy of the baseline parameters for the training video;
selecting a first set of training images from the training video;
determining a first loss associated with processing the first set of training images using the respective copy of the baseline parameters;
optimizing the respective copy of the baseline parameters based on a gradient descent associated with the first loss;
selecting a second set of training images from the training video; and
determining a second loss associated with processing the second set of training images using the respective optimized copy of the baseline parameters;
determining an average of the respective second losses associated with processing the respective second sets of training images of the multiple training videos; and
updating the baseline parameters based on a gradient descent associated with the determined average of the respective second losses.
5. The system of claim 4 , wherein the respective first and second losses are determined based on a loss function and wherein the baseline parameters are updated based on a first order approximation of the loss function.
6. The system of claim 3 , wherein the meta-learning comprises:
for each of the multiple training videos:
deriving the respective copy of the baseline parameters for the training video;
selecting a first set of training images from the training video;
determining a first loss associated with processing the first set of training images using the respective copy of the baseline parameters;
optimizing the respective copy of the baseline parameters based on a gradient descent associated with the first loss;
selecting a second set of training images from the training video; and
determining a second loss associated with processing the second set of training images using the respective optimized copy of the baseline parameters and a gradient descent associated with the second loss;
determining an average of the respective gradient descents associated with processing the respective second sets of training images of the multiple training videos; and
updating the baseline parameters based on the determined average of the respective gradient descents.
7. The system of claim 3 , wherein the baseline parameters are derived based on a first training set characterized by a first distribution and wherein the multiple training videos are derived from a second training set characterized by a second distribution that mismatches the first distribution.
8. The system of claim 1 , wherein the at least one processor being configured to process the images of the anatomical structure comprises the at least one processor being configured to track a motion of the anatomical structure based on the images or perform an image registration based on the images.
9. The system of claim 8 , wherein the anatomical structure comprises a myocardium and the images are derived from a cardiac magnetic resonance (CMR) video.
10. The system of claim 1 , wherein the one or more artificial neural networks comprise an encoder network and a decoder network.
11. A method implemented via at least one processor, the method comprising:
bringing one or more artificial neural networks online to process images of an anatomical structure, wherein the one or more artificial neural networks are configured with a plurality of predetermined parameters for processing the images of the anatomical structure; and
while the one or more artificial neural networks are online:
performing online adjustments to the plurality of predetermined parameters of the one or more artificial neural networks based on a first set of online images of the anatomical structure, wherein the online adjustments are performed by at least determining a loss associated with processing the first set of online images using the plurality of predetermined parameters and adjusting the plurality of predetermined parameters based on a gradient descent associated with the loss, and wherein the plurality of predetermined parameters are acquired via offline meta-learning that facilitates the online adjustments; and
processing a second set of online images of the anatomical structure using the adjusted plurality of predetermined parameters of the one or more artificial neural networks.
12. The method of claim 11 , wherein performing the online adjustments to the plurality of predetermined parameters of the one or more artificial neural networks comprises backpropagating the gradient descent through the one or more artificial neural networks.
13. The method of claim 11 , wherein the meta-learning is performed using multiple training videos of the anatomical structure, the meta-learning comprising:
obtaining baseline parameters for processing the images of the anatomical structure;
optimizing a respective copy of the baseline parameters based on a respective first loss associated with processing each of the multiple training videos using the respective copy of the baseline parameters;
determining a respective second loss associated with processing each of the multiple training videos using the respective optimized copy of the baseline parameters; and
updating the baseline parameters based on the respective second losses associated with the multiple training videos.
14. The method of claim 13 , wherein the meta-learning comprises:
for each of the multiple training videos:
deriving the respective copy of the baseline parameters for the training video;
selecting a first set of training images from the training video;
determining a first loss associated with processing the first set of training images using the respective copy of the baseline parameters;
optimizing the respective copy of the baseline parameters based on a gradient descent associated with the first loss;
selecting a second set of training images from the training video; and
determining a second loss associated with processing the second set of training images using the respective optimized copy of the baseline parameters;
determining an average of the respective second losses associated with processing the respective second sets of training images of the multiple training videos; and
updating the baseline parameters based on a gradient descent associated with the determined average of the respective second losses.
15. The method of claim 14 , wherein the respective first and second losses are determined based on a loss function and wherein the baseline parameters are updated based on a first order approximation of the loss function.
16. The method of claim 13 , wherein the meta-learning comprises:
for each of the multiple training videos:
deriving the respective copy of the baseline parameters for the training video;
selecting a first set of training images from the training video;
determining a first loss associated with processing the first set of training images using the respective copy of the baseline parameters;
optimizing the respective copy of the baseline parameters based on a gradient descent associated with the first loss;
selecting a second set of training images from the training video; and
determining a second loss associated with processing the second set of training images using the respective optimized copy of the baseline parameters and a gradient descent associated with the second loss;
determining an average of the respective gradient descents associated with processing the respective second sets of training images of the multiple training videos; and
updating the baseline parameters based on the determined average of the respective gradient descents.
17. The method of claim 13 , wherein the baseline parameters are derived based on a first training set characterized by a first distribution and wherein the multiple training videos are derived from a second training set characterized by a second distribution that mismatches the first distribution.
18. The method of claim 11 , wherein processing the images of the anatomical structure comprises tracking a motion of the anatomical structure based on the images or performing an image registration based on the images.
19. The method of claim 18 , wherein the anatomical structure comprises a myocardium and the images are derived from a cardiac magnetic resonance (CMR) video.
20. The method of claim 11 , wherein the one or more artificial neural networks comprise an encoder network and a decoder network.