Systems and methods for motion estimation
Described herein are neural network-based systems, methods and instrumentalities associated with estimating the motion of an anatomical structure. The motion estimation may be performed using a feature pyramid and/or a motion pyramid that correspond to multiple image scales. The motion estimation may be performed using neural networks and parameters that are learned via a training process involving a student network and a teacher network pre-pretrained with abilities to apply progressive motion compensation.
1. An apparatus for motion tracking, comprising one or more processors, wherein the one or more processors are configured to:
obtain a source image of an anatomical structure and a reference image of the anatomical structure;
at each of multiple image scales, generate a first representation of features based on the source image, a second representation of features based on the reference image, and a motion field based on the first representation of features and the second representation of features, wherein the motion field indicates a change of the anatomical structure from the source image to the reference image, wherein the respective first representations of features associated with the multiple image scales and the respective second representations of features associated with the multiple image scales form a feature pyramid, and wherein the respective motion fields associated with the multiple image scales form a motion pyramid;
determine a preliminary motion field based on the feature pyramid; and
refine the preliminary motion field based on the motion pyramid to obtain a refined motion field,
wherein the motion field associated with each of the multiple image scales is determined using a student neural network, wherein parameters of the student neural network are learned via a training process guided by a teacher neural network, and wherein, during the training process:
the teacher neural network is configured to predict a motion between two training images of the anatomical structure via a progressive motion compensation process that includes determining multiple intermediary motion fields and predicting the motion between the two training images based on the multiple intermediary motion fields;
the student neural network is configured to directly predict the motion between the two training images; and
the parameters of the student neural network are adjusted based on a difference between the motion predicted by the student neural network and the motion predicted by the teacher neural network.
2. The apparatus of claim 1 , wherein the multiple image scales correspond to respective image resolutions.
3. The apparatus of claim 1 , wherein the motion fields associated with the multiple image scales are determined independently from each other.
4. The apparatus of claim 1 , wherein the one or more processors being configured to refine the preliminary motion field based on the motion pyramid comprises the one or more processors being configured to up-sample the respective motion fields associated with the multiple image scales and fuse the respective up-sampled motion fields with the preliminary motion field to obtain the refined motion field.
5. The apparatus of claim 4 , wherein the one or more processors being configured to refine the preliminary motion field based on the motion pyramid further comprises the one or more processors being configured to apply respective scaling factors to the up-sampled motion fields prior to fusing the up-sampled motion fields with the preliminary motion field.
6. The apparatus of claim 1 , wherein the two training images comprise a source training image and a reference training image and wherein the teacher neural network being configured to predict the motion between the two training images via the progressive motion compensation process comprises the teacher neural network being configured to:
predict a first intermediary motion field based on the source training image and the reference training image;
predict a second intermediary motion field based on the reference training image and a warped image obtained using the source image and the first intermediary motion field; and
predict the motion between the two training images based on the first intermediary motion field and the second intermediary motion field.
7. The apparatus of claim 6 , wherein the parameters of the student neural network are learned using the motion predicted by the teacher neural network as ground truth.
8. The apparatus of claim 1 , wherein the training process comprises multiple training iterations and wherein the parameters of the student neural network obtained via a first training iteration are copied to the teacher neural network and used to guide the student neural network during a second training iteration.
9. The apparatus of claim 1 , wherein the anatomical structure comprises a myocardium and wherein, during the training process, a motion estimation error is decomposed into a first error along a radial direction and a second error along a circumferential direction and the parameters of the student neural network are adjusted based on at least one of the first error or the second error.
10. A method for tracking a motion of an anatomical structure, the method comprising:
obtaining a source image of the anatomical structure and a reference image of the anatomical structure;
at each of multiple image scales, generating a first representation of features based on the source image, a second representation of features based on the reference image, and a motion field based on the first representation of features and the second representation of features, wherein the motion field indicates a change of the anatomical structure from the source image to the reference image, wherein the respective first representations of features associated with the multiple image scales and the respective second representations of features associated with the multiple image scales form a feature pyramid and the respective motion fields associated with the multiple image scales form a motion pyramid;
determining a preliminary motion field based on the feature pyramid; and
refining the preliminary motion field based on the motion pyramid to obtain a refined motion field,
wherein the motion field associated with each of the multiple image scales is determined using a student neural network, wherein parameters of the student neural network are learned via a training process guided by a teacher neural network, and wherein, during the training process:
the teacher neural network is configured to predict a motion between two training images of the anatomical structure via a progressive motion compensation process that includes determining multiple intermediary motion fields and predicting the motion between the two training images based on the multiple intermediary motion fields;
the student neural network is configured to directly predict the motion between the two training images; and
the parameters of the student neural network are adjusted based on a difference between the motion predicted by the student neural network and the motion predicted by the teacher neural network.
11. The method of claim 10 , wherein the motion fields associated with the multiple image scales are determined independently from each other.
12. The method of claim 10 , wherein refining the preliminary motion field based on the motion pyramid comprises up-sampling the respective motion fields associated with the multiple image scales and fusing the respective up-sampled motion fields with the preliminary motion field to obtain the refined motion field.
13. The method of claim 12 , wherein refining the preliminary motion field based on the motion pyramid further comprises applying respective scaling factors to the up-sampled motion fields prior to fusing the up-sampled motion fields with the preliminary motion field.
14. The method of claim 10 , wherein the two training images comprise a source training image and a reference training image and wherein the teacher neural network being configured to predict the motion between the two training images via the progressive motion compensation process comprises the teacher neural network being configured to:
predict a first intermediary motion field based on the source training image and the reference training image;
predict a second intermediary motion field based on the reference training image and a warped image obtained using the source image and the first intermediary motion field; and
predict the motion between the two training images based on the first intermediary motion field and the second intermediary motion field.
15. The method of claim 14 , wherein the training process comprises multiple training iterations and wherein the parameters of the student neural network obtained via a first training iteration are copied to the teacher neural network and used to guide the student neural network during a second training iteration.