Semi-supervised tracking in medical images with cycle tracking
Systems and methods for tracking a location of an object of interest through a sequence of medical images are provided. First and second input medical images of a patient are received. The first input medical image comprises an annotation of a location of an object of interest. Features are extracted from the first and the second input medical images. A location of the object of interest in the second input medical image is determined using a machine learning based location predictor network based on the annotation of the location of the object of interest in the first input medical image and the extracted features from the first and the second input medical images. The location of the object of interest in the second input medical image is output. The machine learning based location predictor network is trained based on a comparison between 1) locations of a particular object in a sequence of training images determined during a forward tracking of the particular object through the sequence of training images and 2) locations of the particular object determined during a backward tracking of the particular object through the sequence of training images.
1. A computer-implemented method comprising:
receiving first and second input medical images of a patient, the first input medical image comprising an annotation of a location of an object of interest;
extracting features from the first and the second input medical images;
determining a location of the object of interest in the second input medical image using a machine learning based location predictor network based on the annotation of the location of the object of interest in the first input medical image and the extracted features from the first and the second input medical images, wherein determining a location of the object of interest in the second input medical image using a machine learning based location predictor network based on the annotation of the location of the object of interest in the first input medical image and the extracted features from the first and the second input medical images comprises:
correlating the features extracted from the first input medical image with the features extracted from the second input medical image, wherein correlating the features extracted from the first input medical image with the features extracted from the second input medical image comprises:
matching features extracted from the first input medical image for regions corresponding to the location of the object of interest with features extracted from the second input medical image, and
determining the location of the object of interest in the second input medical image based on the correlated features; and
outputting the location of the object of interest in the second input medical image,
wherein the machine learning based location predictor network is trained based on a comparison between 1) locations of a particular object in a sequence of training images determined during a forward tracking of the particular object through the sequence of training images and 2) locations of the particular object determined during a backward tracking of the particular object through the sequence of training images.
2. The computer-implemented method of claim 1 , wherein the machine learning based location predictor network determines the locations of the particular object from a first image to a last image of the sequence of training images during the forward tracking and the machine learning based location predictor network determines the locations of the particular object from the last image to the first image in the sequence of training images during the backward tracking.
3. The computer-implemented method of claim 1 , wherein the machine learning based location predictor network is further trained based on a comparison between locations of the particular object in annotated training images of the sequence of training images and annotated ground truth locations of the particular object in the annotated training images.
4. The computer-implemented method of claim 1 , wherein the machine learning based location predictor network is further trained based on a comparison between a location of the particular object in a first training image of the sequence of training images and an annotated ground truth location of the particular object in the first training image of the sequence of training images.
5. The computer-implemented method of claim 1 , wherein a first image and a last image in the sequence of training images are annotated with a ground truth location of the particular object.
6. The computer-implemented method of claim 1 , wherein the first and the second input medical images are of a sequence of medical images, the method further comprising:
repeating the receiving, the extracting, the determining, and the outputting using the second input medical image as the first input medical image and a next image of the sequence of medical images as the second input medical image.
7. The computer-implemented method of claim 1 , wherein the location of the object of interest in the first input medical image is manually annotated by a user or automatically annotated using a machine learning based network.
8. An apparatus comprising:
means for receiving first and second input medical images of a patient, the first input medical image comprising an annotation of a location of an object of interest;
means for extracting features from the first and the second input medical images;
means for determining a location of the object of interest in the second input medical image using a machine learning based location predictor network based on the annotation of the location of the object of interest in the first input medical image and the extracted features from the first and the second input medical images, wherein the means for determining a location of the object of interest in the second input medical image using a machine learning based location predictor network based on the annotation of the location of the object of interest in the first input medical image and the extracted features from the first and the second input medical images comprises:
means for correlating the features extracted from the first input medical image with the features extracted from the second input medical image, wherein the means for correlating the features extracted from the first input medical image with the features extracted from the second input medical image comprises:
means for matching features extracted from the first input medical image for regions corresponding to the location of the object of interest with features extracted from the second input medical image, and
means for determining the location of the object of interest in the second input medical image based on the correlated features; and
means for outputting the location of the object of interest in the second input medical image,
wherein the machine learning based location predictor network is trained based on a comparison between 1) locations of a particular object in a sequence of training images determined during a forward tracking of the particular object through the sequence of training images and 2) locations of the particular object determined during a backward tracking of the particular object through the sequence of training images.
9. The apparatus of claim 8 , wherein the machine learning based location predictor network determines the locations of the particular object from a first image to a last image of the sequence of training images during the forward tracking and the machine learning based location predictor network determines the locations of the particular object from the last image to the first image in the sequence of training images during the backward tracking.
10. The apparatus of claim 8 , wherein the machine learning based location predictor network is further trained based on a comparison between locations of the particular object in annotated training images of the sequence of training images and annotated ground truth locations of the particular object in the annotated training images.
11. The apparatus of claim 8 , wherein the machine learning based location predictor network is further trained based on a comparison between a location of the particular object in a first training image of the sequence of training images and an annotated ground truth location of the particular object in the first training image of the sequence of training images.
12. The apparatus of claim 8 , wherein a first image and a last image in the sequence of training images are annotated with a ground truth location of the particular object.
13. A non-transitory computer readable medium storing computer program instructions, the computer program instructions when executed by a processor cause the processor to perform operations comprising:
receiving first and second input medical images of a patient, the first input medical image comprising an annotation of a location of an object of interest;
extracting features from the first and the second input medical images;
determining a location of the object of interest in the second input medical image using a machine learning based location predictor network based on the annotation of the location of the object of interest in the first input medical image and the extracted features from the first and the second input medical images, wherein determining a location of the object of interest in the second input medical image using a machine learning based location predictor network based on the annotation of the location of the object of interest in the first input medical image and the extracted features from the first and the second input medical images comprises:
correlating the features extracted from the first input medical image with the features extracted from the second input medical image, wherein correlating the features extracted from the first input medical image with the features extracted from the second input medical image comprises:
matching features extracted from the first input medical image for regions corresponding to the location of the object of interest with features extracted from the second input medical image, and
determining the location of the object of interest in the second input medical image based on the correlated features; and
outputting the location of the object of interest in the second input medical image,
wherein the machine learning based location predictor network is trained based on a comparison between 1) locations of a particular object in a sequence of training images determined during a forward tracking of the particular object through the sequence of training images and 2) locations of the particular object determined during a backward tracking of the particular object through the sequence of training images.
14. The non-transitory computer readable medium of claim 13 , wherein the machine learning based location predictor network determines the locations of the particular object from a first image to a last image of the sequence of training images during the forward tracking and the machine learning based location predictor network determines the locations of the particular object from the last image to the first image in the sequence of training images during the backward tracking.
15. The non-transitory computer readable medium of claim 13 , wherein the first and the second input medical images are of a sequence of medical images, the operations further comprising:
repeating the receiving, the extracting, the determining, and the outputting using the second input medical image as the first input medical image and a next image of the sequence of medical images as the second input medical image.
16. The non-transitory computer readable medium of claim 13 , wherein the location of the object of interest in the first input medical image is manually annotated by a user or automatically annotated using a machine learning based network.