Dual domain tracking of target structures
Embodiments described herein provide for determining a probability distribution of a three-dimensional point in a template feature map matching a three-dimensional point in space. A dual-domain target structure tracking end-to-end system receives projection data in one dimension or two dimensions and a three-dimensional simulation image. The end-to-end system extracts a template feature map from the simulation image using segmentation. The end-to-end system extracts features from the projection data, transforms the features of the projection data into three-dimensional space, and sequences the three-dimensional space to generate a three-dimensional feature map. The end-to-end system compares the template feature map to the generated three-dimensional feature map, determining an instantaneous probability distribution of the template feature map occurring in the three-dimensional feature map.
1. A computer-implemented method of location prediction using an end-to-end target structure tracking system comprising:
executing, by a computer, a machine learning model to extract a set of features from imaging projection data associated with a target structure of a patient's anatomy;
executing, by the computer, a reconstruction algorithm to transform the extracted set of features into a set of features in three-dimensional space;
executing, by the computer, a recurrent neural network to obtain three-dimensional feature map data associated with the target structure, the recurrent neural network configured to sequence the imaging projection data using the set of features in three-dimensional space;
extracting, by the computer, a template feature map from a three-dimensional simulation image, the template feature map comprising the target structure;
comparing, by the computer, the template feature map to the three-dimensional feature map data; and
indicating, by the computer, a probability of a three-dimensional point in the template feature map matching a location of a three-dimensional point in the three-dimensional feature map data.
2. The computer-implemented method according to claim 1 , further comprising:
receiving, by the computer, segment information associated with the target structure; and
extracting, by the computer, the template feature map from the three-dimensional simulation image using the segment information and the three-dimensional simulation image.
3. The computer-implemented method according to claim 1 , further comprising:
executing, by the computer, a forward projection algorithm to transform the set of features in three-dimensional space into a set of features in two-dimensional space; and
feeding, by the computer, the set of features in two-dimensional space into the machine learning model.
4. The computer-implemented method according to claim 1 , wherein the reconstruction algorithm is a second machine learning model such that the end-to-end target structure tracking system is trained using a loss function considering a peak-to-sidelobe ratio of a probability distribution corresponding to the probability of a three-dimensional point in the template feature map matching a location of a three-dimensional point in the three-dimensional feature map data.
5. The computer-implemented method according to claim 1 , further comprising:
determining, by the computer, a deformation field associated with the target structure; and
indicating, by the computer, the probability of a three-dimensional point in the template feature map, using the deformation field, matching the location of a three-dimensional point in the three-dimensional feature map data.
6. The computer-implemented method according to claim 1 , further comprising determining, by the computer, a confidence value based on peaks and sidelobes associated with a probability distribution corresponding to the probability of a three-dimensional point in the template feature map matching a location of a three-dimensional point in the three-dimensional feature map data.
7. The computer-implemented method according to claim 1 , further comprising determining, by the computer, a classification for a point in the three-dimensional feature map data based on the probability of a three-dimensional point in the template feature map matching the location of a three-dimensional point in the three-dimensional feature map data satisfying a threshold.
8. The computer-implemented method according to claim 1 , wherein the imaging projection data is based on at least one of a stereo projection pair or a projection set generated using a multi-view imaging system.
9. The computer-implemented method according to claim 1 , further comprising:
extracting, by the computer, an additional template feature map from the three-dimensional simulation image associated with an additional target structure;
comparing, by the computer, the additional template feature map to the three-dimensional feature map data; and
generating, by the computer, a multi-channel probability distribution indicating the probability of a three-dimensional point in the template feature map matching the location of a three-dimensional point in the three-dimensional feature map data and the probability of a three-dimensional point in the additional template feature map matching the location of a three-dimensional point in the three-dimensional feature map data.
10. The computer-implemented method according to claim 1 , wherein comparing the template feature map to the three-dimensional feature map data comprises convolving each point of the template feature map with each point of the three-dimensional feature map data.
11. A system comprising:
a server comprising a processor and a non-transitory computer-readable medium containing instructions that when executed by the processor causes the processor to perform operations comprising:
execute a machine learning model to extract a set of features from imaging projection data associated with a target structure of a patient's anatomy;
execute a reconstruction algorithm to transform the extracted set of features into a set of features in three-dimensional space;
execute a recurrent neural network to obtain three-dimensional feature map data associated with the target structure, the recurrent neural network configured to sequence the imaging projection data using the set of features in three-dimensional space;
extract a template feature map from a three-dimensional simulation image, the template feature map comprising the target structure;
compare the template feature map to the three-dimensional feature map data; and
indicate a probability of a three-dimensional point in the template feature map matching a location of a three-dimensional point in the three-dimensional feature map data.
12. The system according to claim 11 , wherein the processor is further configured to perform operations comprising:
receive segment information associated with the target structure; and
extract the template feature map from the three-dimensional simulation image using the segment information and the three-dimensional simulation image.
13. The system according to claim 11 , wherein the processor is further configured to perform operations comprising:
execute a forward projection algorithm to transform the set of features in three-dimensional space into a set of features in two-dimensional space; and
feed the set of features in two-dimensional space into the machine learning model.
14. The system according to claim 11 , wherein the reconstruction algorithm is a second machine learning model.
15. The system according to claim 11 , wherein the processor is further configured to perform operations comprising:
determine a deformation field associated with the target structure; and
indicate the probability of a three-dimensional point in the template feature map, using the deformation field, matching the location of a three-dimensional point in the three-dimensional feature map data.
16. The system according to claim 11 , wherein the processor is further configured to perform operations comprising:
determine a confidence value based on peaks and sidelobes associated with a probability distribution corresponding to the probability of a three-dimensional point in the template feature map matching a location of a three-dimensional point in the three-dimensional feature map data.
17. The system according to claim 11 , wherein the processor is further configured to perform operations comprising:
determine a classification for a point in the three-dimensional feature map data based on the probability of a three-dimensional point in the template feature map matching the location of a three-dimensional point in the three-dimensional feature map data satisfying a threshold.
18. The system according to claim 11 , wherein the imaging projection data is based on at least one of a stereo projection pair or a projection set generated using a multi-view imaging system.
19. The system according to claim 11 , wherein the processor is further configured to perform operations comprising:
extract an additional template feature map from the three-dimensional simulation image associated with an additional target structure;
compare the additional template feature map to the three-dimensional feature map data; and
generate a multi-channel probability distribution indicating the probability of a three-dimensional point in the template feature map matching the location of a three-dimensional point in the three-dimensional feature map data and the probability of a three-dimensional point in the additional template feature map matching the location of a three-dimensional point in the three-dimensional feature map data.
20. The system according to claim 11 , wherein comparing the template feature map to the three-dimensional feature map data comprises convolving each point of the template feature map with each point of the three-dimensional feature map data.