System and method for real-time fusion of acoustic image with reference image
A system ( 200 ) and method ( 900 ): access ( 910 ) a 3D reference image of a region of interest (ROI) ( 10 ) in a subject which was obtained using a first 3D imaging modality; segment ( 920 ) the ROI in the 3D reference image to define ( 930 ) a reference 3D coordinate system; acquire ( 940 ) 2D acoustic images of the ROI without absolute spatial tracking; generate ( 950 ) a standardized predicted pose for the 2D acoustic images with respect to a standardized 3D coordinate system ( 400 ) which was defined for a plurality of previously-obtained 3D images of corresponding ROIs in a plurality of other subjects which were obtained using the first 3D imaging modality; convert ( 960 ) the standardized predicted poses for the 2D acoustic images from the standardized 3D coordinate system to reference predicted poses in the reference 3D coordinate system; and use ( 970 ) the reference predicted poses for the 2D acoustic images to register the 2D acoustic images to the 3D reference image.
1. A method, comprising:
defining a standardized three dimensional coordinate system for a plurality of three dimensional reference images of corresponding regions of interest (ROIs) in a plurality of subjects, wherein the three dimensional reference images were obtained using a first imaging modality;
for each of the plurality of subjects:
obtaining a series of spatially tracked two dimensional acoustic images of the ROI in the subject;
constructing a three dimensional acoustic image of the ROI in the subject from the series of spatially tracked two dimensional acoustic images of the ROI, wherein the three dimensional acoustic image of the ROI in the subject is in an acoustic three dimensional coordinate system;
accessing a selected three dimensional reference image among the plurality three dimensional reference images, wherein the selected three dimensional reference image is of the ROI in the subject;
segmenting the three dimensional reference image of the ROI in the subject;
defining a reference three dimensional coordinate system for the reference image of the ROI in the subject, based on the segmentation; and
determining for each of the spatially tracked two dimensional acoustic images of the ROI in the subject its actual pose in the standardized three dimensional coordinate system using: a pose of the spatially tracked two dimensional acoustic image in the acoustic three dimensional coordinate system, a first coordinate transformation from the acoustic three dimensional coordinate system to the reference three dimensional coordinate system, and a second coordinate transformation from the reference three dimensional coordinate system to the standardized three dimensional coordinate system; and
performing an optimization process on a convolutional neural network (CNN) by providing the spatially tracked two dimensional acoustic images, and the corresponding actual poses of the spatially tracked two dimensional acoustic images, to the CNN and adjusting parameters of the CNN to minimize differences between predicted poses generated by the CNN for the spatially tracked two dimensional acoustic images and the actual poses of the spatially tracked 2D acoustic images.
2. The method of claim 1 , wherein the ROI in the subject includes an organ, and wherein segmenting the three dimensional reference image of the ROI in the subject comprises segmenting the organ.
3. The method of claim 1 , wherein the first imaging modality is magnetic resonance imaging or X-ray computed tomography imaging.
4. The method of claim 1 , wherein obtaining the series of spatially tracked two dimensional acoustic images of the ROI in the subject comprises receiving one or more imaging signals from an acoustic probe in conjunction with receiving an inertial measurement signal from an inertial measurement unit which spatially tracks movement of the acoustic probe while it provides the one or more imaging signals.