Systems and methods of visualizing a medical device relative to a target

1 . A method for causing display of a 3D rendering of a tool in a catheter relative to a lesion, comprising:

receiving first fluoroscopic images acquired during a fluoroscopic sweep of a fluoroscopic imaging device centered on the catheter;

estimating a pose of the fluoroscopic imaging device for each of the first fluoroscopic images to obtain estimated poses;

generating a first three-dimensional (3D) reconstruction of the first fluoroscopic images acquired during a fluoroscopic sweep of the fluoroscopic imaging device using the estimated poses;

applying a neural network to the first 3D reconstruction generated from the fluoroscopic sweep using the estimated poses to segment the tool within the first 3D reconstruction and obtain a segmented tool representation;

determining, from the segmented tool representation within the first 3D reconstruction, a first point corresponding to a distal end portion of the tool and a second point spaced from the first point to define an orientation of the tool;

receiving a location of the lesion within the first 3D reconstruction;

generating a 3D rendering of the segmented tool relative to the lesion within the first 3D reconstruction based on the first and second points and the location of the lesion; and

causing display of the 3D rendering.

2 . The method of claim 1 , further comprising:

causing display of the first 3D reconstruction; and

causing display of a user interface enabling a user to mark the locations of two points on the tool in the first 3D reconstruction.

3 . The method of claim 1 , further comprising determining the locations of two points on the tool based on the segmented tool.

4 . The method of claim 1 , further comprising:

causing display of the first 3D reconstruction; and

causing display of a user interface enabling a user to mark the location of the lesion in the first 3D reconstruction.

5 . The method of claim 1 , further comprising:

causing display of the first 3D reconstruction;

causing display of a user interface enabling a user to mark the location of the lesion in the first 3D reconstruction;

receiving second fluoroscopic images;

generating a second 3D reconstruction of the second fluoroscopic images;

registering the first 3D reconstruction with the second 3D reconstruction; and

determining the location of the lesion in the second 3D reconstruction based on the registering.

6 . The method of claim 1 , further comprising:

causing display of the first 3D reconstruction;

causing display of a user interface enabling a user to mark a direction of a catheter tip in the first 3D reconstruction;

generating a 3D rendering of the catheter relative to the tool and the lesion based on the direction of the catheter tip; and

causing display of the 3D rendering of the catheter relative to the tool and the lesion.

7 . The method of claim 1 , further comprising:

segmenting a catheter in the first 3D reconstruction to obtain a segmented catheter;

generating a 3D rendering of the catheter relative to the tool and the lesion based on the segmented catheter; and

causing display of the 3D rendering of the catheter relative to the tool and the lesion.

8 . The method of claim 1 , further comprising causing display of a user interface enabling a user to adjust a view of the 3D rendering to a different angle.

9 . The method of claim 1 , further comprising:

determining an orientation of an anatomical structure in the first 3D reconstruction;

generating a 3D rendering of the anatomical structure at the orientation; and

causing display of the 3D rendering of the anatomical structure at the orientation.

10 . The method of claim 1 , wherein the first and second points are determined based on the segmented tool representation and are used to determine an orientation of the tool in the first 3D reconstruction.

11 . A system for displaying a 3D rendering of a tool in a catheter relative to a lesion, comprising:

a display;

a processor in communication with the display; and

a memory having stored thereon a neural network and instructions, which when executed by the processor, cause the processor to:

receive first fluoroscopic images acquired during a fluoroscopic sweep of a fluoroscopic imaging device centered on the catheter;

estimate a pose of the fluoroscopic imaging device for each of the first fluoroscopic images to obtain estimated poses;

generate a first three-dimensional (3D) reconstruction of the first fluoroscopic images acquired during the fluoroscopic sweep using the estimated poses;

apply a neural network to the first 3D reconstruction to segment the tool within the first 3D reconstruction and obtain a segmented tool representation and to determine a location and orientation of the segmented tool representation in the first 3D reconstruction by determining, from the segmented tool representation within the first 3D reconstruction, a first point corresponding to a distal end portion of the tool and a second point spaced from the first point to define an orientation of the tool;

receive a location of the lesion in the first 3D reconstruction;

generate a 3D rendering of the segmented tool relative to the lesion within the first 3D reconstruction based on the location and orientation of the segmented tool in the first 3D reconstruction and the location of the lesion; and

display the 3D rendering on the display.

12 . The system of claim 11 , wherein the instructions further cause the processor to:

receive second fluoroscopic images;

generate a second 3D reconstruction of the second fluoroscopic images;

register the first 3D reconstruction, in which the lesion is marked, with the second 3D reconstruction; and

determine the location of the lesion in the second 3D reconstruction based on the registering.

13 . The system of claim 12 , wherein the instructions further cause the processor to:

segment a shape of the lesion in the first 3D reconstruction to obtain a segmented shape of the lesion; and

generate the 3D rendering based on segmented shape of the lesion.

14 . A method for causing display of a 3D rendering of a tool in a catheter relative to a lesion, comprising:

receiving first fluoroscopic images acquired during a fluoroscopic sweep of a fluoroscopic imaging device centered on the catheter;

estimating a pose of the fluoroscopic imaging device for each of the first fluoroscopic images to obtain estimated poses;

generating a first three-dimensional (3D) reconstruction of the first fluoroscopic images acquired during the fluoroscopic sweep utilizing the estimated poses;

receiving two second fluoroscopic images at angles separated by at least 15 degrees;

registering the first 3D reconstruction with the two second fluoroscopic images;

determining a location of the lesion in the two second fluoroscopic images based on the registering;

segmenting the tool within the first 3D reconstruction to obtain a segmented tool representation by applying a neural network to the first 3D reconstruction;

determining, from the segmented tool representation within the first 3D reconstruction, a first point corresponding to a distal end portion of the tool and a second point spaced from the first point to define an orientation of the tool;

receiving a location of the lesion within the first 3D reconstruction;

generating a 3D rendering of the segmented tool relative to the lesion based on the first and second points and the determined location of the lesion; and

causing display of the 3D rendering.

15 . The method of claim 14 , further comprising:

causing display of the two second fluoroscopic images; and

causing display of a user interface enabling a user to mark the locations of two points on the tool in the two second fluoroscopic images.

16 . The method of claim 14 , further comprising determining the locations of two points on the tool based on the segmented tool.

17 . The method of claim 14 , further comprising:

causing display of the two second fluoroscopic images; and

causing display of a user interface enabling a user to mark the location of the lesion in the two second fluoroscopic images.

18 . The method of claim 14 , further comprising:

causing display of the first 3D reconstruction;

causing display of a user interface enabling a user to mark the location of the lesion in the first 3D reconstruction;

registering the first 3D reconstruction with the two second fluoroscopic images; and

determining the location of the lesion in the two second fluoroscopic images based on the registering.

19 . The method of claim 14 , further comprising:

determining information regarding a position of the tool relative to the lesion; and

causing display of the information.

20 . The method of claim 19 , wherein the information includes a distance between a tip of the tool and a center of the lesion or a status of alignment between the tool and the lesion.