Systems and methods for robotic endoscope with integrated tool-in-lesion-tomosynthesis
A robotic endoscopic system and method confirm whether a tool has reached a target region within a subject using tomosynthesis. While the tool is advanced into the target region, a fluoroscopic imager acquires a plurality of images, and a three-dimensional (3D) fluoroscopic image is reconstructed from the plurality of images. A first slice is identified from the 3D fluoroscopic image in which the target region is in focus and a second slice is identified in which the tool is in focus, and a depth coordinate is determined for each. A difference between the coordinates is compared with a threshold derived from a dimension of the target region to determine, in a quantitative manner, whether the tool is positioned within the target region.
1 . A method for navigating a robotic endoscopic apparatus, the method comprising:
(a) navigating the robotic endoscopic apparatus to a target region within a subject;
(b) acquiring a plurality of fluoroscopic images at different angles using a fluoroscopic imager when a tool is extended through the robotic endoscopic apparatus into the target region, and reconstructing a 3D fluoroscopic image based on the plurality of fluoroscopic images;
(c) identifying, from the 3D fluoroscopic image reconstructed in (b), a first slice with a first coordinate corresponding to a center of the target region in a depth direction, and identifying, from the 3D fluoroscopic image reconstructed in (b), a second slice with a second coordinate corresponding to the tool in the depth direction; and
(d) determining whether the tool is inside the target region based at least in part on a comparison of a difference between the first coordinate and the second coordinate to a threshold, wherein the target region comprises a lesion that is visible in the 3D fluoroscopic image, wherein the threshold is determined based at least in part on a dimension of the lesion and wherein the dimension of the lesion is calculated based at least in part on a 3D model of the lesion obtained from an image acquired prior to (a).
2 . The method of claim 1 , wherein the first slice is identified by i) displaying the 3D fluoroscopic image within a graphical user interface (GUI), ii) selecting the first slice from a stack of slices when the lesion is in focus.
3 . The method of claim 2 , wherein the second slice is identified when the tool is in focus.
4 . The method of claim 1 , wherein the first slice or second slice is automatically identified based on a sharpness metric or contrast metric of each slice in the depth direction.
5 . The method of claim 1 , further comprising displaying the 3D fluoroscopic image within a graphical user interface (GUI) and displaying an overlay of the lesion on each slice from a plurality of stacks in the depth direction.
6 . The method of claim 5 , wherein the overlay is generated based at least in part on a 3D model of the lesion intersecting each slice.
7 . The method of claim 5 , further comprising determining whether the tool is inside the target region by identifying whether the overlay of the lesion appears in the second slice.
8 . The method of claim 1 , further comprising displaying, on a graphical user interface (GUI), the 3D fluoroscopic image, a first graphical visual indicator representing the first coordinate and a second graphical visual indicator representing the second coordinate.
9 . The method of claim 1 , wherein the 3D fluoroscopic image is reconstructed based on a pose of the fluoroscopic imager.
10 . The method of claim 9 , wherein the pose of the fluoroscopic imager is estimated based on markers contained in the acquired plurality of fluoroscopic images.
11 . The method of claim 9 , wherein the pose of the fluoroscopic imager is obtained based on location sensor data.
12 . The method of claim 1 , wherein the threshold comprises a margin and wherein the margin is determined based on empirical data.
13 . A non-transitory computer-readable storage medium including instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
(a) navigating a robotic endoscopic apparatus to a target region within a subject;
(b) acquiring a plurality of fluoroscopic images at different angles using a fluoroscopic imager when a tool is extended through the robotic endoscopic apparatus into the target region, and reconstructing a 3D fluoroscopic image based on the plurality of fluoroscopic images;
(c) identifying, from the 3D fluoroscopic image reconstructed in (b), a first slice with a first coordinate corresponding to a center of the target region in a depth direction, and identifying, from the 3D fluoroscopic image reconstructed in (b), a second slice with a second coordinate corresponding to the tool in the depth direction; and
(d) determining whether the tool is inside the target region based at least in part on a comparison of a difference between the first coordinate and the second coordinate to a threshold, wherein the target region comprises a lesion that is visible in the 3D fluoroscopic image, wherein the threshold is determined based at least in part on a dimension of the lesion and wherein the dimension of the lesion is calculated based at least in part on a 3D model of the lesion obtained from an image acquired prior to (a).
14 . The non-transitory computer-readable storage medium of claim 13 , wherein the first slice is identified by i) displaying the 3D fluoroscopic image within a graphical user interface (GUI), ii) selecting the first slice from a stack of slices when the lesion is in focus.
15 . The non-transitory computer-readable storage medium of claim 14 , wherein the second slice is identified when the tool is in focus.