3D visualization enhancement for depth perception and collision avoidance
A series of images is obtained from an endoscope. Three-dimensional reconstruction is performed on the series of images to reconstruct anatomy shown in the series of images. A graphic, such as a grid, is rendered based on the three-dimensional reconstruction, over the series of images resulting in an enhanced endoscopic video feed to be shown on a display.
1 . A method, including:
capturing, during a surgical procedure and using a camera, a series of images that include a surgical tool and an object within an internal anatomy of a patient;
generating, using the series of images, a three-dimensional (3D) reconstruction of the surgical tool, the object, and the internal anatomy that are included within the series of images;
determining a shortest distance between the surgical tool and the object based on the 3D reconstruction; and
displaying, during the surgical procedure and on a display, a graphic overlaid on the series of images, the graphic indicating the shortest distance between the surgical tool and the object.
2 . The method of claim 1 , wherein the object is another surgical tool, wherein both surgical tools are configured to be used during the surgical procedure.
3 . The method of claim 1 further comprising:
identifying, based on the series of images, the object within the internal anatomy of the patient; and
responsive to the shortest distance being less than a threshold distance and to the surgical tool being inactive, providing an alert based on the identified object.
4 . The method of claim 1 further comprising:
determining whether the shortest distance is within a threshold distance; and
providing a visual alert or an audible alert based on the shortest distance being within the threshold distance.
5 . The method of claim 4 , wherein the visual alert or the audible alert is provided responsive to a determination that 1) the surgical tool has remained inactive for a period of time and 2) the shortest distance is less than the threshold distance.
6 . The method of claim 1 , wherein generating the 3D reconstruction comprises receiving the 3D reconstruction as output of a machine learning (ML) model responsive to the series of images as input into the ML model.
7 . The method of claim 1 , wherein the graphic further comprises a pattern projected over at least a portion of a surface of the internal anatomy of the patient captured in at least a portion of the series of images.
8 . The method of claim 7 , wherein the pattern provides a visualization of a shape and contours of the surface of the internal anatomy.
9 . The method of claim 7 , wherein the pattern is a first pattern, wherein the graphic further comprises a second pattern projected over at least a portion of the surgical tool captured in at least a portion of the series of images.
10 . A system comprising:
a camera;
a display;
a surgical tool; and
at least one processor configured to perform the following:
capturing, during a surgical procedure and using the camera, a series of images that include the surgical tool and an object within an internal anatomy of a patient,
generating, using the series of images, a three-dimensional (3D) reconstruction of the surgical tool, the object, and the internal anatomy that are included within the series of images,
determining a shortest distance between the surgical tool and the object based on the 3D reconstruction, and
displaying, during the surgical procedure and on the display, a graphic overlaid on the series of images, the graphic indicating the shortest distance between the surgical tool and the object.
11 . The system of claim 10 , wherein the object is another surgical tool, wherein both surgical tools are to be used during the surgical procedure.
12 . The system of claim 10 , wherein the object is a portion of the internal anatomy.
13 . The system of claim 10 , wherein the at least one processor is further configured to perform the following:
determining whether the shortest distance is within a threshold distance; and
providing a visual alert or an audible alert based on the shortest distance being within the threshold distance.
14 . The system of claim 13 , wherein the visual alert or the audible alert is provided responsive to a determination that 1) the surgical tool has remained inactive for a period of time and 2) the shortest distance is less than the threshold distance.
15 . The system of claim 10 , wherein generating the 3D reconstruction comprises receiving the 3D reconstruction as output of a machine learning (ML) model responsive to the series of images as input into the ML model.
16 . The system of claim 10 , wherein the graphic further comprises a pattern projected over at least a portion of a surface of the internal anatomy of the patient captured in at least a portion of the series of images.
17 . The system of claim 16 , wherein the pattern provides a visualization of a shape and contours of the surface of the internal anatomy.
18 . The system of claim 16 , wherein the pattern is a first pattern, wherein the graphic further comprises a second pattern projected over at least a portion of the surgical tool captured in at least a portion of the series of images.
19 . The system of claim 10 , wherein the camera is a stereoscopic camera comprising two or more lenses and associated image sensors.
20 . A non-transitory machine-readable medium comprising instructions which when executed by at least one processor of a system causes the system to:
capture, during a surgical procedure and using a camera, a series of images that include a surgical tool and an object within an internal anatomy of a patient;
generate, using the series of images, a three-dimensional (3D) reconstruction of the surgical tool, the object, and the internal anatomy that are included within the series of images;
determine a shortest distance between the surgical tool and the object based on the 3D reconstruction; and
display, during the surgical procedure and on a display, a graphic overlaid on the series of images, the graphic indicating the shortest distance between the surgical tool and the object.