Steerable endoscope system with augmented view
A steerable endoscope system is provided with an augmented view. An augmented reality display includes a rendered model of an anatomical structure corresponding to a patient anatomy pinned in the field of view, an endoscope marker moving through the virtual model, and an image from an endoscope at a location within the patient anatomy.
1. A computer-implemented method for generating an augmented reality display over a field of view, comprising:
receiving, at a controller, a position signal from an endoscope, the position signal comprising position, orientation, or movement data of a steerable distal end of the endoscope;
receiving, at the controller, sensor signals from one or more sensors of the endoscope, the sensor signals comprising real-time data including a two-dimensional (2D) camera image associated with the current position of the endoscope;
rendering, at the controller, virtual objects comprising:
a three-dimensional anatomical model registered to a real object in the field of view;
an endoscope marker positioned within the anatomical model at the current position of the endoscope; and
a three-dimensional image portion, based on the 2D camera image, mapped onto the three-dimensional anatomical model in front of the endoscope marker; and
displaying the virtual objects through a head-mounted viewer.
2. The method of claim 1 , further comprising updating the virtual objects according to the position signal and the 2D camera image from the one or more sensors of the endoscope.
3. The method of claim 2 , wherein updating the virtual objects comprises moving the endoscope marker and the three-dimensional image portion within the anatomical model along with movement of the endoscope.
4. The method of claim 1 , wherein the endoscope marker comprises a tubular body and a camera frustum.
5. The method of claim 1 , wherein the virtual objects further include a floating window comprising the 2D camera image.
6. The method of claim 1 , wherein the three-dimensional image portion includes texturing of the three-dimensional anatomical model with features identified in the 2D camera image.
7. The method of claim 6 , wherein the features identified in the 2D camera image include one of a tracheal ring or a tool.
8. The method of claim 7 , wherein the features identified in the 2D camera image further include an anatomy landmark.
9. The method of claim 1 , wherein the three-dimensional image portion is mapped onto a region of the three-dimensional anatomical model that correlates to a real world field of view of an endoscope camera capturing the 2D camera image.
10. The method of claim 1 , wherein the one or more sensors comprise an ultrasound transducer or a time of flight sensor.
11. The method of claim 1 , further comprising:
identifying, at the controller, a deviation between the real-time data and the anatomical model; and
generating a notification, at the head-mounted display, of the deviation.