Utilization of multiple imagers and computational photography in endoscopy
An endoscopy system having a low-profile multi-imager endoscope. The system is capable of using computational photography to provide enhanced output images using techniques such as super-resolution, foveation, magnification, and two-dimensional to three-dimensional conversion. The enhanced output images can improve clinical decision making and patient treatment. Signals from multiple imagers may be used to affect/adjust handing characteristics of the endoscope or direct semi-robotic guidance thereof.
1 . An endoscopy system comprising:
an elongate insertion tube having a proximal end and a distal end, and an open working channel there through;
two or more distally-facing image sensors mounted at the distal end of the elongate insertion tube, wherein each of the two or more distally-facing image sensors is operatively connected to a corresponding image processing unit (IPU) disposed in the elongate insertion tube;
one or more inertial measurement units (IMUs) disposed on the elongate insertion tube, the one or more IMUs measuring IMU data comprising acceleration, angular rate, and magnetic field surrounding the elongate insertion tube;
one or more light sources fitted at the distal end of the elongate insertion tube; and
at least one computer processor operatively connected to each of the IPUs disposed in the elongate insertion tube, the at least one computer processor being configured for applying computational photography to data received from each of the IPUs disposed in the elongate insertion tube to create an enhanced output image, and further configured for associating the data received from each of the IPUs with the IMU data to remove navigation errors of the elongate insertion tube and create an image pose datum.
2 . The endoscopy system of claim 1 , wherein the enhanced output image is a result of one or more computational photography techniques selected from super-resolution, foveation, magnification, and two-dimensional to three-dimensional conversion.
3 . The endoscopy system of claim 1 , wherein the at least one computer processor is selected from an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA), and is disposed in a handle connected to the proximal end of the elongate insertion tube or in an endoscopy tower.
4 . The endoscopy system of claim 1 , wherein the at least one computer processor comprises a computer circuit dedicated to performing the computational photography and a computer dedicated to performing functions other than the computational photography.
5 . The endoscopy system of claim 1 , wherein the at least one computer processor is configured for applying the computational photography using an image processing algorithm selected from CLEAN and regularized maximum-likelihood (RML) image restoration.
6 . The endoscopy system of claim 1 , wherein the two or more distally-facing image sensors comprise a same type or different types of image sensors selected from color sensors, infrared sensors and grayscale sensors.
7 . The endoscopy system of claim 1 , wherein the one or more light sources comprise a same type or different types of light sources selected from distally-mounted light-emitting diodes (LEDs) or fiber optic light guides.
8 . The endoscopy system of claim 1 , wherein the two or more distally-facing image sensors consist of four ultra-small 200×200 pixel image sensors measuring 0.575 mm×0.575 mm, the open working channel has a diameter of 2.1 mm, and the elongate insertion tube has an outer diameter of 3.25 mm.
9 . The endoscopy system of claim 1 , further comprising a computer circuit disposed in a handle connected to the proximal end of the elongate insertion tube, the computer circuit being operatively connected to each of the IPUs in the endoscopy system and being configured for applying the computational photography to the data received from each of the IPUs disposed in the elongate insertion tube to create the enhanced output image.
10 . The endoscopy system of claim 1 , wherein the one or more IMUs provide pitch, roll, and yaw as well as linear movement.
11 . The endoscopy system of claim 1 , wherein the IMU data is utilized to calculate attitude, angular rates, linear velocity and position relative to a global reference frame.
12 . The endoscopy system of claim 1 , wherein the at least one computer processor is configured to create a digital three-dimensional or spatial image map for an anatomy model by associating a sampling of image frames with a discrete IMU pose data point to create a discrete image pose datum.
13 . The endoscopy system of claim 12 , wherein the at least one computer processor is configured to progressively stitch together the sampling of image frames using an associated discrete IMU pose data point to orient the sampling of image frames in a set of three-dimensional planes surrounding a path, thereby creating the digital three-dimensional or spatial image map for the anatomy model.