3D corrected imaging
A system and method for corrected imaging including an optical camera that captures at least one optical image of an area of interest, a depth sensor that captures at least one depth map of the area of interest, and circuitry that correlates depth information of the at least one depth map to the at least one optical image to generate a depth image, corrects the at least one optical image by applying a model to address alteration in the respective at least one optical image, the model using information from the depth image, and outputs the corrected at least one optical image for display in 2D and/or as a 3D surface.
1. A system for corrected imaging, comprising:
an optical camera that captures at least one optical image of an area of interest;
a depth sensor that captures at least one depth map of the area of interest; and
circuitry configured to
associate depth information of the at least one depth map to the at least one optical image to generate a depth image,
correct the at least one optical image by applying a distortion model and a deformation model to address alteration in the respective at least one optical image due to undesired effects of distortion caused by imaging abnormalities and image artifacts caused by imaging of a three-dimensional deformed surface, the distortion model using information from the depth image, optical image information associated with the captured at least one optical image, and prior information, the deformation model using the information from the depth image and the optical image information, but not the prior information, and the prior information including intensity and directionality of illumination in the area of interest, and
output the corrected at least one optical image for display in two-dimensions (2D) and/or in three-dimensions (3D).
2. The system for corrected imaging according to claim 1 , wherein the optical camera and the depth sensor are positioned at the distal end of a laparoscopic tool.
3. The system for corrected imaging according to claim 1 , wherein the circuitry is further configured to weigh the correction applied by the distortion model against the correction applied by the deformation model,
output a preview image, and
reapply, based on a determination of quality acceptability of the preview image, correction of the at least one optical image using at least one new system parameter, the at least one new system parameter including a change to one or more of optical image-sensing circuitry, depth-sensing circuitry, camera or depth sensing sensor position, and camera or depth-sensor orientation.
4. The system for corrected imaging according to claim 1 , wherein the distortion model addresses alteration in the respective at least one optical image based on a two dimensional image with additional information, including at least one of expected luminance, spectral response, and depth at each pixel producing a three-dimensional surface and applying this information to determine at least one of an amount of extra light due to reflection, an adjustment in illumination due to surface orientation and occlusion, and an expected radiance and diffusion due to surface roughness.
5. The system for corrected imaging according to claim 1 , wherein the deformation model addresses alteration in the respective at least one optical image by segmenting the depth image into a multitude of polygons with a known size and orientation and grouping the polygons into discrete patches to generate a three-dimensional representation of a surface corresponding to the area of interest.
6. The system for corrected imaging according to claim 1 , wherein the area of interest is a physiological portion of a medical patient.
7. The system for corrected imaging according to claim 1 , wherein the optical camera is configured to capture multispectral or hyperspectral images.
8. The system for corrected imaging according to claim 1 , wherein the depth sensor is a light field camera.
9. The system for corrected imaging according to claim 1 , wherein the depth sensor includes at least two stereo-vision cameras.
10. The system for corrected imaging according to claim 9 , wherein a baseline of the stereo-vision cameras is adjustable.
11. The system for corrected imaging according to claim 1 , wherein the depth sensor utilizes a structured light.
12. The system for corrected imaging according to claim 1 , further comprising an adjustable light source configured to illuminate an object at a particular angle and intensity.
13. The system for corrected imaging according to claim 1 , wherein the optical camera and the depth sensor are tracked by a navigational system.
14. The system for corrected imaging according to claim 1 , wherein the optical camera and the depth sensor are actuated by a robotic system.
15. A method for corrected imaging, comprising:
capturing at least one optical image of an area of interest using an optical camera;
capturing at least one depth map of the area of interest using a depth sensor;
associating depth information of the at least one depth map to the at least one optical image to generate a depth image;
correcting the at least one optical image by applying a distortion model and a deformation model to address alteration in the respective at least one optical image due to undesired effects of distortion caused by imaging abnormalities and image artifacts caused by imaging of a three-dimensional deformed surface, the distortion model using information from the depth image, optical image information associated with the captured at least one optical image, and prior information, the deformation model using the information from the depth image and the optical image information, but not the prior information, and the prior information including intensity and directionality of illumination in the area of interest; and
outputting the corrected at least one optical image for display in 2D and/or as a 3D surface.
16. The method for corrected imaging according to claim 15 , wherein the optical camera and the depth sensor are positioned at the distal end of a laparoscopic tool.
17. The method for corrected imaging according to claim 15 , further comprising:
weighting the correction applied by the distortion model against the correction applied by the deformation model;
outputting a preview image; and
reapplying, based on a determination of quality acceptability of the preview image, correction of the at least one optical image using at least one new system parameter, the at least one new system parameter including a change to one or more of optical image-sensing circuitry, depth-sensing circuitry, camera or depth sensing sensor position, and camera or depth-sensor orientation.
18. The method for corrected imaging according to claim 15 , wherein said correcting further comprises: applying the distortion model to address alteration in the respective at least one optical image based on a two dimensional image with additional information, including at least one of expected luminance, spectral response, and depth at each pixel producing a three-dimensional surface and applying the three-dimensional surface to determine at least one of an amount of extra light due to reflection, an adjustment in illumination due to surface orientation and occlusion, and an expected radiance and diffusion due to surface roughness.
19. The method for corrected imaging according to claim 15 , wherein said correcting applied by the deformation model further comprises:
applying the deformation model to address alteration in the respective at least one optical image by segmenting the depth image into a multitude of polygons with a known size and orientation and grouping the polygons into discrete patches to generate a three-dimensional representation of a surface corresponding to the area of interest.
20. A non-transitory computer readable medium having stored thereon a program that when executed by a computer causes the computer to implement a method for corrected imaging, comprising:
capturing at least one optical image of an area of interest using an optical camera;
capturing at least one depth map of the area of interest using a depth sensor;
associating depth information of the at least one depth map to the at least one optical image to generate a depth image;
correcting the at least one optical image by applying a distortion model and a deformation model to address alteration in the respective at least one optical image due to undesired effects of distortion caused by imaging abnormalities and image artifacts caused by imaging of a three-dimensional deformed surface, the distortion model using information from the depth image, optical image information associated with the captured at least one optical image, and prior information, the deformation model using the information from the depth image and the optical image information, but not the prior information, and the prior information including intensity and directionality of illumination in the area of interest; and
outputting the corrected at least one optical image for display in 2D and/or as a 3D surface.