Methods for optical tracking and surface acquisition in surgical environments and devices thereof
A computer assisted system is disclosed that includes an optical tracking system and one or more computing devices. The optical tracking system includes an RGB sensor and is configured to capture color images of an environment in the visible light spectrum and tracking images of fiducials in the environment in a near-infrared spectrum. The computer assisted system is configured to generate a color image of the environment using the color images, identify fiducial locations using the tracking images, generate depth maps from the color images, reconstruct three-dimensional surfaces of structures based on the depth maps, and output a display comprising the reconstructed three-dimensional surface and one or more surgical objects that are associated with the tracked fiducials. The computer assisted system can further include a monitor or a head-mounted display (HMD) configured to present augmented reality (AR) images during a procedure.
1. An optical tracking system, comprising:
at least one optical module comprising an image sensor;
a structured light module configured to project one or more patterns, in the near-infrared (NIR) spectrum, into an environment;
a non-transitory computer readable medium comprising programmed instructions stored thereon for optical tracking and surface acquisition and one or more processors coupled to the computer-readable medium and configured to execute the stored programmed instructions to:
project at least one of the one or more patterns, with the structured light module, into the environment;
capture, with the at least one optical module, a first image;
capture, with the at least one optical module, a second image;
process the second image to determine tracking information comprising at least a three-dimensional position of each of one or more fiducials within the environment;
generate at least one depth map from the first image;
reconstruct a three-dimensional surface of one or more structures in the environment based on the at least one generated depth map; and
output the reconstructed three-dimensional surface and an indication of one or more additional objects, wherein at least one of the one or more fiducials are affixed to each of the one or more additional objects and each of the one or more additional objects are located based on the tracking information.
2. The system of claim 1 , wherein the image sensor comprises a red-green-blue (RGB) complementary metal-oxide semiconductor (CMOS) image sensor.
3. The system of claim 1 , wherein the first image is a color image; and
wherein the second image is a NIR image.
4. The system of claim 1 , wherein the at least one optical module is in a first housing;
wherein the structured light module is in a second housing; and
wherein the first housing is detached from the second housing.
5. The system of claim 4 , wherein the second housing further comprises a tracking marker.
6. The system of claim 1 , wherein the at least one optical module comprises a first optical module comprising a first image sensor and a second optical module comprising a second image sensor;
wherein the stored programmed instructions to capture the first image comprise stored program instructions to use the first image sensor; and
wherein the stored programmed instructions to capture the second image comprise stored program instructions to use the second image sensor.
7. The system of claim 6 , wherein the first and second optical modules are fixed on a baseline.
8. The system of claim 1 , further comprising one or more NIR light-emitting diodes (LEDs) configured to emit NIR light in the environment, wherein the one or more fiducials comprise passive fiducial markers configured to reflect NIR light.
9. The system of claim 1 , wherein the one or more patterns comprise one or more of a plurality of sequential patterns, a plurality of stripes, a random pattern, a pseudo-random pattern, or a plurality of points.
10. The system of claim 1 , wherein the at least one optical module is rigidly fixed to the structured light module, and wherein the first and second images are in a common referential space.
11. The system of claim 1 , further comprising an augmented reality display configured to display an augmented reality image of one or more portions of the environment generated based on the reconstructed three-dimensional surface.
12. The system of claim 1 , wherein the first image comprises a color image, a Bayer raw image, or a grayscale image.
13. The system of claim 1 , wherein the stored programmed instructions further comprise instructions to:
detect a gesture within the environment via analysis of one or more of the first or second images; and
instruct a computer-assisted system based on the detected gesture to control one or more aspects of a workflow.
14. The system of claim 1 , wherein the at least one optical module comprises a plurality of optical modules;
wherein each of the plurality of optical modules are housed in a separate housing; and
wherein the stored programmed instructions further comprise instructions to:
detect an artifact in at least two of the plurality of optical modules; and
estimate a respective pose of the at least two of the plurality of optical modules, based on the pose of the artifact.
15. The system of claim 1 , wherein the structured light module emits light at a wavelength between 820 nm and 1000 nm.
16. The system of claim 15 , wherein the image sensor is unfiltered in NIR.
17. An optical tracking system, comprising:
at least one optical module comprising an image sensor;
a structured light module configured to project one or more patterns, in the near-infrared (NIR) spectrum, into an environment;
a non-transitory computer readable medium comprising programmed instructions stored thereon for optical tracking and surface acquisition and one or more processors coupled to the computer-readable medium and configured to execute the stored programmed instructions to:
project at least one of the one or more patterns, with the structured light module, into the environment;
capture, with the at least one optical module, a plurality of first images;
capture, with the at least one optical module, a plurality of second images;
for each of the plurality of second images, determine tracking information comprising at least a three-dimensional position of each of one or more fiducials within the environment;
for each of the plurality of first images, generate at least one depth map from each of the plurality of first images and reconstruct a three-dimensional surface of one or more structures in the environment based on the at least one generated depth map;
stitch the plurality of three-dimensional surfaces together to generate a complete three-dimensional surface of the one or more structures; and
output the complete three-dimensional surface and an indication of one or more additional objects, wherein at least one of the one or more fiducials are affixed to each of the one or more additional objects and each of the one or more additional objects are located based on the tracking information.
18. A non-transitory computer readable medium having stored thereon instructions for optical tracking and surface acquisition comprising executable code that, when executed by one or more processors, causes the one or more processors to:
project, with a structured light module of an optical tracking system, a pattern into an environment, wherein the pattern is projected in the near-infrared (NIR) spectrum;
capture, with an image sensor of at least one optical module of the optical tracking system, a first image during the projection of the pattern;
capture, with the image sensor, a second image;
process the second image to determine tracking information comprising at least a three-dimensional position of each of one or more fiducials within the environment;
generate at least one depth map from the first image;
reconstruct a three-dimensional surface of one or more structures in the environment based on the at least one generated depth map; and
output the reconstructed three-dimensional surface and an indication of one or more additional objects, wherein at least one of the one or more fiducials are affixed to each of the one or more additional objects and each of the one or more additional objects are located based on the tracking information.
19. The non-transitory computer readable medium of claim 18 , further comprising executable code that, when executed by one or more processors, causes the one or more processors to:
project, with the structured light module, one or more of a plurality of sequential patterns, a plurality of stripes, a random pattern, a pseudo-random pattern, or a plurality of points.
20. The non-transitory computer readable medium of claim 18 , further comprising executable code that, when executed by one or more processors, causes the one or more processors to:
repeat the projecting, capturing, processing, and generating for a plurality of first and second images;
generate a plurality of three-dimensional surfaces of the one or more structures from a plurality of depth maps; and
stitch the plurality of three-dimensional surfaces to reconstruct the three-dimensional surface of the one or more structures.