Augmented reality ready optical tracking system
A computer assisted system is disclosed that includes an optical tracking system. The optical tracking system includes an RGB sensor and is configured to capture color images of an environment in the visible light spectrum and tracking images of fiducials in the environment in a near-infrared spectrum. The computer assisted surgical system is configured to generate a color image of the environment using the color images, identify fiducial locations using the tracking images, register pre-operative and/or intra-operative data to the color images using the fiducial locations, and generate an augmented reality (AR) image of the environment by overlaying the data over the color image. The computer assisted surgical system can further include a monitor or a head-mounted display (HMD) configured to present the AR image.
1. A method for improved optical tracking, the method comprising:
capturing with a same image sensor a color image and a tracking image of an environment comprising one or more fiducials, wherein the image sensor is a red-green-blue (RGB) complementary metal-oxide semiconductor (CMOS) image sensor, wherein the tracking image is obtained during an emission of near-infrared (NIR) light within the environment, wherein a first integration time for the tracking image is shorter than a second integration time for the color image, and wherein the image sensor is unfiltered with respect to the NIR light;
processing the tracking image to determine tracking information comprising three-dimensional positional data of each of the one or more fiducials within the environment, wherein one or more of the fiducials comprise active or passive fiducial markers configured to emit or reflect the NIR light, respectively;
registering the positional data to the color image based on the tracking information, wherein the tracking information and the color image are in a same referential space; and
outputting to a display device a representation of the environment generated based on the registration and comprising contents of the color image overlaid with the registered data.
2. The method of claim 1 , wherein capturing the color image comprises capturing the color image in a visible light spectrum, wherein capturing the tracking image comprises capturing the tracking image in a NIR spectrum, and wherein capturing the color image and the tracking image comprises alternately capturing the color image and the tracking image.
3. The method of claim 1 , wherein the image sensor comprises an oversampled binary image sensor.
4. The method of claim 1 , further comprising applying to the color image one or more image optimization techniques comprising demosaicing, color correction, gamma correction, automatic exposure, or white balancing.
5. The method of claim 1 , wherein the registered positional data comprises surgical data comprising pre-operative surgical data comprising one or more trajectories, cutting planes, or three-dimensional anatomical reconstructions, and wherein the method further comprises registering the pre-operative surgical data to patient anatomy in the color image.
6. The method of claim 5 , further comprising outputting an alert to the display device when a deviation from a surgical plan is determined to have occurred based on the tracking information.
7. The method of claim 1 , further comprising emitting the NIR light with one or more NIR light-emitting diodes (LEDs) when the one or more of the fiducials comprise passive fiducial markers.
8. The method of claim 1 , wherein the representation comprises an augmented reality (AR) image of one or more portions of the environment, and wherein the display device comprises a head-mounted display (HMD).
9. The method of claim 1 , further comprising tracking one or more of a location or an orientation of one or more objects in the environment based on an analysis of the color image to identify one or more optical codes attached to each of the one or more objects.
10. The method of claim 1 , further comprising automatically segmenting one or more anatomical structures from the color image.
11. The method of claim 1 , wherein the registered positional data overlaid on the contents of the color image comprises one or more trajectories of one or more surgical tools in the environment.