Markerless navigation using AI computer vision
Provided herein are devices, systems, and methods for a three-dimensional registering, tracking, and/or guiding an object of interest, such a body part, a surgical tool, or an implant, during a surgical procedure. Such devices, systems, and methods may offer minimally invasive, high precision registering, tracking, and/or guiding of the object of interest using a patterned light beam and data processing using artificial intelligence. The methods, devices, and systems disclosed herein may be compatible with a simple marker or markers placed by a minimally invasive method on the object of interest.
1 . A system for tracking an object in real-time or near real-time during a surgery, the system comprising:
a memory; and
a processor in communication with the memory and configured to:
a) receive, during the surgery, a plurality of images of a patterned light beam passed through a patterned filter and projected onto the object during the surgery, wherein the plurality of images comprise a contour of the object;
b) input the plurality of images into a software module during the surgery;
c) determine, with the software module during the surgery, an output of a location, orientation, or rotation of the object in three-dimensional space; and
d) generate, during the surgery, a three-dimensional visual representation of the object based at least in part on the location, orientation, or rotation of the object in three- dimensional space.
2 . The system of claim 1 , wherein the processor is configured to display the three-dimensional visual representation of the object.
3 . The system of claim 1 , wherein the processor is configured to repeat steps (a)-(c) to track the object during the surgery.
4 . The system of claim 1 , wherein the generating the three-dimensional visual representation provides real-time tracking of the object.
5 . The system of claim 1 , wherein the surgery comprises a robot-assisted surgery.
6 . The system of claim 1 , wherein the surgery comprises an orthopedic surgery.
7 . The system of claim 1 , wherein the surgery comprises an arthroplasty, a total joint replacement, or a partial joint replacement.
8 . The system of claim 1 , wherein the object comprises a knee joint, a hip joint, an elbow joint, a shoulder joint, an ankle joint, or a combination thereof.
9 . The system of claim 1 , wherein the system is configured for tracking the object without the use of a marker placed on the object.
10 . The system of claim 1 , wherein the patterned light beam comprises an infrared wave.
11 . The system of claim 1 , wherein the plurality of images are captured by a video camera.
12 . The system of claim 1 , wherein the three-dimensional visual representation of the object is registered to a prior image of the object obtained before the surgery.
13 . The system of claim 12 , wherein the prior image of the object comprises a computer tomography scan, a magnetic resonance imaging scan, an x-ray scan, a positron emission tomography scan, an ultrasound scan, or a combination thereof.
14 . The system of claim 1 , wherein a display comprising the three-dimensional visual representation of the object is configured to be manipulated by a user.
15 . The system of claim 14 , wherein the manipulation by the user comprises zooming in and out, rotating, or moving the display comprising the three-dimensional visual representation of the object.
16 . The system of claim 1 , wherein the software module comprises a neural network.
17 . The system of claim 16 , wherein the software module comprises a multilayer perceptron.
18 . The system of claim 1 , wherein the three-dimensional visual representation of the object comprises a recommendation, guidance, or instruction for performing the surgery.
19 . A non-transitory computer-readable storage medium, having a computer executable instruction executable by one or more processors to:
a) receive, during a surgery, a plurality of images of a patterned light beam passed through a patterned filter and projected onto an object during the surgery;
b) input the plurality of images into a software module, wherein the plurality of images comprise a contour of the object;
c) determine, with the software module, an output of a location, orientation, or rotation of the object in three-dimensional space during the surgery, based on the plurality of images; and
d) generate a three-dimensional visual representation of the object based at least in part on the location, orientation, or rotation of the object in three-dimensional space during the surgery.
20 . A system for real-time or near real-time tracking a joint during an orthopedic surgery, the system comprising:
a memory; and
a processor in communication with the memory and configured to:
a) receive, during the orthopedic surgery, a plurality of images of a patterned light beam passed through a patterned filter and projected onto the joint during the orthopedic surgery;
b) input the plurality of images into a software module, wherein the plurality of images comprise a contour of the joint;
c) determine, with the software module, an output of a location, orientation, or rotation of the joint in three-dimensional space during the orthopedic surgery, based on the plurality of images;
d) generate a three-dimensional visual representation of the joint in real-time or near real-time based at least in part on the location, orientation, or rotation of the joint in three-dimensional space during the orthopedic surgery, wherein the three-dimensional visual representation of the joint is registered to an image of the joint; and
e) display the three-dimensional visual representation of the joint, wherein the three-dimensional visual representation of the joint is configured to be manipulated by a user.