Markerless navigation using AI computer vision
Provided herein are devices, systems, and methods for a three-dimensional registering, tracking, and/or guiding an object of interest, such a body part, a surgical tool, or an implant, during a surgical procedure. Such devices, systems, and methods may offer minimally invasive, high precision registering, tracking, and/or guiding of the object of interest using a patterned light beam and data processing using artificial intelligence. The methods, devices, and systems disclosed herein may be compatible with a simple marker or markers placed by a minimally invasive method on the object of interest.
1. A computer-based method for tracking an object of interest comprising:
(a) inputting data of an image comprising a light beam projected onto a contour of an object of interest into a software module using a processor;
(b) applying a first set of predetermined number (N) of convolution filters to the data of the image to generate first filtered images and merging the first filtered images into a first merged image using the software module;
(c) quantizing the data of the image by dividing the data of the image in to M bins using a comb mask having M teeth and selecting for pixel data above a threshold in the data divided into M bins using the software module;
(d) reconstructing a three-dimensional profile from the image using the software module;
(e) converting the three-dimensional profile to a two-dimensional profile using the software module;
(f) generating a feature vector by normalizing and concatenating the two-dimensional profile using the software module; and
(g) generating a pose vector by inputting the feature vector to a machine learning model, wherein the pose vector provides at least one of location, orientation, and rotation of the object of interest.
2. The method of claim 1 , wherein the method further comprises segmenting the data of the image into at least two segmented data of the image, wherein a number of segmented data is determined by a number of featured patterns of the light beam.
3. The method of claim 1 , wherein step (b) further comprises applying a second set of predetermined number (N) of convolution filters to the first merged image to generate the predetermined number (N) of second filtered images and merging the second filtered images into a second merged image.
4. The method of claim 1 , wherein the convolution filter comprises a two-dimensional convolution filter of D×D×1 dimension.
5. The method of claim 1 , wherein M of M teeth of the comb mask is sufficiently high to cover the data of the image.
6. The method of claim 1 , wherein the selecting for pixel data above a threshold in step (c) uses a bitwise AND operator.
7. The method of claim 1 , wherein the reconstructing the three-dimensional profile in step (d) comprises applying triangulation technique to the data of the image.
8. The method of claim 1 , wherein converting the three-dimensional profile to the two-dimensional profile in step (e) comprises transforming the three-dimensional profile to a local coordinate system.
9. The method of claim 1 , wherein the feature vector comprises normalized and concatenated two-dimensional profiles generated from all segmented data of the image.
10. The method of claim 1 , wherein the machine learning model comprises a neural network.
11. The method of claim 10 , wherein the neural network comprises a convolutional neural network with fully connected layers for regression of the pose vector.
12. The method of claim 1 , wherein the machine learning model is trained.
13. The method of claim 1 , wherein the convolution filter is trained.
14. The method of claim 1 , wherein the light beam from the light source passes through a patterned filter.
15. The method of claim 14 , wherein the patterned filter has a patterned slit in a crosshair shape and the light beam has a crosshair pattern.
16. The method of claim 1 , wherein the light beam from the source comprises one or more wavelengths.
17. The method of claim 1 , wherein the object of interest comprises a body part of a subject.
18. The method of claim 17 , wherein the body part comprises bones of a joint.
19. The method of claim 1 , wherein the method occurs during a surgical procedure.
20. The method of claim 1 , wherein the steps (a)-(g) are repeated during a surgical procedure.