Systems and methods for AI-assisted surgery
Various embodiments of the invention provide systems and methods to assist or guide an arthroscopic surgery or other surgical procedure e.g., surgery of the shoulder, knee or hip. The method comprises steps of receiving an image from an interventional imaging device, identifying a feature in the image using an image recognition algorithm, overlaying the features on a video feed on a display device and making recommendations or suggestions to an operator based on the identified feature in the image.
1 . A system for implementing a hierarchical pipeline for guiding a minimally invasive procedure, the system comprising one or more computer processors and one or more non-transitory computer-readable storage media storing instructions that are operable, when executed by said one or more computer processors, to cause said one or more computer processors to perform operations comprising:
receiving at least one image captured by an interventional imaging device;
identify, by at least one upstream module, one or more image features of a region of treatment or a portal of entry in the at least one image based on said at least one upstream module, wherein said at least one upstream module comprises a first trained image processing algorithm;
transferring the received image to a first downstream module to identify one or more image features of a pathology in the at least one image identified by the at least one upstream module, wherein said first downstream module comprises a second trained image processing algorithm;
transferring the received image to a second downstream module to identify one or more image features of a surgical tool, a surgical tool element, an operational procedure or action relating to said minimally invasive procedure in the at least one image identified by the at least one upstream module, wherein said second downstream module comprises a third trained image processing algorithm;
labeling said one or more image features identified by the upstream module, the first downstream module, and the second downstream module;
displaying said labeled one or more image features identified by the upstream module, the first downstream module, and the second downstream module in said at least one image continuously to an operator in the course of said minimally invasive procedure; and
discarding the received image when the at least one upstream module fails to identify one or more image features of a region of treatment in the at least one received image.
2 . The system of claim 1 , wherein activating the first downstream module is independent from activating the second downstream module.
3 . The system of claim 1 , wherein said first, second, or third trained image processing algorithms comprise at least a machine learning algorithm, a deep learning algorithm, or a combination of both and is trained using at least one training dataset.
4 . The system of claim 3 , wherein said training dataset is configured for a shoulder surgery or for a knee surgery.
5 . The system of claim 3 , wherein said training dataset comprises a plurality of training images comprising one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies.
6 . The system of claim 5 , wherein said first, second, or third trained image processing algorithms further comprise a plurality of augmentation techniques performed on the training images.
7 . The system of claim 6 , wherein said augmentation techniques comprise rotating said training images.
8 . The system of claim 6 , wherein said augmentation techniques comprise flipping said training images along a vertical axis.
9 . The system of claim 6 , wherein said augmentation techniques comprise enlarging or cropping said training images.
10 . The system of claim 1 , wherein said first, second, or third trained image processing algorithms store said displayed images with labeled features in a memory device.
11 . The system of claim 1 , wherein said first, second, or third trained image processing algorithms discard said displayed images with labeled features to minimize memory usage.
12 . A method for implementing a hierarchical pipeline for guiding a minimally invasive procedure, the method comprising:
receiving at least one image captured by an interventional imaging device;
identify, by at least one upstream module, one or more image features of a region of treatment or a portal of entry in the at least one image based on said at least one upstream module, wherein said at least one upstream module comprises a first trained image processing algorithm;
transferring the received image to a first downstream module to identify one or more image features of a pathology in the at least one image identified by the at least one upstream module, wherein said first downstream module comprises a second trained image processing algorithm;
transferring the received image to a second downstream module to identify one or more image features of a surgical tool, a surgical tool element, an operational procedure or action relating to said minimally invasive procedure in the at least one image identified by the at least one upstream module, wherein said second downstream module comprises a third trained image processing algorithm;
labeling said one or more image features identified by the upstream module, the first downstream module, and the second downstream module;
displaying said labeled one or more image features identified by the upstream module, the first downstream module, and the second downstream module in said at least one image continuously to an operator in the course of said minimally invasive procedure; and
discarding the received image when the at least one upstream module fails to identify one or more image features of a region of treatment in the at least one received image.
13 . The method of claim 12 , wherein activating the first downstream module is independent from activating the second downstream module.
14 . The method of claim 12 , wherein said first, second, or third trained image processing algorithms comprise a machine learning or a deep learning algorithms and is trained using at least one training dataset.
15 . The method of claim 14 , wherein said training dataset is configured for a shoulder surgery or for a knee surgery.
16 . The method of claim 14 , wherein said training dataset comprises a plurality of training images comprising one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies.
17 . The method of claim 14 , wherein the first, second, or third trained image processing algorithms further include a plurality of augmentation techniques.
18 . The method of claim 17 , wherein said augmentation techniques comprise rotating said training images.
19 . The method of claim 17 , wherein said augmentation techniques comprise flipping said training images along a vertical axis.
20 . The method of claim 17 , wherein said augmentation techniques comprise enlarging or cropping said training images.
21 . The method of claim 12 , wherein said first, second, or third trained image processing algorithms store said displayed images with labeled features in a memory device.
22 . The method of claim 12 , wherein said first, second, or third trained image processing algorithms discard said displayed images with labeled features to minimize memory usage.