Automatic segmentation and registration system and method
A holographic augmented reality visualization and guidance system for performing a medical procedure includes an augmented reality system for displaying operating information to a user. The operating information can include preoperative data, intraoperative data, and fused data of an anatomical structure. A computer system is in communication with the augmented reality system and is configured to selectively generate the fused data by merging the preoperative data and the intraoperative data, identify deformation of the anatomical structure via differences between the preoperative data and the intraoperative data, transmit the operating information to the augmented reality system, and compensate for the deformation of the anatomical structure according to the deformation engine in real-time.
1 . A holographic augmented reality visualization and guidance system for performing a medical procedure on an anatomical structure of a subject by a user, comprising:
an augmented reality system configured to show a plurality of operating information to the user, the plurality of operating information including at least one of preoperative data of the anatomical structure, intraoperative data of the anatomical structure, and fused data of the anatomical structure; and
a computer system in communication with the augmented reality system, the computer system having a deformation engine, the computer system including a graphics and physics engine configured to set an upper boundary condition and a lower boundary condition on the preoperative data based on a characteristic of the anatomical structure,
wherein the computer system is configured to selectively generate the fused data by merging the preoperative data and the intraoperative data, identify deformation of the anatomical structure via a difference between the preoperative data and the intraoperative data, transmit the plurality of operating information including the fused data to the augmented reality system, and compensate for the deformation of the anatomical structure according to the deformation engine in real-time.
2 . The system of claim 1 , wherein the deformation engine automatically compensates for the deformation of the anatomical structure by aligning the preoperative data with the intraoperative data through artificial intelligence and machine learning before the computer system transmits the plurality of operating information to the augmented reality system.
3 . The system of claim 1 , wherein the deformation engine manually compensates for the deformation of the anatomical structure by allowing the user to manually align the preoperative data with the intraoperative data.
4 . The system of claim 1 , wherein the computer system generates the fused data by super-imposing the intraoperative data over the preoperative data.
5 . The system of claim 1 , wherein the computer system generates the fused data by super-imposing the preoperative data over the intraoperative data.
6 . A method for holographic augmented reality visualization and guidance in performing a medical procedure on a patient by a user, the method comprising the steps of:
providing a system having an augmented reality system configured to display a plurality of operating information of the patient in an augmented reality environment, a first holographic image acquisition system, a second holographic image acquisition system, and a computer system having a deformation engine, the computer system further including a graphics and physics engine configured to set an upper boundary condition and a lower boundary condition on the preoperative data based on a characteristic of the preoperative data;
acquiring, by the first holographic image acquisition system, preoperative data from the patient;
acquiring, by the second holographic image acquisition system, intraoperative data from the patient;
selectively registering the preoperative data to the plurality of operating information;
selectively registering the intraoperative data to the plurality of operating information;
selectively generating, by the computer system, fused data;
selectively adjusting the preoperative data according to the deformation engine where the preoperative data does not align with the intraoperative data;
transmitting, by the computer system, the plurality of operating information to the augmented reality system;
selectively displaying, by the augmented reality system, the intraoperative data where the intraoperative data permits the user to navigate a body of a patient;
selectively displaying, by the augmented reality system, the preoperative data where the intraoperative data does not substantially permit the user to navigate the body of the patient;
selectively displaying, by the augmented reality system, the fused data where both the preoperative data and the intraoperative data permit the user to navigate the body of the patient; and
selectively adjusting, by the user, the plurality of operating information in real-time.
7 . The method of claim 6 , wherein the first holographic image acquisition system is selected from a group consisting of a computerized tomography (CT) apparatus, cone beam computed tomography (CBCT) apparatus, a magnetic resonance imaging (MRI) apparatus, a projectional radiography apparatus, a positron emission tomography (PET) apparatus, a volumetric ultrasound and fluoroscopy system, and combinations thereof.
8 . The method of claim 6 , wherein the second holographic image acquisition system is selected from a group consisting of a general ultrasound, a transesophageal ultrasound, an endoscopic ultrasound, a point of care ultrasound, an ultrasound echocardiogram (ECG) imaging apparatus, a fluoroscopy apparatus, a transthoracic echocardiogram (TTE), a transesophageal echocardiogram (TEE), an intracardiac echocardiogram (ICE), and combinations thereof.