Extended-reality operating room for robotic surgery

1. A computer-implemented method for design of an immersive experiential extended-reality (XR) environment for a robotic surgical procedure, the method comprising:

generating the immersive experiential XR environment by associating at least one virtual model of at least one or more surgical tools and at least one surgical robot with at least one image of a patient,

wherein the immersive experiential XR environment comprises a three-dimensional (3D) digital twin of an anatomy of the patient and a simulation of a physical presence of a first user immersively collaborating with a second user for simulating the surgical procedure;

generating, using a machine learning model, a robotic process for performing the surgical procedure,

wherein the robotic process is generated based on features extracted from the at least one image of the patient and a surgical plan for the surgical procedure;

displaying the immersive experiential XR environment to the first user on a first VR headset and the second user on a second VR headset to allow viewing of,

the simulation of the physical presence of the first user immersively collaborating with the second user and one or more actions performed by the first user or the second user in the immersive experiential XR environment, and

one or more virtual surgical steps performed on the 3D digital twin according to the surgical plan;

receiving a selection of at least two objects,

wherein the at least two objects are displayed in the immersive experiential XR environment;

determining that the virtual surgical steps have been performed on the 3D digital twin by the one or more actions;

adjusting the robotic process based on the virtual surgical steps; and

transmitting the adjusted robotic process to the at least one surgical robot to configure the at least one surgical robot with the adjusted robotic process,

wherein the adjusted robotic process comprises the at least two objects and information describing the virtual surgical steps,

wherein the at least one surgical robot is configured to perform the robotic surgical procedure on the patient according to the adjusted robotic process and monitor the adjusted robotic process being performed by the at least one surgical robot.

2. The method of claim 1 , wherein generating the robotic process comprises:

determining a surgery type for the surgical procedure;

determining a correlation coefficient for a data entry corresponding to the surgery type;

based on determining that the correlation coefficient is greater than a threshold correlation coefficient, determining a highest correlated data entry; and

generating the robotic process based on the highest correlated data entry.

3. The method of claim 1 , comprising displaying the adjusted robotic process on a heads-up display.

4. The method of claim 1 , comprising:

detecting an abnormality in the surgical procedure; and

generating a warning based on the abnormality for display in the immersive experiential XR environment.

5. The method of claim 1 , wherein transmitting the adjusted robotic process comprises:

identifying a sensitive body part; and

transmitting an indication of the sensitive body part to the at least one surgical robot.

6. The method of claim 1 , wherein displaying the 3D digital twin comprises:

extracting imaging data corresponding to the patient from a patient database, wherein the imaging data comprises a cross-sectional magnetic resonance imaging image; and

generating the 3D digital twin based on the imaging data.

7. The method of claim 1 , comprising:

retrieving historical data associated with a historical surgical procedure;

generating a color code to distinguish between the historical surgical procedure and the surgical procedure; and

based on the historical data and the color code, displaying the historical surgical procedure and the surgical procedure simultaneously.

8. The method of claim 1 , wherein adjusting the robotic process based on the virtual surgical steps comprises:

determining that at least one of the first user or the second user made an adjustment to the virtual surgical steps, wherein the adjustment comprises a surgical step that differs from a step in the surgical plan;

determining an identifier of the at least one of the first user or the second user; and

storing the adjustment and the identifier in a surgery database.

9. A robotic surgical system for design of an immersive experiential XR environment for a robotic surgical procedure, comprising:

a non-transitory, computer-readable storage medium storing computer instructions, which when executed by one or more computer processors cause the robotic surgical system to:

generate the immersive experiential XR environment by associating at least one virtual model of at least one or more surgical tools and at least one surgical robot with at least one image of a patient,

wherein the immersive experiential XR environment comprises a three-dimensional (3D) digital twin of an anatomy of the patient and a simulation of a physical presence of a first user immersively collaborating with a second user for simulating the surgical procedure;

generate, using a machine learning model, a robotic process for performing the surgical procedure,

wherein the robotic process is generated based on features extracted from the at least one image of the patient and a surgical plan for the surgical procedure;

display the immersive experiential XR environment to the first user on a first VR headset and a second user on the second VR headset to allow viewing of,

the simulation of the physical presence of the first user immersively collaborating with the second user and one or more actions performed by the first user or the second user in the immersive experiential XR environment, and

one or more virtual surgical steps performed on the 3D digital twin according to the surgical plan;

receive a selection of at least two objects,

wherein the at least two objects are displayed in the immersive experiential XR environment for viewing by the first user and the second user;

determine that the virtual surgical steps have been performed on the 3D digital twin by the one or more actions;

adjust the robotic process based on the virtual surgical steps; and

transmit the adjusted robotic process to the at least one surgical robot to configure the at least one surgical robot with the adjusted robotic process,

wherein the adjusted robotic process comprises the at least two objects and information describing the virtual surgical steps, and

wherein the at least one surgical robot is configured to perform the robotic surgical procedure on the patient according to the adjusted robotic process and monitor the adjusted robotic process being performed by the at least one surgical robot.

10. The robotic surgical system of claim 9 , wherein the computer instructions for generating the robotic process cause the robotic surgical system to:

determine a surgery type for the surgical procedure;

determine a correlation coefficient for a data entry corresponding to the surgery type;

based on determining that the correlation coefficient is greater than a threshold correlation coefficient, determine a highest correlated data entry; and

generate the robotic process based on the highest correlated data entry.

11. The robotic surgical system of claim 9 , wherein the computer instructions for generating the robotic process cause the robotic surgical system to display the adjusted robotic process on a heads-up display.

12. The robotic surgical system of claim 9 , wherein the computer instructions cause the robotic surgical system to:

detect an abnormality in the surgical procedure; and

generate a warning based on the abnormality for display in the immersive experiential XR environment.

13. The robotic surgical system of claim 9 , wherein the computer instructions for transmitting the adjusted robotic process cause the robotic surgical system to:

identify a sensitive body part; and

transmit an indication of the sensitive body part to the at least one surgical robot.

14. The robotic surgical system of claim 9 , wherein the computer instructions for displaying the 3D digital twin cause the robotic surgical system to:

extract imaging data corresponding to the patient from a patient database, wherein the imaging data comprises a cross-sectional magnetic resonance imaging image; and

generate the 3D digital twin based on the imaging data.

15. The robotic surgical system of claim 9 , wherein the computer instructions cause the robotic surgical system to:

retrieve historical data associated with a historical surgical procedure;

generate a color code to distinguish between the historical surgical procedure and the surgical procedure; and

based on the historical data and the color code, display the historical surgical procedure and the surgical procedure simultaneously.

16. The robotic surgical system of claim 9 , wherein the computer instructions for adjusting the robotic process based on the virtual surgical steps cause the robotic surgical system to:

determine that at least one of the first user or the second user made an adjustment to the virtual surgical steps, wherein the adjustment comprises a surgical step that differs from a step in the surgical plan;

determine an identifier of the at least one of the first user or the second user; and

store the adjustment and the identifier in a surgery database.

17. A surgical robot for design of an immersive experiential XR environment for a robotic surgical procedure, comprising:

one or more computer processors; and

a non-transitory, computer-readable medium storing computer instructions, which when executed by the one or more computer processors cause the surgical robot to:

generate the immersive experiential XR environment by associating at least one virtual model of at least one or more surgical tools and at least one surgical robot with at least one image of a patient,

wherein the immersive experiential XR environment comprises a three-dimensional (3D) digital twin of an anatomy of the patient and a simulation of a physical presence of a first user immersively collaborating with a second user for simulating the surgical procedure;

generate, using a machine learning model, a robotic process for performing the surgical procedure,

wherein the robotic process is generated based on features extracted from the at least one image of the patient and a surgical plan for the surgical procedure;

display, via an electronic display, the 3D digital twin within the immersive experiential XR environment to the first user on a first VR headset and the second user on a second VR headset to allow viewing of,

the simulation of the physical presence of the first user immersively collaborating with the second user and one or more actions performed by the first user or the second user in the immersive experiential XR environment, and

one or more virtual surgical steps performed on the 3D digital twin according to the surgical plan;

receive a selection of at least two objects,

wherein the at least two objects are displayed in the immersive experiential XR environment for viewing by the first user the second user;

determine that virtual surgical steps have been performed on the 3D digital twin by the one or more actions;

adjust the robotic process based on the virtual surgical steps; and

transmit the adjusted robotic process to the at least one surgical robot to configure the at least one surgical robot with the adjusted robotic process,

wherein the adjusted robotic process comprises the at least two objects and information describing the virtual surgical steps, and

wherein the at least one surgical robot is configured to perform the robotic surgical procedure on the patient according to the adjusted robotic process and monitor the adjusted robotic process being performed by the at least one surgical robot.

18. The surgical robot of claim 17 , wherein the computer instructions for generating the robotic process cause the surgical robot to:

determine a surgery type for the surgical procedure;

determine a correlation coefficient for a data entry corresponding to the surgery type;

based on determining that the correlation coefficient is greater than a threshold correlation coefficient, determine a highest correlated data entry; and

generate the robotic process based on the highest correlated data entry.

19. The surgical robot of claim 17 , wherein the computer instructions for generating the robotic process cause the surgical robot to display the adjusted robotic process on a heads-up display.

20. The surgical robot of claim 17 , wherein the computer instructions cause the surgical robot to:

detect an abnormality in the surgical procedure; and

generate a warning based on the abnormality for display in the immersive experiential XR environment.