Augmented-reality navigation system for a medical robot
An augmented-reality navigation system is used to assist a practitioner during a surgical operation on an anatomical structure of interest of a patient. The operation is planned on a pre-operational medical image. The navigation system comprises a camera that is intended to be worn by the practitioner on his head, a display device for displaying, in real-time, the real images acquired by the camera and an augmented-reality content superposed on the real images, and a control unit connected to the camera and to the display device. The control unit is configured to generate a predictive model of the movement of a marker placed in proximity to the anatomical structure of interest on the basis of the movement followed by the marker during one or more respiratory cycles of the patient. The predictive model is used to determine an opportune moment to perform the surgical operation as planned on the pre-operational image, and/or to display in augmented reality a three-dimensional anatomical model of the anatomical structure of interest.
1. An augmented-reality navigation system for assisting a practitioner when performing a surgical intervention on an anatomy of interest of a patient, said system comprising:
a camera for acquiring real images, said camera being intended to be worn by the practitioner on their head,
a display device for displaying, in real-time, the real images acquired by the camera together with augmented-reality content overlaid on said real images, and
a control unit of the augmented reality navigation system connected to the camera and to the display device, wherein said control unit of the augmented reality navigation system is configured to:
detect, on the real images, the position/orientation of a marker placed on the patient in the proximity of said anatomy of interest,
record the movement followed by the marker during a recording period corresponding to one or more respiratory cycles of the patient,
determine the position/orientation of the marker at a first instant (t 1 ) belonging to said recording period and corresponding to an instant of acquisition of a pre-intervention medical image of the anatomy of interest of the patient by a medical imaging device,
formulate a predictive model of the movement of the marker from the movement followed by the marker during the recording period;
determine, from the predictive model, a second instant (t 2 ) corresponding to a candidate instant of insertion of a medical instrument into the anatomy of interest of the patient,
compare the position/orientation of the marker at the first instant (t 1 ) with the position/orientation of the marker at the second instant (t 2 ), and
display a result of the comparison on the display device in the form of augmented-reality content overlaid on the real images.
2. The system according to claim 1 , wherein the control unit of the augmented-reality navigation system is further configured to receive the pre-intervention medical image from the medical imaging device and to generate, from said pre-intervention medical image, a three-dimensional anatomical model of the anatomy of interest of the patient and to display said anatomical model on the display device in the form of augmented-reality content overlaid on the real images, the position/orientation of the anatomical model overlaid on the real images being updated continuously and in real time according to the predictive model of the movement of the marker and according to a biomechanical model of anatomical structures of the human body.
3. The system according to claim 2 , wherein the control unit of the augmented-reality navigation system is further configured to display, on the anatomical model, in the form of augmented-reality content overlaid on the real images, an intended path to be followed by a medical instrument, said path being updated in real time according to the position/orientation of the anatomical model.
4. The system according to claim 3 , wherein the intended path to be followed by the medical instrument is defined beforehand on a preoperative medical image, and the control unit of the augmented-reality navigation system is configured to receive said preoperative image and to realign the preoperative image to the pre-intervention medical image in order to display the path on the anatomical model.
5. The system according to claim 3 , further comprising an interaction device allowing the practitioner to target a particular location on the anatomical model and wherein the intended path to be followed by the medical instrument is defined by the practitioner using the interaction device.
6. The system according to claim 2 , wherein the control unit of the augmented-reality navigation system is configured to segment at least one element on the anatomical model from among the following elements:
different anatomical structures within the anatomy of interest,
a region to be treated within the anatomy of interest,
an ablation region estimated from parameters of the surgical intervention, and
ablation margins for a region to be treated within the anatomy of interest and determined by comparing the region to be treated and the estimated ablation region, and to display said segmented element on the anatomical model in the form of augmented-reality content overlaid on the real images, the segmented element being updated in real time according to the position/orientation of the anatomical model.
7. The system according to claim 1 , wherein the control unit of the augmented-reality navigation system is configured to display, on the display device, in the form of augmented-reality content, at least one virtual configuration object regarding a surgical treatment parameter, and the system comprises an interaction device allowing the practitioner to interact with said virtual configuration object in order to select a particular value for said parameter.
8. The system according to claim 1 , wherein the display device is a display screen integrated into an augmented-reality headset.
9. The system according to claim 1 , wherein the control unit of the augmented-reality navigation system is configured to modify the position/orientation of augmented-reality content according to the position/orientation of the head of the practitioner.
10. The system according to claim 9 , wherein the position/orientation of the head of the practitioner is determined using a practitioner-head motion sensor.
11. An assembly formed by an augmented-reality navigation system according to claim 1 and a medical robot for assisting a practitioner in a surgical intervention on an anatomy of interest of a patient, the medical robot comprising:
a mobile base,
an articulated arm of which one end is fixed to the mobile base and the other end has a tool guide intended to hold a medical instrument, and
a control unit configured to determine a configuration of the articulated arm that allows a surgical action to be performed with the medical instrument along a predetermined path, the configuration of the articulated arm being determined according to information relating to the position/orientation of the marker which information is transmitted by the navigation system.
12. The assembly according to claim 11 , wherein the control unit is further configured to:
determine, from the predictive model, a second instant (t 2 ) corresponding to a candidate instant of insertion of a medical instrument into the anatomy of interest of the patient,
compare the position/orientation of the marker at the first instant (t 1 ) with the position/orientation of the marker at the second instant (t 2 ), and
display a result of the comparison on the display device in the form of augmented-reality content overlaid on the real images, wherein the information relating to the position/orientation of the marker is an indication that the difference between the position/orientation of the marker at the first instant (t 1 ) and the position/orientation of the marker at the second instant (t 2 ) is below a predetermined threshold.
13. The assembly according to claim 11 , wherein the control unit is further configured to receive the pre-intervention medical image from the medical imaging device and to generate, from said pre-intervention medical image, a three-dimensional anatomical model of the anatomy of interest of the patient and to display said anatomical model on the display device in the form of augmented-reality content overlaid on the real images, the position/orientation of the anatomical model overlaid on the real images being updated continuously and in real time according to the predictive model of the movement of the marker and according to a biomechanical model of anatomical structures of the human body, wherein the information relating to the position/orientation of the marker corresponds to a predictive model indicative of the position/orientation of the anatomical model during the respiratory cycles of the patient, and the control unit of the medical robot is configured to adjust, continuously and in real time, the configuration of the articulated arm according to said predictive model indicative of the position/orientation of the anatomical model.
14. The assembly according to claim 11 , wherein the control unit of the augmented-reality navigation system is configured to:
model an outer envelope of the body of the patient,
determine the position/orientation of the articulated arm or of the tool guide of the medical robot, and
detect a situation in which there is a risk of collision when the distance between the articulated arm or the tool guide of the medical robot and the outer envelope of the patient is below a predetermined threshold.
15. The assembly according to claim 11 , wherein the control unit of the augmented-reality navigation system is configured to detect a situation in which there is a risk of injury from the medical instrument when a deviation in the position/orientation of the marker with respect to the predictive model of the movement of the marker is above a predetermined threshold.
16. The assembly according to claim 15 , wherein the tool guide of the medical robot comprises an actuator allowing the medical instrument to be released instantly, said actuator being commanded by the control unit of the medical robot, and the control unit of the augmented-reality navigation system is configured to transmit to the control unit of the medical robot a command to instantly release the medical instrument when the risk-of-injury situation is detected.
17. The assembly according to claim 11 , wherein the articulated arm has at least six degrees of freedom, so that several different candidate configurations of the articulated arm allow the surgical action to be performed with the medical instrument along the predetermined path, and the control unit of the augmented-reality navigation system is configured to display, on the display device, in the form of augmented-reality content overlaid on the real images, said candidate configurations and to receive an indication relating to a selection, by the practitioner of one particular configuration from among the various candidate configurations.
18. The assembly according to claim 11 , wherein, when the medical instrument is inserted into the body of the patient along the predetermined path, the control unit of the augmented-reality navigation system is configured to display, on the display device, in the form of augmented-reality content overlaid on the real images, a part of the medical instrument which part is inserted into the body of the patient.
19. The assembly according to claim 18 , wherein the control unit of the augmented reality navigation system is configured to calculate and display on the display device, in the form of augmented-reality content overlaid on the real images, the distance between the medical instrument and a target point on the anatomy of interest and/or to detect an instant at which the medical instrument has reached said target point.
20. An augmented-reality navigation system for assisting a practitioner when performing a surgical intervention on an anatomy of interest of a patient, said system comprising:
a camera for acquiring real images, said camera being intended to be worn by the practitioner on their head,
a display device for displaying, in real-time, the real images acquired by the camera together with augmented-reality content overlaid on said real images, and
a control unit of the augmented-reality navigation system connected to the camera and to the display device, wherein said control unit of the augmented-reality navigation system is configured to:
detect, on the real images, the position/orientation of a marker placed on the patient in the proximity of said anatomy of interest,
record the movement followed by the marker during a recording period corresponding to one or more respiratory cycles of the patient,
determine the position/orientation of the marker at a first instant (t 1 ) belonging to said recording period and corresponding to an instant of acquisition of a pre-intervention medical image of the anatomy of interest of the patient by a medical imaging device,
formulate a predictive model of the movement of the marker from the movement followed by the marker during the recording period,
determine, from the predictive model, a second instant (t 2 ) corresponding to a candidate instant of insertion of a medical instrument into the anatomy of interest of the patient,
compare the position/orientation of the marker at the first instant (t 1 ) with the position/orientation of the marker at the second instant (t 2 ), and
display a result of the comparison on the display device in the form of augmented-reality content overlaid on the real images, wherein the information relating to the position/orientation of the marker is an indication that the difference between the position/orientation of the marker at the first instant (t 1 ) and the position/orientation of the marker at the second instant (t 2 ) is below a predetermined threshold.