System and method, for training an interventionalist to perform an invasive percutaneous intervention or an endoscopic intervention
System for training an interventionalist to perform an invasive percutaneous or endoscopic intervention on an organ includes a pipe having a size and/or shape similar to a body vessel or tubular body cavity connected to the organ. An exit of the pipe simulates or represents an exit of the vessel or cavity at the organ. A tool is inserted at an entrance of the pipe and pushed through the pipe. A stereoscopic camera acquires images of an end portion of the tool as it exits from the pipe. A model generating unit generates a real-time 3D model of this end portion from the images. A merging unit merges in real time the real-time model and a pre-computed 3D model of the organ into a common environment displayed so that the interventionalist can see in real-time where the real-time model of the tool is located with respect to the pre-computed model.
1 . System for training an interventionalist to perform an invasive percutaneous intervention or an endoscopic intervention on an organ, by using a tool in the organ, comprises:
a pipe comprising an entrance and an exit and having a size and/or a shape equal to a body vessel or a tubular body cavity, the body vessel or the tubular body cavity being connected to the organ, wherein the exit of the pipe physically simulates or represents the exit of the vessel or of the tubular body cavity at a junction of the vessel or of the tubular body cavity with the organ;
said tool, arranged to be inserted by the interventionalist at the entrance of the pipe and to be pushed by the interventionalist through the pipe;
at least one stereoscopic camera arranged to acquire images of an end portion of the tool starting from a moment when said end portion starts emerging from the exit of the pipe;
a real-time 3D model generating unit, arranged for generating a real-time 3D model of said end portion of the tool from said images,
a merging unit, arranged for merging in real-time in a common environment said real-time 3D model and a pre-computed 3D model of at least a portion of the organ;
a display for receiving data from the merging unit in order to show to the interventionalist said common environment, so that the interventionalist can see in real-time on the display where the real-time 3D model of the tool is located with respect to the pre-computed 3D model of the portion of the organ, thus making the training of the interventionalist possible.
2 . System of claim 1 , the 3D model of the portion of the organ being a static 3D model.
3 . System of claim 1 , comprising a real-time 3D model generating module, being a machine learning-based module arranged to generate said static 3D model from images from a Magnetic Resonance Imaging scanner, a CT scanner, or a device able to generate volumetric images of organs.
4 . System of claim 1 , comprising a tool tracking module arranged to compute and/or track in real-time a position of the end portion of the tool with regard to the exit of said pipe.
5 . System of claim 1 , wherein said real-time 3D model generating unit is arranged to generate from the images taken by the stereoscopic camera a cloud of 3D points that denote the position of the end portion tool with regard to the exit of said pipe.
6 . System of claim 5 , wherein said tracking module is arranged to use said cloud of 3D points so as to output a predicted binary occupancy grid having a value of one at a location of the tool and a value of zero at all other locations.
7 . System of claim 5 , wherein said tool tracking module is arranged to use said cloud of 3D points so as to output coordinates of 3D points that define a 3D position of a centerline of said tool with regard to the exit of said pipe.
8 . System of claim 7 , wherein said tool tracking module comprises a unit arranged to use a latent representation at an output of an encoder so as to extract the 3D position of the 3D points.
9 . System of claim 8 , wherein said unit is a Multi-Layer-Perceptron.
10 . System of claim 8 , wherein said unit is a fully connected architecture.
11 . System of claim 1 , wherein said 3D model of the portion of the organ comprises at least one element characterizing a lesion to be operated.
12 . System of claim 1 , wherein said pipe comprises a gel or a liquid simulating at least a physical property of the physical property of a physical liquid contained in the body vessel or the tubular body cavity.
13 . System of claim 1 , wherein said 3D model of the portion of the organ as displayed by said display is augmented by at least one target designating at least one target area of the organ that has been treated and/or that is to be treated.
14 . System of claim 1 , wherein said merging unit, before merging in the common environment both the real-time 3D model and the 3D model of the portion of the organ, performs calibration step so as to align a position of an end of the pipe with a position of an entry portion of the 3D model of the portion of the organ.
15 . Method for training an interventionalist to an invasive percutaneous or endoscopic intervention of an organ, by using a tool in the organ comprises:
providing a pipe, said pipe comprising an entrance and an exit and having a size and/or a shape equal to a body vessel or to a tubular body cavity, the body vessel or the tubular body cavity being connected to the organ, wherein the exit of the pipe physically simulates or represents the exit of the vessel or of the tubular body cavity at a junction of the vessel or of the tubular body cavity with the organ;
inserting said tool by the interventionalist at the entrance of the pipe and pushing said tool by the interventionalist through the pipe;
acquiring by at least one stereoscopic camera images of an end portion of the tool starting from a moment when said end portion starts emerging from the exit of the pipe;
generating, by a real-time 3D model generating unit, a real-time 3D model of said end portion of the tool from said images,
merging, by a merging unit, in real-time in a common environment said real-time 3D model and a pre-computed 3D model of at least a portion of the organ;
displaying said common environment on a display arranged for receiving data from the merging unit, so that the interventionalist can see in real-time on said display where the real-time 3D model of the tool is located with respect to the pre-computed 3D model of the portion of the organ, thus making the training of the interventionalist possible.