Computer assisted surgery navigation using intra-operative tactile sensing feedback through machine learning system
A surgery navigation system is disclosed for computer assisted navigation during surgery. The surgery navigation system includes processing circuitry that is operative to obtain from a sensor circuit, intra-operative tactile sensing data indicating a sensed characteristic of a tool contacting a location on patient anatomy within a surgical site. The processing circuitry is further operative to process the intra-operative tactile sensing data through a machine learning model to generate intra-operative navigated guidance data, and to provide the intra-operative navigated guidance data to a display device and/or to a surgical robot to control movement of the tool.
1 . A surgery navigation system for computer assisted navigation during surgery, the surgery navigation system comprising:
a processing circuitry;
a tool for performing a surgery and having a sensor circuit positioned proximate to a working potion of the tool for sensing tactile sensing data;
a machine learning model that has been trained with collected intra-operative tactile sensing data;
the processing circuitry configured to:
obtain from the sensor circuit, intra-operative tactile sensing data indicating a sensed characteristic of the tool contacting a location on patient anatomy within a surgical site;
process the intra-operative tactile sensing data through the trained machine learning model to generate intra-operative navigated guidance data including a level of force between the tool and the location on the patient anatomy; and
provide the intra-operative navigated guidance data to a display device and/or to a surgical robot to control movement of the tool,
wherein the intra-operative navigated guidance data controls movement of the tool through the surgical robot.
2 . The surgery navigation system of claim 1 , wherein the sensor circuit comprises a force sensor and/or torque sensor, and wherein operation of the processing circuitry to process the intra-operative tactile sensing data through the machine learning model to generate intra-operative navigated guidance data, comprises to:
determine from output of the force sensor and/or torque sensor, a level of force between the tool and the location on the patient anatomy and/or a level of torque between the tool and the location on the patient anatomy; and
process the level of force and/or the level of torque through the machine learning model to generate the intra-operative navigated guidance data to characterize progress of the tool through soft tissue and/or bone of the patient anatomy.
3 . The surgery navigation system of claim 1 , wherein the sensor circuit comprises a vibration sensor, wherein operation of the processing circuitry to process the intra-operative tactile sensing data through the machine learning model to generate intra-operative navigated guidance data, comprises to:
determine from output of the vibration sensor, a vibration characteristic of the tool working at the location on the patient anatomy; and
process the vibration characteristic through the machine learning model to generate the intra-operative navigated guidance data to characterize progress of the tool through soft tissue and/or bone of the patient anatomy.
4 . The surgery navigation system of claim 1 , wherein the sensor circuit comprises a sound sensor, wherein operation of the processing circuitry to process the intra-operative tactile sensing data through the machine learning model to generate intra-operative navigated guidance data, comprises to:
determine from output of the sound sensor, a sound characteristic of the tool working at the location on the patient anatomy; and
process the sound characteristic through the machine learning model to generate the intra-operative navigated guidance data to characterize progress of the tool through soft tissue and/or bone of the patient anatomy.
5 . The surgery navigation system of claim 1 , wherein the sensor circuit comprises an electrical sensor configured to sense electrical charge transfer and/or impedance between the tool and the location on the patient anatomy, wherein operation of the processing circuitry to process the intra-operative tactile sensing data through the machine learning model to generate intra-operative navigated guidance data, comprises to:
determine from output of the electrical sensor, electrical charge transfer characteristics and/or impedance characteristics between the tool and the location on the patient anatomy; and
process the electrical charge transfer characteristics and/or impedance characteristics through the machine learning model to generate the intra-operative navigated guidance data to characterize progress of the tool through soft tissue and/or bone of the patient anatomy.
6 . The surgery navigation system of claim 1 , wherein the processing circuitry is further operative to:
obtain tool tracking data indicating pose of the tool relative to the patient anatomy; and
process the tool tracking data and the intra-operative tactile sensing data through the machine learning model to generate the intra-operative navigated guidance data.
7 . The surgery navigation system of claim 6 , wherein the processing circuitry is further operative to:
obtain pre-operative patient data and a surgical plan defining location of a region of the patient anatomy indicated in the pre-operative patient data that is to be removed; and
process the surgical plan, the tool tracking data, and the pre-operative patient data through the machine learning model to generate the intra-operative navigated guidance data to include plan progress information characterizing what portion of the region of the patient anatomy has been removed according to the surgical plan.
8 . The surgery navigation system of claim 7 , wherein the processing circuitry is further operative to:
control movement of the tool by a robot arm of the surgical robot based on the plan progress information to direct the tool toward a remaining portion of the region of the patient anatomy to be removed according to the surgical plan.
9 . The surgery navigation system of claim 7 , wherein the processing circuitry is further operative to:
control a display device based on the plan progress information to display a graphical indication of a remaining portion of the region of the patient anatomy to be removed according to the surgical plan.
10 . The surgery navigation system of claim 1 , further comprising:
an extended reality (XR) headset including at least one see-through display device,
wherein the processing circuitry is operative to control the see-through display device to display information to guide operator movement of the tool based on the intra-operative navigated guidance data.
11 . The surgical navigation system of claim 1 , further comprising:
a vibrator device connected to a probe and operative to vibrationally excite a tip of the probe;
a vibration sensor connected to output a vibration signal indicating a vibration characteristic of the tip of the probe contacting the location on the patient anatomy while the tip is vibrationally excited by the vibrator device;
wherein the processing circuitry is further operative to process the vibration characteristic indicated by the vibration signal through the machine learning model to generate the intra-operative navigated guidance data to indicate locations where the probe is contacting bone versus soft tissue of the patient anatomy.
12 . The surgical navigation system of claim 11 , wherein the processing circuitry is further operative to:
determine from the intra-operative navigated guidance data, completed locations within an intradiscal space between interior and superior vertebral bodies where anatomical material has been removed according to a surgical plan; and
control movement of the tool by an arm of the surgical robot based on the completed locations to direct the tool toward a remaining portion of the anatomical material to be removed according to the surgical plan.
13 . The surgical navigation system of claim 11 , wherein the processing circuitry is further operative to:
determine from the intra-operative navigated guidance data, completed locations within an intradiscal space between interior and superior vertebral bodies where anatomical material has been removed according to a surgical plan; and
control a display device based on the completed locations and the surgical plan, to display a graphical indication of where anatomical material has been removed from the intradiscal space and where anatomical material remains to be removed from the intradiscal space according to the surgical plan.
14 . The surgery navigation system of claim 1 , wherein the processing circuitry is further operative to:
obtain pre-operative patient data and a surgical plan defining location of a region of the patient anatomy indicated in the pre-operative patient data that is to be removed;
obtain tool tracking data indicating pose of the tool relative to the patient anatomy;
process the intra-operative tactile sensing data, the tool tracking data, the surgical plan, and the pre-operative patient data through the machine learning model to generate the intra-operative navigated guidance data;
obtain patient post-operative feedback data characterizing deviation between the surgical plan and an intra-operative surgery process which was performed on the patient responsive to the intra-operative navigated guidance data; and
train the machine learning model based on the patient post-operative feedback data.
15 . The surgery navigation system of claim 1 , wherein the processing circuitry is further operative to:
obtain patient post-operative feedback data characterizing at least one of
post-operative measurements of disc material removed from the spine of the patient,
post-operative measurements of spinal fusion rate of the patient,
post-operative measurements of spine decompression of the patient, and
post-operative measurements of spinal deformation of the patient; and
train the machine learning model based on the patient post-operative feedback data.
16 . The surgery navigation system of claim 1 ,
wherein the surgery navigation system comprises a plurality of the sensor circuits, and at least two of the sensor circuits are different ones of: a force sensor operative to output an indication of level of force between the tool and the location on the patient anatomy; a torque sensor operative to output an indication of level of torque between the tool and the location on the patient anatomy; vibration sensor operative to output a vibration characteristic of the tool working at the location on the patient anatomy; sound sensor operative to output a sound characteristic of the tool working at the location on the patient anatomy; and an electrical sensor operative to output an electrical charge transfer characteristic and/or an impedance characteristic between the tool and the location on the patient anatomy; and
wherein the machine learning model comprises:
a neural network component including an input layer having input nodes, a sequence of hidden layers each having a plurality of combining nodes, and an output layer having output nodes;
at least one processing circuit configured to provide outputs of the sensor circuits to different ones of the input nodes of the neural network model, and to generate the intra-operative navigated guidance data based on output of output nodes of the neural network component; and
a feedback training component configured to adapt weights and/or firing thresholds used by the combining nodes of the neural network component based on patient post-operative feedback data.
17 . The surgery navigation system of claim 1 , further comprising:
a tracking sub-system operative to determine pose of the tool relative to pose of the patient anatomy in a defined coordinate system,
wherein the processing circuitry is further operative to obtain a surgery plan, determine a target pose of the tool based on the surgery plan indicating where a surgical procedure is to be performed on the patient anatomy, and generate the intra-operative navigated guidance data based on comparison of the determined pose of the tool and the target pose of the tool, and based on output of the machine learning model from processing the intra-operative tactile sensing data.
18 . The surgery navigation system of claim 1 , further comprising:
the surgical robot including
a robot base,
a robot arm connected to the robot base and configured to guide movement of the tool, and
at least one motor operatively connected to control movement of the robot arm relative to the robot base,
wherein the processing circuitry is further operative to
obtain pre-operative patient data and a surgical plan defining a surgical procedure to be performed on the patient anatomy indicated in the pre-operative patient data;
obtain tool tracking data indicating pose of the tool relative to the patient anatomy;
process the tool tracking data, the surgical plan, and the pre-operative patient data through the machine learning model to generate the intra-operative navigated guidance data;
control movement of the at least one motor based on the intra-operative navigated guidance data to guide movement of the tool.
19 . A method by a surgery navigation system for computer assisted navigation during surgery, the method comprising:
training a machine learning model with intra-operative tactile sensing data that have been collected from previous surgeries;
obtaining from a sensor circuit positioned proximate to a working potion of a tool, intra-operative tactile sensing data indicating a sensed characteristic of the tool contacting a location on patient anatomy within a surgical site;
processing the intra-operative tactile sensing data through the trained machine learning model to generate intra-operative navigated guidance data including a level of force between the tool and the location on the patient anatomy; and
providing the intra-operative navigated guidance data to a display device and/or to a surgical robot controlling movement of the tool,
wherein the intra-operative navigated guidance data controls movement of the tool through the surgical robot.
20 . A computer program product comprising:
a non-transitory computer readable medium storing instructions executable by processing circuitry of a surgery navigation system for computer assisted navigation during surgery, the processing circuitry when executing the instructions is operative to:
obtain from a sensor circuit, intra-operative tactile sensing data indicating a sensed characteristic of a tool contacting a location on patient anatomy within a surgical site;
process the intra-operative tactile sensing data through a machine learning model that has been trained with intra-operative tactile sensing data that have been collected from previous surgeries to generate intra-operative navigated guidance data including a level of force between the tool and the location on the patient anatomy; and
provide the intra-operative navigated guidance data to a display device and/or to a surgical robot to control movement of the tool,
wherein the intra-operative navigated guidance data controls movement of the tool through the surgical robot, and
wherein the sensor circuit if positioned proximate to a working potion of the tool.