Trocar pose estimation using machine learning for docking surgical robotic arm to trocar
A surgical robotic system senses position or orientation of an object, which may be a trocar that has a magnetic field. Magnetic field sensors are coupled to a surgical robotic arm. A machine learning model coupled to the magnetic field sensors is trained to output three-dimensional position and/or three-dimensional orientation of the trocar or other object. Other aspects are also described.
1. A surgical robotic system for sensing pose, the system comprising:
a surgical robotic arm;
a plurality of magnetic field sensors coupled to the surgical robotic arm; and
a machine learning model coupled to the plurality of magnetic field sensors and trained to output an estimate of a pose of a trocar that is producing a magnetic field, based on output data from the plurality of magnetic field sensors.
2. The surgical robotic system of claim 1 , wherein the machine learning model is a convolutional neural network configured for image processing of an input two-dimensional array of output data from the magnetic field sensors.
3. The surgical robotic system of claim 1 , further comprising the trocar having one or more discrete magnets attached thereto that produce the magnetic field.
4. The surgical robotic system of claim 1 wherein output data from the plurality of magnetic field sensors are arranged in a two-dimensional array as input to the machine learning model.
5. The surgical robotic system of claim 1 , further comprising:
a controller having a physical model that describes how the magnetic field sensors can sense a magnetic field produced by one or magnets, the controller configured to guide the surgical robotic arm to dock with the trocar based on i) a physical model estimate of the pose of the trocar produced using the physical model and ii) a machine learning model estimate of the pose of trocar produced using the machine learning model.
6. The surgical robotic system of claim 1 , further comprising:
a tool drive coupled to the surgical robotic arm, wherein the plurality of magnetic field sensors and a docking interface are located on a base of the tool drive, and wherein the docking interface is configured to receive an attachment portion of the trocar.
7. The surgical robotic system of claim 6 , wherein:
the plurality of magnetic field sensors are arranged in two or more physical arrays that are spaced apart from each other;
the machine learning model is arranged to receive input from the plurality of magnetic field sensors as a two-dimensional array of sensor measurements; and
wherein adjacent elements in the two-dimensional array that are input to the machine learning model are from nonadjacent magnetic field sensors in the two or more physical arrays.
8. The surgical robotic system of claim 1 , wherein:
the plurality of magnetic field sensors is arranged in two or more physical arrays;
the machine learning model is arranged to receive input from the plurality of magnetic field sensors as a two-dimensional array of sensor measurements; and
wherein adjacent elements in the two-dimensional array that are input to the machine learning model are from nonadjacent magnetic field sensors in the two or more physical arrays.
9. A surgical robotic system, comprising:
a surgical robotic arm;
a tool drive coupled to the arm and having a docking interface that comprises a plurality of magnetic field sensors; and
a programmed processor that implements i) a machine learning model which outputs three-dimensional position and three-dimensional orientation of a trocar as a machine learning model estimate of a pose of the trocar, based on input to the machine learning model from the plurality of magnetic field sensors, ii) a physical model of a magnetic field system wherein the magnetic field system comprises i) a magnetic field produced by a magnetic trocar and ii) the plurality of magnetic field sensors, wherein the physical model produces a physical model estimate of the pose of the trocar, and wherein the processor produces a final estimate of the pose of the trocar based on the machine learning model estimate and based on the physical model estimate, and iii) a robot control algorithm for guiding the surgical robotic arm based on the final estimate of the pose until the docking interface is ready to dock with the trocar.
10. The surgical robotic system of claim 9 , wherein the machine learning model is a convolutional neural network configured for propagating an input two-dimensional array of output data from the magnetic field sensors, and the neural network comprises:
an input layer arranged to receive the input two dimensional array of output data from the magnetic field sensors;
a plurality of convolutional layers;
one or more fully connected layers; and
a regression output layer to output the three-dimensional position or three-dimensional orientation.
11. The surgical robotic system of claim 9 , wherein the machine learning model is a convolutional neural network arranged to receive the input as a two-dimensional array of output data from magnetic field sensors, and wherein output data of adjacent magnetic field sensors, of the plurality of magnetic field sensors, are arranged as nonadjacent elements in the two-dimensional array of output data.
12. The surgical robotic system of claim 9 , further comprising:
a controller coupled to the plurality of magnetic field sensors and the surgical robotic arm, the controller configured to use both a physical model and the machine learning model for redundant estimation of the three-dimensional position or three-dimensional orientation.
13. A method for sensing position or orientation of a trocar by a surgical robotic system, the method comprising:
sensing, with a plurality of magnetic field sensors attached to a surgical robotic arm of the surgical robotic system, a magnetic field produced by the trocar;
providing as an input of a machine learning model sensed data from the plurality of magnetic field sensors, and wherein the machine learning model is configured to output a first estimate of three-dimensional position or three-dimensional orientation of the trocar; and
guiding by the surgical robotic system the surgical robotic arm to the trocar, based on the first estimate of three-dimensional position or three-dimensional orientation of the trocar as output by the machine learning model.
14. The method of claim 13 further comprising:
signaling by the surgical robotic system that the surgical robotic arm is ready to dock with the trocar, in response to determining that a pose of a docking interface attached to the arm matches an estimated three-dimensional position and an estimated three-dimensional orientation of the trocar as output by the machine learning model.
15. The method of claim 13 , further comprising:
re-training the machine learning model, responsive to determining that the magnetic field produced by the trocar has changed.
16. The method of claim 13 , further comprising:
producing a second estimate of three-dimensional position or three-dimensional orientation of the trocar using a physical model whose input is the sensed data from the plurality of magnetic field sensors, wherein the sensed data from the magnetic field sensors is also used by the machine learning model to output the first estimate; and
determining whether the second estimate is at fault, based on the first estimate.
17. The method of claim 16 further comprising:
producing a final estimate of three-dimensional position or three-dimensional orientation of the trocar responsive to determining whether the second estimate is at fault, wherein the guiding the surgical robotic arm is based on the final estimate.
18. The method of claim 13 , further comprising:
arranging the sensed data from adjacent ones of the plurality of magnetic field sensors as non-adjacent elements of a two dimensional array that is input to the machine learning model.