Systems and methods for estimating needle pose

1 . A system for tissue suturing guidance in a surgical site, the system comprising:

an imaging device configured to capture stereoscopic image data of a surgical needle within the surgical site; and

an imaging device control unit configured to control the imaging device, the imaging device control unit including:

a processor; and

a memory storing instructions which, when executed by the processor, cause the system to:

capture stereoscopic image data comprising an image of a surgical needle and a surgical tool within the surgical site via the imaging device;

estimate a pose of the surgical needle based on the captured stereoscopic image data;

generate an augmented image based on the estimated pose of the surgical needle; and

display on a display, the augmented image of the surgical needle.

2 . The system of claim 1 , wherein the instructions, when executed by the processor, further cause the system to:

estimate a pose of the surgical tool based on the captured stereoscopic image data; and

generate a position refinement signal based on the estimated pose of the surgical needle and the estimated pose of the surgical tool.

3 . The system of claim 2 , wherein the instructions, when executed by the processor, further cause the system to:

receive a first control signal for the surgical tool;

generate a second control signal based on the position refinement signal and the first control signal; and

adjust a trajectory of the surgical needle based on the second control signal.

4 . The system of claim 1 , wherein estimating the pose of the surgical needle, is based on a machine learning network.

5 . The system of claim 3 , wherein the instructions, when executed by the processor, cause the system to:

estimate a location of a tip of the surgical needle based on the captured stereoscopic image data.

6 . The system of claim 5 , wherein the instructions, when executed by the processor, further cause the system to:

generate a third control signal for adjusting a trajectory of the surgical needle based on the estimated pose of the tip of the surgical needle.

7 . The system of claim 5 , wherein the instructions, when executed by the processor, further cause the system to:

determine whether the tip of the surgical needle is touching tissue; and

further augment the augmented image based on the surgical needle touching the tissue.

8 . The system of claim 7 , wherein the instructions, when executed, further cause the system to:

generate a third control signal for controlling a robotic surgical system based on the determined touch of the tissue by the tip of the surgical needle.

9 . The system of claim 5 , wherein the augmented image includes at least one of highlighting the surgical needle or highlighting the tip of the surgical needle.

10 . The system of claim 1 , wherein the stereoscopic image data comprises a left image and a right image of the surgical needle and the surgical tool.

11 . A system for tissue suturing guidance in a surgical site, the system comprising:

an imaging device configured to capture stereoscopic image data of a surgical needle and a surgical tool within the surgical site, the stereoscopic image data comprising a left image and a right image; and

an imaging device control unit configured to control the imaging device, the imaging device control unit including:

a processor; and

a memory storing instructions which, when executed by the processor, cause the system to:

capture, via the imaging device, the stereoscopic image data of the surgical needle and the surgical tool within the surgical site;

process the left image and the right image to generate segmentations identifying a body portion and a tip portion of the surgical needle in each of the left image and the right image;

generate, based on the segmentations and calibration data of the imaging device, a three-dimensional model of the surgical needle by fusing the left and right segmentations using triangulation;

estimate a pose of the surgical needle based on the three-dimensional model;

generate an augmented image based on the estimated pose of the surgical needle, the augmented image including a prediction overlay representing at least a portion of the surgical needle derived from the three-dimensional model; and

display on a display, the augmented image of the surgical needle.

12 . The system of claim 11 , wherein generating the three-dimensional model of the surgical needle includes reconstructing a three-dimensional central line of the surgical needle by triangulating central line points based on stereo image pairs, including, for each point in a central line in one of the left image or the right image, triangulating with points located near an epipolar line in the other of the left image or the right image.

13 . The system of claim 11 , wherein estimating the pose of the surgical needle based on the three-dimensional model includes determining at least one of a yaw, a pitch, or a tilt of a central line of the surgical needle represented in the three-dimensional model.

14 . The system of claim 11 , wherein processing of the left image and the right image to generate segmentations is performed by a machine learning network comprising a deep neural network configured to perform pixel-wise segmentation of the body portion and the tip portion of the surgical needle in the left image and the right image.

15 . The system of claim 12 , wherein reconstructing the three-dimensional central line of the surgical needle includes fitting a curve to triangulated central line points to represent the central line of the surgical needle in three-dimensional space.

16 . The system of claim 11 , wherein the prediction overlay includes a graphical representation of a current pose of the surgical needle and a predicted trajectory of the surgical needle through tissue derived from the three-dimensional model.

17 . The system of claim 14 , wherein the memory further stores instructions which, when executed by the processor, cause the system to:

compute, from outputs of the machine learning network, a heatmap indicating a likelihood that the tip portion of the surgical needle is contacting tissue; and

modify the prediction overlay in the augmented image in response to the likelihood that the tip portion of the surgical needle is contacting tissue exceeding a threshold.

18 . The system of claim 11 , wherein the memory further stores instructions which, when executed by the processor, cause the system to generate a control signal for a robotic surgical system based on at least one of the estimated pose of the surgical needle or the predicted trajectory of the surgical needle.

19 . The system of claim 11 , wherein the display comprises at least one of a three-dimensional display device or a head-mounted display configured to present the augmented image with depth perception corresponding to the stereoscopic image data.

20 . The system of claim 11 , wherein the memory further stores instructions which, when executed by the processor, cause the system to compute a confidence measure associated with the three-dimensional model of the surgical needle, and to selectively generate the prediction overlay in the augmented image in response to the confidence measure satisfying a predetermined criterion.