Use of eye tracking input to initiate computer-vision identification of an object in a surgical image
A robotic surgical system includes an eye gaze sensing system in conjunction with a visual display of a camera image from a surgical work site. Detected gaze of a surgeon towards the display is used as input to the system. This input may be used by the system to assign an instrument to a control input device (when the user is prompted to look at the instrument), or it may be used as input to a computer vision algorithm to aid in object differentiation and seeding information, facilitating identification/differentiation of instruments, anatomical features or regions.
1. A method of detecting an object within a surgical work site, comprising:
positioning a surgical instrument within a surgical work area within a body cavity;
mounting the surgical instrument to a robotic manipulator;
causing the robotic manipulator to move the surgical instrument in response to input commands from a hand controller;
capturing an image of a portion of the surgical work area within the body cavity;
displaying the image on an image display;
receiving from an eye gaze sensor input corresponding to a direction of a user's gaze towards the image of the work area on the display, the direction of the user's gaze corresponding to a region of the image; and
executing a computer vision algorithm, said computer vision algorithm analyzing the region within the captured image to detect in the image at least a portion of the surgical instrument that is at the surgical site and displayed in the image; and
following detection of the surgical instrument, initiating a mode in which input commands from the hand controller do not cause movement of the surgical instrument by the robotic manipulator.
2. The method of claim 1 , further including the step of, prior to the receiving step, prompting the user to look at the object on the image display.
3. The method of claim 1 , further including the step of receiving a user instruction to recognize an object visible on the image display.
4. The method of claim 1 , wherein the method further includes, following detection of the surgical instrument, displaying a graphical overlay on a camera display showing an image of the surgical work area.
5. The method of claim 1 , wherein executing the computer vision algorithm includes performing image segmentation in the region within the captured image to detect at least a portion of the surgical instrument displayed in the image.
6. The method of claim 5 , wherein executing the computer vision algorithm includes performing image segmentation in the region within the captured image to detect at least an edge of the surgical instrument displayed in the image.
7. The method of claim 5 , wherein executing the computer vision algorithm includes performing image segmentation in the region by applying an active contour model in the region within the captured image to detect at least a portion of the surgical instrument displayed in the image.
8. The method of 5 , wherein executing the computer vision algorithm includes performing image segmentation in the region by applying a region growing algorithm in the region within the captured image to detect at least a portion of the surgical instrument displayed in the image.
9. The method of claim 1 , wherein the method includes positioning an endoscope in the body cavity and capturing the image using the endoscope.
10. A surgical system, comprising:
a surgical instrument positionable within a surgical work area within a body cavity;
a robotic manipulator, the surgical instrument removably mountable to the robotic manipulator, the robotic manipulator operable to move the surgical instrument in response to input commands from a hand controller;
a camera positionable in a body cavity to capture images of a surgical site within the body cavity;
an image display for displaying the images;
an eye gaze sensor positionable to detect a direction of the user's gaze towards the image of the surgical site on the display, and
a processor configured to
receive, from the eye gaze sensor, input corresponding to a direction of a user's gaze towards the image of the surgical site on the display, the direction of the user's gaze corresponding to a region of the surgical site; and
execute a computer vision algorithm, said computer vision algorithm analyzing the region within the captured image to detect in the image at least a portion of the surgical instrument that is at the surgical site and displayed in the image; and
following detection of the surgical instrument, initiating a clutching mode in which input commands from the hand controller do not cause movement of the surgical instrument by the robotic manipulator.
11. The surgical system of claim 10 , wherein the processor is further configured to receive a user instruction to recognize an object visible on the image display.
12. The method of claim 10 , wherein the processor is configured to perform image segmentation in the region within the captured image to detect at least a portion of the surgical instrument displayed in the image.
13. The method of claim 12 , wherein the processor is configured to perform image segmentation in the region within the captured image to detect at least an edge of the the surgical instrument displayed in the image.
14. The method of claim 12 , wherein the processor is configured to perform image segmentation in the region by applying an active contour model in the region within the captured image to detect at least a portion of the surgical instrument displayed in the image.
15. The method of 12 , wherein the processor is configured to perform image segmentation in the region by applying a region growing algorithm in the region within the captured image to detect at least a portion of the surgical instrument displayed in the image.