Use of eye tracking for tool identification and assignment in a robotic surgical system
A robotic surgical system includes an eye gaze sensing system in conjunction with a visual display of a camera image from a surgical work site. Detected gaze of a surgeon towards the display is used as input to the system. This input may be used by the system to assign an instrument to a control input device (when the user is prompted to look at the instrument), or it may be used as input to a computer vision algorithm to aid in object differentiation and seeding information, facilitating identification/differentiation of instruments, anatomical features or regions.
1. A robotic surgical system, comprising:
at least two surgical instruments, each instrument moveable by a robotic manipulator within a work area;
a first hand controller and a second hand controller;
a camera positioned to capture an image of a portion of the work area;
an image display for displaying the image;
an eye gaze sensor positionable to detect a direction of the user's gaze towards the image of the work area on the display, and
a processor configured to
prompt the user to direct the user's gaze towards a portion of the displayed image at which the one of said at least two surgical instruments the user wants to assign to the first hand controller is located,
determine, based on the direction detected by the eye gaze sensor, which of the at least two instruments displayed on the image display the user is gazing towards, and
assign the first hand controller to the determined instrument.
2. The system of claim 1 , wherein the processor is configured to receive input from the first hand controller and to control the robotic manipulator to move the instrument in response to the input.
3. The system according to claim 1 , wherein the processor is configured to determine, based on the direction detected by the eye gaze sensor, which of the at least two instruments the user is viewing on the image display, and to assign the first hand controller to an instrument identified to the system using detected eye gaze.
4. The system according to claim 1 , wherein the processor is configured to determine, based on the direction detected by the eye gaze sensor, the region of the image display of the work site that is occupied by the instrument the user wants to assign to the first hand controller.
5. A method of using a robotic surgical system, comprising:
providing a first hand controller and a second hand controller;
positioning at least two surgical instruments, within a work area, each instrument moveable by a robotic manipulator;
capturing an image of the surgical instruments within the work area;
displaying the image on an image display;
prompting a user to direct the user's gaze towards a portion of the displayed image at which the one of said at least two surgical instruments the user wants to assign to the first hand controller is located;
receiving from an eye gaze sensor input representing a direction of a user's gaze towards the image;
determining, using the eye gaze sensor input, which of the at least two instruments displayed on the image display the user is gazing towards; and
assigning the first hand controller to the determined instrument to enable control, using input from the first hand controller, of the robotic manipulator operatively associated with the determined instrument.
6. The method of claim 5 , wherein the method further includes the step of receiving input from the first hand controller and controlling the robotic manipulator to move the determined instrument in response to the input.
7. The method according to claim 5 , wherein the determining step includes determining, based on the direction detected by the eye gaze sensor, which of the at least two instruments the user is viewing on the image display, and wherein the assigning step includes assigning the first hand controller to an instrument identified to the system using detected eye gaze.
8. The method according to claim 5 , wherein the determining step includes determining, based on the direction detected by the eye gaze sensor, the region of the image display of the work site that is occupied by the instrument the user wants to assign to the first hand controller.