Tool position and identification indicator displayed in a boundary area of a computer display screen
An endoscope captures images of a surgical site for display in a viewing area of a monitor. When a tool is outside the viewing area, a GUI indicates the position of the tool by positioning a symbol in a boundary area around the viewing area so as to indicate the tool position. The distance of the out-of-view tool from the viewing area may be indicated by the size, color, brightness, or blinking or oscillation frequency of the symbol. A distance number may also be displayed on the symbol. The orientation of the shaft or end effector of the tool may be indicated by an orientation indicator superimposed over the symbol, or by the orientation of the symbol itself. When the tool is inside the viewing area, but occluded by an object, the GUI superimposes a ghost tool at its current position and orientation over the occluding object.
1. A method for indicating a position of a tool at a work site when the tool is entirely out of a field of view of an imaging device, the method comprising:
a processor causing images captured by the imaging device to be displayed in a viewing area on a display screen;
the processor determining a position of the tool in a reference frame of the imaging device, wherein the tool has a shaft and an end effector including opposing members coupled to the shaft;
the processor determining a position to display a non-depictive symbol for the tool in a boundary area circumscribing the viewing area on the display screen so as to indicate a direction of the determined position of the tool relative to the field of view of the imaging device, by determining a trajectory of the tool from current and past positions of the tool, and determining where an extrapolation of the trajectory enters the field of view of the imaging device; wherein the non-depictive symbol for the tool does not resemble the end effector of the tool in appearance; and wherein the position of the non-depictive symbol for the tool in the boundary area does not indicate a distance of the tool from the field of view of the imaging device; and
the processor causing the non-depictive symbol to be displayed at the determined position in the boundary area while images that were captured by the imaging device are restricted to being displayed in the viewing area on the display screen.
2. The method according to claim 1 , wherein the non-depictive symbol for the tool provides information identifying the tool.
3. The method according to claim 1 , wherein the non-depictive symbol for the tool provides information identifying a manipulator coupled to the tool.
4. The method according to claim 3 , wherein the non-depictive symbol for the tool is a color that is associated with and indicated on the manipulator.
5. The method according to claim 3 , wherein the non-depictive symbol for the tool is marked with a number that is associated with and indicated on the manipulator.
6. The method according to claim 1 , wherein the processor determines the position of the tool by using kinematics for a manipulator coupled to the tool.
7. The method according to claim 1 , wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises: determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the imaging device, and causing the non-depictive symbol for the tool to be displayed so that a size of the non-depictive symbol for the tool indicates the distance.
8. The method according to claim 1 , wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises: determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the imaging device, and causing the non-depictive symbol for the tool to be displayed so that a color of the non-depictive symbol for the tool indicates the distance.
9. The method according to claim 1 , wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises: determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the imaging device, and causing the non-depictive symbol for the tool to be displayed so that an intensity of a color of the non-depictive symbol for the tool indicates the distance.
10. The method according to claim 1 , wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises: determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the imaging device, and causing the non-depictive symbol for the tool to be displayed so that a color of the non-depictive symbol for the tool indicates the distance by a position of the color in a color spectrum.
11. The method according to claim 1 , wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises: determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the imaging device, and causing the non-depictive symbol for the tool to be displayed so that a frequency of blinking of the non-depictive symbol for the tool indicates the distance.
12. The method according to claim 1 , wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises: determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the imaging device, and causing the non-depictive symbol for the tool to be displayed so that a frequency of oscillation of the non-depictive symbol for the tool about the determined position in the boundary area indicates the distance.
13. The method according to claim 1 , wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises: determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the imaging device, and causing the non-depictive symbol for the tool to be displayed so that the distance is indicated by overlaying a distance number over the non-depictive symbol for the tool.
14. The method according to claim 1 ,
wherein the processor determining the position of the tool further comprises determining an orientation of the end effector in the reference frame of the imaging device, and
wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises: causing an orientation indicator to be displayed over the non-depictive symbol for the tool such that the orientation indicator is oriented so as to indicate the orientation of the end effector.
15. The method according to claim 1 ,
wherein the tool includes a shaft having an axis extending along a length of the shaft,
wherein the processor determining the position of the tool further comprises determining an orientation of the axis in the reference frame of the imaging device, and
wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises: causing an orientation indicator to be displayed over the non-depictive symbol for the tool such that the orientation indicator is oriented so as to indicate the orientation of the axis.