TOOL POSITION AND IDENTIFICATION INDICATOR DISPLAYED IN A BOUNDARY AREA OF A COMPUTER DISPLAY SCREEN
An endoscope captures images of a surgical site for display in a viewing area of a monitor. When a tool is outside the viewing area, a GUI indicates the position of the tool by positioning a symbol in a boundary area around the viewing area so as to indicate the tool position. The distance of the out-of-view tool from the viewing area may be indicated by the size, color, brightness, or blinking or oscillation frequency of the symbol. A distance number may also be displayed on the symbol. The orientation of the shaft or end effector of the tool may be indicated by an orientation indicator superimposed over the symbol, or by the orientation of the symbol itself. When the tool is inside the viewing area, but occluded by an object, the GUI superimposes a ghost tool at its current position and orientation over the occluding object.
1 - 27 . (canceled)
28 . An apparatus comprising:
a memory device non-transitorily storing program instructions; and
a processor coupled to a display device, an image capture device, and the memory device, the processor configured to execute the program instructions to:
determine a position of a tool in a reference frame of the image capture device;
determine a position to display a non-depictive symbol, which is non-depictive of the tool, in a boundary area circumscribing a viewing area on the display device, so as to indicate the position of the tool, while the viewing area is displaying an image captured by the image capture device; and
cause the non-depictive symbol to be displayed at the determined position in the boundary area.
29 . The apparatus according to claim 28 ,
wherein the processor is configured to execute the program instructions to:
cause the non-depictive symbol to be displayed at the determined position in the boundary area so that the non-depictive symbol provides information identifying one of: the tool, and a manipulator used for moving the tool.
30 . (canceled)
31 . The apparatus according to claim 29 ,
wherein the processor is configured to execute the program instructions to:
cause the non-depictive symbol to be displayed at the determined position in the boundary area so that the non-depictive symbol is marked with one of: a color that is uniquely associated with the manipulator, and a number that is uniquely associated with the manipulator.
32 . (canceled)
33 . (canceled)
34 . The apparatus according to claim 29 ,
wherein the processor is configured to execute the program instructions to:
determine the position of the tool by using at least one of:
kinematics for the manipulator, and an image identification technique.
35 . (canceled)
36 . The apparatus according to claim 28 ,
wherein the processor is configured to execute the program instructions to:
determine the position to display the non-depictive symbol in the boundary area by determining where a line passing through an axis, which extends along a length of a shaft of the tool, enters a field of view of the image capture device.
37 . (canceled)
38 . The apparatus according to claim 37 ,
wherein the processor is configured to execute the program instructions to:
determine the position of the non-depictive symbol in the boundary area by determining where a line extending from the position of the tool to a reference point, which is in the field of view of the image capture device, enters the field of view.
39 . The apparatus according to claim 28 ,
wherein the processor is configured to execute the program instructions to:
determine the position of the non-depictive symbol in the boundary area by: determining a trajectory of the tool from current and past positions of the tool, and determining where an extrapolation of the trajectory enters the field of view of the image capture device.
40 . The apparatus according to claim 28 ,
wherein the processor is configured to execute the program instructions to:
cause the non-depictive symbol to be displayed at the determined position in the boundary area by: determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the image capture device, and causing the non-depictive symbol to be displayed so that its size indicates the distance.
41 . The apparatus according to claim 28 ,
wherein the processor is configured to execute the program instructions to:
cause the non-depictive symbol to be displayed at the determined position in the boundary area by: determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the camera, and causing the non-depictive symbol to be displayed so that its color indicates the distance.
42 . The apparatus according to claim 41 ,
wherein the processor is configured to execute the program instructions to:
cause the non-depictive symbol to be displayed in the boundary area so that the intensity of the color of the non-depictive symbol indicates the distance.
43 . The apparatus according to claim 41 ,
wherein the processor is configured to execute the program instructions to:
cause the non-depictive symbol to be displayed in the boundary area so that the color of the non-depictive symbol relative to a color spectrum indicates the distance.
44 . The apparatus according to claim 28 ,
wherein the processor is configured to execute the program instructions to:
cause the non-depictive symbol to be displayed at the determined position in the boundary area by: determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the camera, and causing the non-depictive symbol to be displayed so that a frequency of blinking of the non-depictive symbol indicates the distance.
45 . The apparatus according to claim 28 ,
wherein the processor is configured to execute the program instructions to:
cause the non-depictive symbol to be displayed at the determined position in the boundary area by: determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the camera, and causing the non-depictive symbol to be displayed so that a frequency of oscillation of the non-depictive symbol about the determined position in the boundary area indicates the distance.
46 . The apparatus according to claim 28 ,
wherein the processor is configured to execute the program instructions to:
cause the non-depictive symbol to be displayed at the determined position in the boundary area by: determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the camera, and causing the non-depictive symbol to be displayed so that the distance is indicated by overlaying a distance number over the non-depictive symbol.
47 . The apparatus according to claim 28 ,
wherein the processor is configured to execute the program instructions to:
determine the position of the tool by determining a position and an orientation of an end effector of the tool, and
cause the non-depictive symbol to be displayed at the determined position in the boundary area by causing an orientation indicator to be displayed over the non-depictive symbol such that the orientation indicator is oriented so as to indicate the orientation of the end effector.
48 . The apparatus according to claim 28 ,
wherein the processor is configured to execute the program instructions to:
determine an orientation of an axis that extends along a length of a shaft of the tool, and
cause the non-depictive symbol to be displayed at the determined position in the boundary area by causing an orientation indicator to be displayed over the non-depictive symbol such that the orientation indicator is oriented so as to indicate the orientation of the axis.
49 . (canceled)
50 . (canceled)
51 . An apparatus comprising:
a memory device non-transitorily storing program instructions; and
a processor is configured to execute the program instructions to:
cause images captured by an image capture device to be displayed in a viewing area on a display device;
determine a position of a tool, which is within, but occluded in a field of view of the image capture device, by determining a position of the tool in a reference frame of the image capture device by using kinematics of manipulator used for moving the tool; and
cause a ghost tool to be displayed in the viewing area where the tool is occluded by using the determined position of the tool in the reference frame of the image capture device.
52 . The apparatus according to claim 51 ,
wherein the viewing area includes stereoscopic right and left two-dimensional views, and
wherein the processor is configured to further execute the program instructions to:
determine that the tool is occluded in the field of view of the image capture device by:
generating a three-dimensional computer model of the tool;
positioning and orientating the computer model to coincide with the current position and orientation of the tool in the reference frame of the computer display screen;
generating a two-dimensional outline of the computer model that projects onto a selected one of the right and left two-dimensional views; and
cross-correlating the two-dimensional outline of the computer model with the selected one of the right and left two-dimensional views.
53 . The apparatus according to claim 52 ,
wherein the processor is configured to execute the program instructions to:
cause the ghost tool to be displayed in the viewing area by causing at least the two-dimensional outline of the computer model of the tool to be displayed as an overlay on the selected one of the right and left two-dimensional views as it is being displayed on the computer screen.
54 . The apparatus according to claim 52 ,
wherein the processor is configured to execute the program instructions to:
cause the ghost tool to be displayed in the viewing area by causing right and left two-dimensional ghost images of the tool to be overlaid in corresponding of the right and left two-dimensional views, so as to appear as a three-dimensional ghost image of the tool at the position and orientation of the tool in the reference frame of the computer display screen.
55 - 76 . (canceled)