Tool position and identification indicator displayed in a boundary area of a computer display screen
An endoscope captures images of a surgical site for display in a viewing area of a monitor. When a tool is outside the viewing area, a GUI indicates the position of the tool by positioning a symbol in a boundary area around the viewing area so as to indicate the tool position. The distance of the out-of-view tool from the viewing area may be indicated by the size, color, brightness, or blinking or oscillation frequency of the symbol. A distance number may also be displayed on the symbol. The orientation of the shaft or end effector of the tool may be indicated by an orientation indicator superimposed over the symbol, or by the orientation of the symbol itself. When the tool is inside the viewing area, but occluded by an object, the GUI superimposes a ghost tool at its current position and orientation over the occluding object.
1. A medical robotic system comprising:
a robotic arm mechanically coupleable to a tool so as to be capable of positioning and orienting the tool;
an imaging device;
a display screen; and
a processor configured to:
cause images captured by the imaging device to be displayed in a viewing area of the display screen;
determine a position of the tool in a reference frame of the imaging device;
determine a position to display a non-depictive symbol for the tool, in a boundary area circumscribing the viewing area of the display screen, so as to indicate a direction of the position of the tool relative to a field of view of the image capture device when the tool is entirely out of the field of view of the image capture device, by determining a trajectory of the tool from current and past positions of the tool, and determining where an extrapolation of the trajectory enters the field of view of the imaging device; wherein the non-depictive symbol for the tool does not resemble the tool in appearance from the perspective of the images being displayed in the viewing area; and wherein the position of the non-depictive symbol for the tool in the boundary area does not indicate a distance of the tool from the field of view of the image capture device; and
cause the non-depictive symbol for the tool to be displayed at the determined position in the boundary area while images that were captured by the image capture device are restricted to being displayed in the viewing area on the display device.
2. The system according to claim 1 ,
wherein the processor is further configured to:
cause a visual characteristic of the non-depictive symbol for the tool being displayed in the boundary area of the computer display screen to be altered so as to indicate a distance from a position of a reference point on the tool to a reference point in the field of view of the imaging device when the tool is outside of the field of view of the imaging device.
3. The system according to claim 2 ,
wherein the processor is configured to:
alter the visual characteristic of the non-depictive symbol for the tool in the boundary area so as to indicate the distance by at least one of: a size of the non-depictive symbol for the tool, a color of the non-depictive symbol for the tool, a brightness of the non-depictive symbol for the tool, and a frequency of blinking of the non-depictive symbol for the tool.
4. The system according to claim 3 ,
wherein the processor is configured to:
alter the visual characteristic of the non-depictive symbol for the tool in the boundary area so that the frequency of blinking increases as the distance decreases.
5. The system according to claim 2 ,
wherein the processor is configured to:
alter the visual characteristic of the non-depictive symbol for the tool in the boundary area so that a frequency of oscillation of the non-depictive symbol for the tool about the determined position in the boundary area indicates the distance.
6. The system according to claim 1 ,
wherein the processor is configured to:
cause a distance indicator to be superimposed over the non-depictive symbol for the tool being displayed in the boundary area so that the distance is indicated by a distance number on the distance indicator.
7. The system according to claim 1 ,
wherein the processor is further configured to:
cause an orientation indicator to be superimposed over the non-depictive symbol for the tool in the boundary area of the computer display screen so as to indicate an orientation of an end effector of the tool when the tool is outside of the field of view of the imaging device.
8. The system according to claim 1 ,
wherein the processor is configured to:
cause the non-depictive symbol for the tool to be displayed at the determined position in the boundary area so that the non-depictive symbol for the tool is marked with one of: a color that is uniquely associated with the robotic arm, and a number that is uniquely associated with the robotic arm.
9. The system according to claim 1 ,
wherein the processor is configured to:
cause the non-depictive symbol for the tool to be displayed at the determined position in the boundary area so that the non-depictive symbol for the tool provides information identifying one of: the tool, and the robotic arm.
10. The system according to claim 1 ,
wherein the processor is configured to:
determine the position of the tool by using kinematics for the robotic arm.
11. The system according to claim 1 , wherein the imaging device includes at least one camera.
12. The system according to claim 1 , wherein the imaging device captures MRI images.
13. The system according to claim 1 , wherein the imaging device captures ultrasound images.