Tool position and identification indicator displayed in a boundary area of a computer display screen
An endoscope captures images of a surgical site for display in a viewing area of a monitor. When a tool is outside the viewing area, a GUI indicates the position of the tool by positioning a symbol in a boundary area around the viewing area so as to indicate the tool position. The distance of the out-of-view tool from the viewing area may be indicated by the size, color, brightness, or blinking or oscillation frequency of the symbol. A distance number may also be displayed on the symbol. The orientation of the shaft or end effector of the tool may be indicated by an orientation indicator superimposed over the symbol, or by the orientation of the symbol itself. When the tool is inside the viewing area, but occluded by an object, the GUI superimposes a ghost tool at its current position and orientation over the occluding object.
1. A computer implemented method for indicating a position of a tool at a work site when the tool is within, but occluded in a field of view of a camera, the method comprising:
capturing images of the work site using the camera;
displaying the captured images in a viewing area of a computer display screen;
determining a position of the tool in a reference frame of the camera by using kinematics of a manipulator moving the tool; and
displaying a ghost tool in the viewing area where the tool is occluded by using the determined position of the tool in the reference frame of the camera.
2. The computer implemented method according to claim 1 , wherein the viewing area includes stereoscopic right and left two-dimensional views, and further comprising determining that the tool is occluded in the field of view of the camera by failing to identify an image of the tool in the captured images after generating a three-dimensional computer model of the tool; positioning and orientating the computer model to coincide with the current position and orientation of the tool in the reference frame of the computer display screen; generating a two-dimensional outline of the computer model that projects onto a selected one of the right and left two-dimensional views; and cross-correlating the two-dimensional outline of the computer model with the selected one of the right and left two-dimensional views.
3. The computer implemented method according to claim 2 , wherein the displaying of the ghost tool in the viewing area where the tool is occluded comprises: displaying at least the two-dimensional outline of the computer model of the tool so as to be displayed as an overlay on the selected one of the right and left two-dimensional views as it is being displayed on the computer display screen.
4. The computer implemented method according to claim 2 , wherein the displaying of the ghost tool in the viewing area where the tool is occluded comprises: overlaying right and left two-dimensional ghost images of the tool in corresponding of the right and left two-dimensional views, so as to appear as a three-dimensional ghost image of the tool at the position and orientation of the tool on the computer display screen.