Rendering tool information as graphic overlays on displayed images of tools
An operator telerobotically controls tools to perform a procedure on an object at a work site while viewing real-time images of the work site on a display. Tool information is provided in the operator's current gaze area on the display by rendering the tool information over the tool so as not to obscure objects being worked on at the time by the tool nor to require eyes of the user to refocus when looking at the tool information and the image of the tool on a stereo viewer.
1. A robotic system comprising:
a tool having a working end, the working end adapted to physically interact with an object;
a manipulator coupled to the tool for robotic manipulation of the tool;
a camera spatially disposed relative to the tool so that the camera captures an image of the working end of the tool from a perspective of an image reference frame;
a display coupled to the camera for displaying the image of the working end of the tool; and
a processor programmed to render information of the tool over the image of the working end of the tool which was captured by the camera and is being displayed on the display by:
determining a pose of the tool in the image reference frame;
registering the information of the tool with a designated place on the working end of the tool in the image reference frame by using the pose of the tool in the image reference frame; and
rendering the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display after registering the information of the tool with the designated place on the working end of the tool, so that the entire information of the tool appears as a decal that is rendered over the working end of the tool in the image of the working end of the tool being displayed on the display by bending the decal to conform in shape and be applied to an outer surface of the tool at the designated place.
2. The robotic system of claim 1 , wherein the processor is programmed to render the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool by rendering an indication that the tool is energized on the decal.
3. The robotic system of claim 2 , wherein the processor is programmed to render the indication that the tool is energized by rendering an icon on the decal, wherein presence of the icon indicates the tool is energized.
4. The robotic system of claim 2 , wherein the processor is programmed to render the indication that the tool is energized by rendering a color on the decal, wherein the color indicates the tool is energized.
5. The robotic system of claim 2 , wherein the processor is programmed to render the indication that the tool is energized by rendering a different brightness level on the decal, wherein the brightness level indicates the tool is energized.
6. The robotic system of claim 1 , wherein the working end of the tool includes an end effector, a wrist, and a distal end of a shaft, and the decal is rendered over one of the end effector, the wrist, and the distal end of the shaft.
7. The robotic system of claim 1 ,
wherein the processor is programmed to render the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display by rendering an indication of an identity of the manipulator on the decal.
8. The robotic system of claim 7 ; wherein the processor is programmed to render the indication of the identity of the manipulator by rendering a manipulator identification code on the decal, wherein the manipulator identification code uniquely identifies the manipulator.
9. The robotic system of claim 1 , further comprising:
an input device operatively coupled to the manipulator, the input device being manipulatable by an operator;
wherein the processor is programmed to render information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display by rendering an indication of an identity of at least one of the input device and the operator on the decal.
10. The robotic system of claim 1 , wherein the information of the tool indicates whether or not the tool is locked in position at the time.
11. The robotic system of claim 1 , wherein the information of the tool includes at least one of static and dynamic information.
12. The robotic system of claim 1 , wherein the processor is programmed to determine the pose of the tool relative to the image reference frame by: determining the pose of the tool in a reference frame of the tool and translating the pose of the tool from the reference frame of the tool to the pose of the tool in the image reference frame.
13. The robotic system of claim 12 , wherein the processor is programmed to determine the pose of the tool relative to the reference frame of the tool by using kinematics information of the manipulator.
14. The robotic system of claim 1 , wherein the display is a stereo viewer and the image of the tool being shown on the display is a three-dimensional image of the tool.
15. A method, implemented in a robotic system, the robotic system comprising: a tool having a working end; the working end adapted to physically interact with an object; a manipulator coupled to the tool for robotic manipulation of the tool; a camera spatially disposed relative to the tool so that the camera captures an image of the working end of the tool from a perspective of an image reference frame; and a display coupled to the camera for displaying the image of the working end of the tool;
the method comprising:
rendering information of the tool over the image of the working end of the tool which was captured by the camera and is being displayed on the display by:
determining a pose of the tool in the image reference frame;
registering the information of the tool with a designated place on the working end of the tool in the image reference frame by using the pose of the tool in the image reference frame; and
rendering the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display after registering the information of the tool with the designated place on the working end of the tool, so that the entire information of the tool appears as a decal that is rendered over the working end of the tool in the image of the working end of the tool being displayed on the display by bending the decal to conform in shape and be applied to an outer shape of the tool at the designated place.
16. The method of claim 15 , wherein the rendering of the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display comprises:
rendering an indication that the tool is energized on the decal.
17. The method of claim 16 , wherein the rendering of the indication that the tool is energized comprises:
rendering an icon on the decal, wherein presence of the icon indicates the tool is energized.
18. The method of claim 16 , wherein the rendering of the indication that the tool is energized comprises:
rendering a color on the decal, wherein the color indicates the tool is energized.
19. The method of claim 16 , wherein the rendering of the indication that the tool is energized comprises:
rendering a different brightness level on the decal, wherein the brightness level indicates the tool is energized.
20. The method of claim 15 , wherein the working end of the tool includes an end effector, a wrist, and a distal end of a shaft, and the decal is rendered over one of the end effector, the wrist, and the distal end of the shaft.
21. The method of claim 15 , wherein the rendering of the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display comprises:
rendering an indication of an identity of the manipulator on the decal.
22. The method of claim 21 , wherein the rendering of the indication of the identity of the manipulator comprises:
rendering a manipulator identification code on the decal, wherein the manipulator identification code uniquely identities the manipulator.
23. The method of claim 15 , wherein the rendering of the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display comprises:
rendering an indication of an identity of at least one of an input device and an operator on the decal, wherein the input device is operatively coupled to the manipulator and the input device is manipulatable by the operator.
24. The method of claim 15 , wherein the rendering of the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display comprises:
rendering an indication on the decal of whether or not the tool is locked in position at the time.
25. The method of claim 15 , wherein the information of the tool includes at least one of static and dynamic information.
26. The method of claim 15 , wherein the determining of the pose of the tool in the image reference frame comprises:
determining the pose of the tool relative to a reference frame of the tool; and
translating the pose of the tool from the reference frame of the tool to the pose of the tool in the image reference frame.
27. The method of claim 26 , wherein the determining of the pose of the tool relative to the reference frame of the tool employs kinematics information of the manipulator.
28. The method of claim 15 , wherein the display is a stereo viewer and the image of the tool being shown on the display is a three-dimensional image of the tool.