Granted Patent B2

US 10,282,881 · App. 15/946,408 · Granted May 7, 2019

Inventors: Brandon D. Itkowitz (San Jose, CA); Simon P. DiMaio (San Carlos, CA); Daniel J. Halabe (Los Altos, CA); Christopher J. Hasser (Los Altos, CA); Brian D. Hoffman (Mountain View, CA); David Q. Larkin (Menlo Park, CA); Catherine J. Mohr (Mountain View, CA); Paul W. Mohr (Mountain View, CA); Tao Zhao (Sunnyvale, CA); Wenyi Zhao (Mountain View, CA)

Assignee: Intuitive Surgical Operations, Inc.

G06T11/60A61B34/30A61B34/37A61B90/36B25J9/1666B25J9/1671B25J9/1692G05B19/4202G06T11/40A61B34/20A61B34/25A61B90/361A61B2034/2061A61B2090/371G05B2219/36432G05B2219/39083G05B2219/39096G05B2219/39449G05B2219/40607G05B2219/45117G05B2219/45123Y02P90/265

Abstract

Claims (86)

1. A system comprising:

a first input device;

a first tool having a first working end, the first tool operatively coupled to the first input device;

an image capture device adapted to generate a captured image from a perspective of an image reference frame, the captured image including an image of the first working end;

a display; and

a processor programmed to:

cause the captured image to be displayed on the display; and

cause information of the first input device to be rendered on the image of the first working end in the captured image being displayed on the display by:

determining a pose of the first working end in the image reference frame;

registering the information of the first input device with a first designated place on the first working end in the image reference frame by using the pose of the first working end in the image reference frame; and

rendering the information of the first input device over the first designated place on the first working end on the image of the first working end in the captured image being displayed on the display, after registering the information of the first input device with the first designated place on the first working end, so that the entire information of the first input device appears as a first decal that is rendered over the image of the first working end in the captured image being displayed on the display by bending the first decal to conform in shape and be applied to an outer surface of the first working end at the first designated place.

2. The system of claim 1 ,

wherein the first input device is being operated by a user; and

wherein the information of the first input device comprises information identifying the user.

3. The system of claim 1 ,

wherein the first input device is being operated by an appendage of a user; and

wherein the information of the first input device comprises information identifying the appendage of the user.

4. The system of claim 1 ,

wherein the first working end of the first tool includes an end effector, a wrist, and a distal end of a shaft, and the first decal is rendered over one of the end effector, the wrist, and the distal end of the shaft.

5. The system of claim 1 , further comprising:

a second input device;

a second tool having a second working end, the second tool operatively coupled to the second input device;

wherein the captured image includes an image of the second working end; and

wherein the processor is further programmed to:

cause information of the second input device to be rendered on the image of the second working end in the captured image being displayed on the display by:

determining a pose of the second working end in the image reference frame;

registering the information of the second input device with a second designated place on the second working end in the image reference frame by using the pose of the second working end in the image reference frame; and

rendering the information of the second input device over the second designated place on the second working end on the image of the second working end in the captured image being displayed on the display, after registering the information of the second input device with the second designated place on the second working end, so that the entire information of the second input device appears as a second decal that is rendered over the image of the second working end in the captured image being displayed on the display by bending the second decal to conform in shape and be applied to an outer surface of the second working end at the second designated place.

6. The system of claim 5 ,

wherein the first input device is being operated by a first user;

wherein the second input device is being operated by a second user;

wherein the information of the first input device comprises information identifying the first user; and

wherein the information of the second input device comprises information identifying the second user.

7. The system of claim 5 ,

wherein the first input device is being operated by a first appendage of a user;

wherein the second input device is being operated by a second appendage of the user; and

wherein the information of the first input device comprises information identifying the first appendage; and

wherein the information of the second input device comprises information identifying the second appendage.

8. The system of claim 1 ,

wherein the processor is further programmed to determine the pose of the first working end relative to the image reference frame by:

determining the pose of the first working end in a reference frame of the first tool and translating the pose of the first working end from the reference frame of the first tool to the pose of the first working end in the image reference frame.

9. The system of claim 8 , further comprising:

a first manipulator adapted to manipulate the first tool;

wherein the processor is further programmed to determine the pose of the first working end relative to the reference frame of the first tool by using kinematics information of the first manipulator.

10. The system of claim 1 ,

wherein the image capture device comprises a stereoscopic camera, so that the captured image includes a stereo view of the first working end; and

wherein the display is a stereo viewer.

11. A method comprising:

a processor causing a captured image to be displayed on a display, the captured image generated from a perspective of an image reference frame by an image capture device, the captured image including an image of a first working end of a first tool, the first tool operatively coupled to a first input device; and

the processor causing information of the first input device to be rendered on the image of the first working end in the captured image being displayed on the display by:

determining a pose of the first working end in the image reference frame;

registering the information of the first input device with a first designated place on the first working end in the image reference frame by using the pose of the first working end in the image reference frame; and

rendering the information of the first input device over the first designated place on the first working end on the image of the first working end in the captured image being displayed on the display, after registering the information of the first input device with the first designated place on the first working end, so that the entire information of the first input device appears as a first decal that is rendered over the image of the first working end in the captured image being displayed on the display by bending the first decal to conform in shape and be applied to an outer surface of the first working end at the first designated place.

12. The method of claim 11 ,

wherein the first input device is being operated by a user; and

wherein the information of the first input device comprises information identifying the user.

13. The method of claim 11 ,

wherein the first input device is being operated by an appendage of a user; and

wherein the information of the first input device comprises information indicating the appendage of the user.

14. The method of claim 11 ,

wherein the first working end of the first tool includes an end effector, a wrist, and a distal end of a shaft, and the first decal is rendered over one of the end effector, the wrist, and the distal end of the shaft.

15. The method of claim 11 ,

wherein the captured image includes an image of a second working end of a second tool, the second tool operatively coupled to a second input device; and

further comprising:

the processor causing information of the second input device to be rendered on the image of the second working end in the captured image being displayed on the display by:

determining a pose of the second working end in the image reference frame;

registering the information of the second input device with a second designated place on the second working end in the image reference frame by using the pose of the second working end in the image reference frame; and

rendering the information of the second input device over the second designated place on the second working end on the image of the second working end in the captured image being displayed on the display, after registering the information of the second input device with the second designated place on the second working end, so that the entire information of the second input device appears as a second decal that is rendered over the image of the second working end in the captured image being displayed on the display by bending the second decal to conform in shape and be applied to an outer surface of the second working end at the second designated place.

16. The method of claim 15 ,

wherein the first input device is being operated by a first user;

wherein the second input device is being operated by a second user;

wherein the information of the first input device comprises information identifying the first user; and

wherein the information of the second input device comprises information identifying the second user.

17. The method of claim 15 ,

wherein the first input device is being operated by a first appendage of a user;

wherein the second input device is being operated by a second appendage of the user; and

wherein the information of the first input device comprises information identifying the first appendage; and

wherein the information of the second input device comprises information identifying the second appendage.

18. The method of claim 11 ,

wherein the processor determines the pose of the first working end relative to the image reference frame by:

determining the pose of the first working end in a reference frame of the first tool and translating the pose of the first working end from the reference frame of the first tool to the pose of the first working end in the image reference frame.

19. The method of claim 18 ,

wherein the processor determines the pose of the first working end relative to the reference frame of the first tool by using kinematics information of a first manipulator that is adapted to manipulate the first tool.

20. The method of claim 11 ,

wherein the image capture device comprises a stereoscopic camera, so that the captured image includes a stereo view of the first working end; and

wherein the display is a stereo viewer.

Assignments (2)

Continuity (3)

Cited By (55)

US 1,066,378 US 1,066,379 US 1,066,380 US 1,066,381 US 1,066,382 US 1,066,383 US 1,066,404 US 1,066,405 US 1,087,135 US 1,087,995 US 12,186,040 US 12,207,894 US 12,223,629 US 12,239,396 US 12,251,178 US 12,256,890 US 12,262,863 US 12,262,964 US 12,266,040 US 12,295,681 US 12,310,685 US 12,315,109 US 12,329,475 US 12,350,828 US 12,357,400 US 12,369,998 US 12,376,934 US 12,380,998 US 12,390,294 US 12,409,003 US 12,419,711 US 12,433,699 US 12,433,700 US 12,465,447 US 12,479,098 US 12,491,028 US 12,496,119 US 12,496,728 US 12,548,667 US 12,564,457 US 12,599,445 US 12,599,449 US 12,611,086 US 12,611,272 US 12,616,474 US 12,617,078 US 12,626,422 US 12,642,613 US 12,648,685 US 12,661,192 US 12,672,927 US 12,678,238 US 12,685,603 US 12,708,466 US 12,733,950