Granted Patent B2

US 12,293,472 · App. 17/793,595 · Granted May 6, 2025

Inventors: Maximilian H. Allan (San Francisco, CA); Mahdi Azizian (San Jose, CA); A. Jonathan McLeod (Sunnyvale, CA)

Assignee: Intuitive Surgical Operations, Inc.

G06T19/006G06T7/20G06T7/50G06T2207/10016G06T2210/41

Abstract

Claims (68)

1. A system comprising:

a memory storing instructions; and

a processor communicatively coupled to the memory and configured to execute the instructions to:

access a model of a recognized object depicted in an original image of a scene;

associate the model with the recognized object; and

generate presentation data for use by a presentation system to present an augmented version of the original image in which a synthetic element added to the original image is, based on the model as associated with the recognized object, prevented from occluding at least a portion of the recognized object,

wherein the associating of the model with the recognized object includes:

generating a depth map of imagery depicted by the original image, the depth map including first depth data for a depiction of the recognized object within the imagery and second depth data for a remainder of the imagery, the first depth data based on the model of the recognized object and denser than the second depth data; and

segmenting the original image to distinguish pixels of the original image that depict the recognized object from pixels of the original image that do not depict the recognized object by

identifying the pixels of the original image that depict the recognized object based on the first depth data; and

identifying the pixels of the original image that do not depict the recognized object based on the second depth data.

2. The system of claim 1 , wherein:

the original image is a video image consisting of a sequence of image frames each depicting the recognized object as the recognized object moves with respect to other imagery depicted by the original image; and

the processor is further configured to execute the instructions to identify the recognized object within imagery depicted by the video image by

initially recognizing the recognized object in a first image frame of the sequence of image frames, and

tracking, based on the initial recognition, the recognized object from image frame to image frame as the recognized object moves with respect to the other imagery in later image frames of the sequence of image frames.

3. The system of claim 1 , wherein the processor is further configured to execute the instructions to:

access kinematic data representative of a pose of the recognized object; and

identify the recognized object within imagery depicted by the original image based on the kinematic data.

4. The system of claim 1 , wherein:

the scene depicted by the original image includes a surgical site at which a surgical procedure is being performed by way of a computer-assisted surgical instrument being used to perform the surgical procedure;

the recognized object is the computer-assisted surgical instrument; and

the synthetic element added to the original image for the augmented version of the original image is an anatomical model generated prior to the surgical procedure using a preoperative imaging modality.

5. The system of claim 1 , wherein the recognized object is a computer-assisted surgical instrument being used to perform a surgical procedure, and the model is a three-dimensional (“3D”) computer-aided design (“CAD”) model of the computer-assisted surgical instrument.

6. The system of claim 1 , wherein the recognized object is held by a computer-assisted surgical instrument being used to perform a surgical procedure, and the model is generated by way of a 3D scan of the recognized object.

7. The system of claim 1 , wherein the processor is further configured to execute the instructions to provide the generated presentation data to the presentation system for use in presenting the augmented version of the original image.

8. A method comprising:

accessing, by an object masking system, a model of a recognized object depicted in an original image of a surgical scene;

associating, by the object masking system, the model with the recognized object; and

generating, by the object masking system, presentation data for use by a presentation system to present an augmented version of the original image in which a synthetic element added to the original image is, based on the model as associated with the recognized object, prevented from occluding at least a portion of the recognized object,

wherein the associating of the model with the recognized object includes:

generating a depth map of imagery depicted by the original image, the depth map including first depth data for a depiction of the recognized object within the imagery and second depth data for a remainder of the imagery, the first depth data based on the model of the recognized object and denser than the second depth data; and

segmenting the original image to distinguish pixels of the original image that depict the recognized object from pixels of the original image that do not depict the recognized object by

identifying the pixels of the original image that depict the recognized object based on the first depth data; and

identifying the pixels of the original image that do not depict the recognized object based on the second depth data.

9. The method of claim 8 , wherein:

the original image is a video image consisting of a sequence of image frames each depicting the recognized object as the recognized object moves with respect to other imagery depicted by the original image; and

the method further comprises identifying the recognized object within imagery depicted by the video image by

initially recognizing the recognized object in a first image frame of the sequence of image frames, and

tracking, based on the initial recognition, the recognized object from image frame to image frame as the recognized object moves with respect to the other imagery in later image frames of the sequence of image frames.

10. The method of claim 8 , further comprising:

accessing, by the object masking system, kinematic data representative of a pose of the recognized object; and

identifying, by the object masking system, the recognized object within imagery depicted by the original image based on the kinematic data.

11. The method of claim 8 , wherein:

the scene depicted by the original image includes a surgical site at which a surgical procedure is being performed by way of a computer-assisted surgical instrument being used to perform the surgical procedure;

the recognized object is the computer-assisted surgical instrument; and

the synthetic element added to the original image for the augmented version of the original image is an anatomical model generated prior to the surgical procedure using a preoperative imaging modality.

12. The method of claim 8 , wherein the recognized object is a computer-assisted surgical instrument being used to perform a surgical procedure, and the model is a three-dimensional (“3D”) computer-aided design (“CAD”) model of the computer-assisted surgical instrument.

13. The method of claim 8 , wherein the recognized object is held by a computer-assisted surgical instrument being used to perform a surgical procedure, and the model is generated by way of a 3D scan of the recognized object.

14. The method of claim 8 , further comprising:

presenting, based on the presentation data, an augmented image that includes the original image augmented with only a portion of the synthetic element.

15. A non-transitory computer-readable medium storing instructions that, when executed, direct a processor of a computing device to:

access a model of a recognized object depicted in an original image of a surgical scene

associate the model with the recognized object; and

generate presentation data for use by a presentation system to present an augmented version of the original image in which a synthetic element added to the original image is, based on the model as associated with the recognized object, prevented from occluding at least a portion of the recognized object,

wherein the associating of the model with the recognized object includes:

generating a depth map of imagery depicted by the original image, the depth map including first depth data for a depiction of the recognized object within the imagery and second depth data for a remainder of the imagery, the first depth data based on the model of the recognized object and denser than the second depth data; and

segmenting the original image to distinguish pixels of the original image that depict the recognized object from pixels of the original image that do not depict the recognized object by

identifying the pixels of the original image that depict the recognized object based on the first depth data; and

identifying the pixels of the original image that do not depict the recognized object based on the second depth data.

16. The non-transitory computer-readable medium of claim 15 , wherein:

the original image is a video image consisting of a sequence of image frames each depicting the recognized object as the recognized object moves with respect to other imagery depicted by the original image; and

the instructions further direct the processor to identify the recognized object within imagery depicted by the video image by

initially recognizing the recognized object in a first image frame of the sequence of image frames, and

tracking, based on the initial recognition, the recognized object from image frame to image frame as the recognized object moves with respect to the other imagery in later image frames of the sequence of image frames.

17. The non-transitory computer-readable medium of claim 15 , wherein the instructions further direct the processor to:

access kinematic data representative of a pose of the recognized object; and

identify the recognized object within imagery depicted by the original image based on the kinematic data.

Assignments (1)

Continuity (2)

References Cited (9)