Granted Patent B2

US 12,475,578 · App. 17/912,784 · Granted Nov 18, 2025

Inventors: A. Jonathan McLeod (Sunnyvale, CA); Maximilian H. Allan (San Francisco, CA); Mahdi Azizian (San Jose, CA); Daniel Proksch (San Jose, CA); Pourya Shirazian (Menlo Park, CA)

Assignee: Intuitive Surgical Operations, Inc.

G06T7/33G06T2207/10068G06T2207/30004

Abstract

Claims (76)

1 . A system comprising:

a memory storing instructions;

a processor communicatively coupled to the memory and configured to execute the instructions to:

access semantic information regarding an anatomical object in a surgical scene, wherein the semantic information includes scene segmentation information for an image of the surgical scene and for a model of the anatomical object; and

register, based at least in part on the semantic information regarding the anatomical object, the image of the surgical scene with the model of the anatomical object, wherein the registering, based at least in part on the semantic information, the image of the surgical scene with the model of the anatomical object includes:

determining, based on the scene segmentation information, a reliability of a set of tissues in the surgical scene; and

registering, based at least in part on the reliability of the set of tissues, the image of the surgical scene and the model of the anatomical object.

2 . The system of claim 1 , wherein the registering, based at least in part on the semantic information, the image of the surgical scene with the model of the anatomical object includes:

determining, based on the scene segmentation information, a set of reliable tissues for registering the image of the surgical scene with the model of the anatomical object;

determining, in the image of the surgical scene, first regions depicting one or more tissues of the set of reliable tissues;

determining, in the image of the surgical scene, a first plurality of feature points in the first regions depicting the one or more tissues;

determining, in an image of the model of the anatomical object, second regions depicting the one or more tissues;

determining, in the image of the model of the anatomical object, a second plurality of feature points in the second regions depicting the one or more tissues; and

registering, based at least in part on an alignment of the first plurality of feature points with the second plurality of feature points, the image of the surgical scene and the model of the anatomical object.

3 . The system of claim 2 , wherein the determining the set of reliable tissues includes selecting at least one tissue from the scene segmentation information based on at least one characteristic of the selected at least one tissue.

4 . The system of claim 3 , wherein the at least one characteristic of the selected at least one tissue includes at least one of a hardness of the tissue, a rigidity of the tissue, a mobility of the tissue, or a typical amount of movement of the tissue during a procedure on the anatomical object.

5 . The system of claim 2 , wherein the determining the set of reliable tissues includes selecting at least one tissue from the scene segmentation information and comparing the selected at least one tissue to a predefined list.

6 . The system of claim 5 , wherein the predefined list includes bone, vasculature, and the anatomical object.

7 . The system of claim 1 , wherein the registering, based at least in part on the semantic information, the image of the surgical scene with the model of the anatomical object includes:

determining, based on the scene segmentation information, a set of unreliable tissues for registering the image of the surgical scene with the model of the anatomical object;

determining, in the image of the surgical scene, first regions depicting one or more tissues of the set of unreliable tissues;

determining, in the image of the surgical scene, a first plurality of feature points in regions of the image of the surgical scene other than the first regions;

determining, in an image of the model of the anatomical object, second regions depicting the one or more tissues;

determining, in the image of the model of the anatomical object, a second plurality of feature points in regions of the image of the model of the anatomical object other than the second regions; and

registering, based at least in part on an alignment of the first plurality of feature points with the second plurality of feature points, the image of the surgical scene and the model of the anatomical object.

8 . The system of claim 1 , wherein the registering, based at least in part on the semantic information, the image of the surgical scene with the model of the anatomical object includes:

determining, based on the scene segmentation information, a first set of the set of tissues and a second set of the set of tissues, the first and second sets determined based on the reliability of the tissues for registering the image of the surgical scene with the model of the anatomical object;

determining, in the image of the surgical scene, first regions depicting one or more tissues of the first set of tissues;

determining, in the image of the surgical scene, second regions depicting one or more tissues of the second set of tissues;

determining, in the image of the surgical scene, a first plurality of feature points in the first regions;

determining, in the image of the surgical scene, a second plurality of feature points in the second regions;

determining, in an image of the model of the anatomical object, third regions depicting one or more tissues of the first set of tissues;

determining, in the image of the model of the anatomical object, fourth regions depicting one or more tissues of the second set of tissues;

determining, in the image of the model of the anatomical object, a third plurality of feature points in the third regions;

determining, in the image of the model of the anatomical object, a fourth plurality of feature points in the fourth regions;

registering, based at least in part on a weighted alignment of the first plurality of feature points with the third plurality of feature points and the second plurality of feature points with the fourth plurality of feature points, the image of the surgical scene and the model of the anatomical object.

9 . The system of claim 1 , wherein the registering, based at least in part on the semantic information, the image of the surgical scene with the model of the anatomical object includes:

determining, based on the scene segmentation information, in the image of the surgical scene, a first boundary between a first tissue and a second tissue;

determining, based on the scene segmentation information, in an image of the model of the anatomical object, a second boundary between the first tissue and the second tissue; and

registering, based at least in part on an alignment of the first boundary and the second boundary, the image of the surgical scene and the model of the anatomical object.

10 . The system of claim 1 , wherein the semantic information includes a current phase of a procedure performed on the anatomical object.

11 . The system of claim 10 , wherein the registering, based at least in part on the semantic information, the image of the surgical scene with the model of the anatomical object includes selectively updating, based on the current phase of the procedure, an initial registration of the image of the surgical scene with the model of the anatomical object.

12 . The system of claim 10 , wherein the registering, based at least in part on the semantic information, the image of the surgical scene with the model of the anatomical object includes:

determining a set of procedure phases in which the anatomical object moves more than a threshold amount; and

selectively refraining, when the current phase of the procedure is one of the set of procedure phases, from registering the image of the surgical scene with the model of the anatomical object.

13 . The system of claim 1 , wherein the semantic information includes force sensing data of a computer-assisted surgical system performing a procedure on the anatomical object.

14 . The system of claim 13 , wherein the registering, based at least in part on the semantic information, the image of the surgical scene with the model of the anatomical object includes:

detecting, based on the force sensing data, an interaction with the anatomical object in the image of the surgical scene; and

modeling the interaction in the model of the anatomical object.

15 . The system of claim 1 , wherein the model of the anatomical object includes one of:

an ultrasound scan of the anatomical object or a surgical space;

a computerized tomography (“CT”) scan of the anatomical object or the surgical space;

a magnetic resonance imaging (“MRI”) scan of the anatomical object or the surgical space; or

a fluoroscopic imaging scan of the anatomical object or the surgical space.

16 . The system of claim 1 , wherein the model of the anatomical object includes at least one of a virtual object or an annotation.

17 . A method comprising:

accessing, by a processor, scene segmentation information for an image of a surgical scene of a surgical space including an anatomical object and a model of the anatomical object;

determining, by the processor, based on the scene segmentation information, a set of reliable tissues for registering a data set of the image of a surgical scene with the model of the anatomical object;

determining, by the processor, in the image of the surgical scene, first regions depicting one or more tissues of the set of reliable tissues;

determining, by the processor, in the image of the surgical scene, a first plurality of feature points in the first regions depicting the one or more tissues;

determining, by the processor, in an image of the model of the anatomical object, second regions depicting the one or more tissues;

determining, by the processor, in the image of the model of the anatomical object, a second plurality of feature points in the second regions depicting the one or more tissues; and

registering, by the processor, based at least in part on an alignment of the first plurality of feature points with the second plurality of feature points, the image of the surgical scene and the model of the anatomical object.

18 . The method of claim 17 , wherein the determining the set of reliable tissues includes selecting at least one tissue from the scene segmentation information based on at least one characteristic of the selected at least one tissue.

19 . A non-transitory computer-readable medium storing instructions that, when executed, direct at least one processor of a computing device to:

access semantic information regarding an anatomical object in a surgical scene, wherein the semantic information includes scene segmentation information for an image of the surgical scene and for a model of the anatomical object; and

register, based at least in part on the semantic information regarding the anatomical object, the image of the surgical scene with the model of the anatomical object, wherein the registering, based at least in part on the semantic information, the image of the surgical scene with the model of the anatomical object includes:

determining, based on the scene segmentation information, a reliability of a set of tissues in the surgical scene; and

registering, based at least in part on the reliability of the set of tissues, the image of the surgical scene and the model of the anatomical object.

20 . The computer-readable medium of claim 19 , wherein the registering, based at least in part on the semantic information, the image of the surgical scene with the model of the anatomical object includes:

determining, based on the scene segmentation information, a set of reliable tissues for registering the image of the surgical scene with the model of the anatomical object;

determining, in the image of the surgical scene, first regions depicting one or more tissues of the set of reliable tissues;

determining, in the image of the surgical scene, a first plurality of feature points in the first regions depicting the one or more tissues;

determining, in an image of the model of the anatomical object, second regions depicting the one or more tissues;

determining, in the image of the model of the anatomical object, a second plurality of feature points in the second regions depicting the one or more tissues; and

registering, based at least in part on an alignment of the first plurality of feature points with the second plurality of feature points, the image of the surgical scene and the model of the anatomical object.

Assignments (1)

Continuity (2)

References Cited (18)