Granted Patent B2

US 12,657,769 · App. 18/531,766 · Granted Jun 16, 2026

Inventors: Sanjay M. Joshi (Andover, MA); Norbert Johnson (North Andover, MA); Samuel Paster (Jamaica Plain, MA)

Assignee: Globus Medical, Inc.

G06T7/75G06T7/344G06T7/55G06T7/60G06T19/00G06T2207/30004G06T2210/41

Abstract

Claims (77)

1 . A computer implemented method comprising:

obtaining a plurality of 2D images of an object implanted in a patient;

obtaining a computer-aided design (CAD) model corresponding to the object; and

augmenting a 3D scan of the patient to provide a representation of the CAD model with a pose in a reference system of the 3D scan determined based on pose of the object in the 2D images;

wherein, for each of the 2D images, the method comprises:

determining poses of an emitter and a detector of an imaging device corresponding to when the 2D image was generated by the imaging device, and

creating a back-propagation pyramid estimation of the object in a 3D voxel array based on the object oriented in the 2D image corresponding to the emitter posed at an apex and the detector posed at the base of the back-propagation pyramid, and where the number of voxels in the 3D voxel array is based on a number of pixels in the 2D image; and

determining a 3D shape of the object in the 2D images based on superimposing the back-propagation pyramids.

2 . The method of claim 1 , wherein the augmenting of the 3D scan of the patient to provide the representation of the CAD model with the pose in the reference system of the 3D scan determined based on pose of the object in the 2D images, comprises:

rendering a graphical representation of the object based on the CAD model; and

overlaying the graphical representation of the object with the pose in the reference system of the 3D scan.

3 . The method of claim 1 , wherein the obtaining of the CAD model corresponding to the object, comprises:

selecting the CAD model from a set of CAD models residing in a memory based on information identify the CAD model as corresponding to the object.

4 . The method of claim 1 , wherein:

the CAD model defines shape and dimensions of the object; and

the augmenting of the 3D scan of the patient provides the representation of the CAD model scaled based on the dimensions of the object defined by the CAD model and posed in the reference system of the 3D scan based on the pose of the object in the 2D images.

5 . The method of claim 4 , further comprising:

projecting features of the shape defined by the CAD model into the 2D images; and

determining the pose of the CAD model in the reference system of the 3D scan based on correlating features of the object in the 2D images to the projected features of the shape defined by the CAD model.

6 . The method of claim 5 , wherein the projecting of the features of the shape defined by the CAD model into the 2D images and the determining of the pose of the CAD model in the reference system of the 3D scan, comprise:

estimating pose of the object in the 2D images;

computing expected object images based on the features having the shape defined by the CAD model projected into the 2D images using the estimated pose of the object in the 2D images;

comparing the expected object images to the 2D images of the object to compute a difference measure; and

determining whether the difference measure satisfies a rule, and

when the difference measure satisfies the rule

determining a pose of the CAD model in a 3D space based on the estimating pose of the object in the 2D images, and

determining the pose of the CAD model in the reference system of the 3D scan based on the pose of the CAD model in the 3D space.

7 . The method of claim 6 , further comprising:

when the difference measure does not satisfies the rule,

adjusting the estimate of the pose of the object in the 2D images; and

repeating the computing of new expected object images based on the features having the shape defined by the CAD model projected into the 2D images using the adjusted estimated pose of the object in the 2D images, the comparing of the new expected object images to the 2D images of the object to compute a new difference measure, and the determining whether the new difference measure satisfies the rule.

8 . The method of claim 1 ,

wherein the CAD model defines shape of the object, and

further comprising:

correlating features of the shape of the object defined by the CAD model to corresponding features of the object in the 2D images;

estimating dimensions of the corresponding features of the object in the 2D images; and

estimating dimensions of the shape of the object defined by the CAD model based on the estimated dimensions of the corresponding features of the object in the 2D images.

9 . The method of claim 1 , wherein the superimposing the back-propagation pyramids comprises:

performing a binary OR operation on values at corresponding locations in the 3D voxel array of the back-propagation pyramids.

10 . The method of claim 1 , further comprising:

determining a minimum number of 2D images to be used to create the back-propagation pyramids for determining the 3D shape of a defined cross-sectional feature of the object, based on a mathematical square of a minimum number of pixels in the 2D images to define an area enclosing the defined cross-sectional feature of the object.

11 . The method of claim 1 , further comprising:

registering a reference system of the 2D images to a reference system of the 3D scan; and

determining pose of the object in the reference system of the 3D scan based on the pose of the object in the reference system of the 2D images and the registration of the reference system of the 2D images to the reference system of the 3D scan,

wherein the augmenting of the 3D scan of the patient provides the representation of the CAD model with the determined pose of the object in the reference system of the 3D scan.

12 . The method of claim 11 , further comprising:

tracking poses of an imaging device relative to a dynamic reference base (DRB) attached to the patient at instances when the 2D images are captured by the imaging device,

wherein the registering of the reference system of the 2D images to the reference system of the 3D scan is further based on the tracked poses of the imaging device relative to the DRB.

13 . The method of claim 1 , wherein:

the plurality of 2D images have a 2D imaging modality; and

the 3D scan has a 3D imaging modality different from the 2D imaging modality.

14 . The method of claim 13 , wherein:

the 2D imaging modality is one of x-ray image, fluoroscopy image, and ultrasound image; and

the 3D imaging modality is one of computerized tomography (CT), magnetic resonance imaging (MRI), Cone Beam Computerized Tomography (CBCT), Micro Computerized Tomography (MCT), and 3D ultrasound.

15 . A computer processing platform comprising:

at least one network interface;

at least one processor connected to communicate through the at least one network interface; and

at least memory storing instructions executable by the at least one processor to perform operations to:

obtain a plurality of 2D images of an object implanted in a patient;

obtain a computer-aided design (CAD) model corresponding to the object; and

augment a 3D scan of the patient to provide a representation of the CAD model with a pose in a reference system of the 3D scan determined based on pose of the object in the 2D images,

wherein, for each of the 2D images, the computer processing platform comprises instructions configured to perform operations to:

determine poses of an emitter and a detector of an imaging device corresponding to when the 2D image was generated by the imaging device;

create a back-propagation pyramid estimation of the object in a 3D voxel array based on the object oriented in the 2D image corresponding to the emitter posed at an apex and the detector posed at the base of the back-propagation pyramid, wherein the number of voxels in the 3D voxel array is based on a number of pixels in the 2D image; and

determine a 3D shape of the object in the 2D images based on superimposing the back-propagation pyramids.

16 . The computer processing platform of claim 15 , wherein the operations to augment the 3D scan of the patient to provide the representation of the CAD model with the pose in the reference system of the 3D scan determined based on pose of the object in the 2D images, comprises to:

render a graphical representation of the object based on the CAD model; and

overlay the graphical representation of the object with the pose in the reference system of the 3D scan.

17 . The computer processing platform of claim 15 , wherein

the CAD model defines shape and dimensions of the object, and

the operations further comprise to project features of the shape defined by the CAD model into the 2D images, and to determine the pose of the CAD model in the reference system of the 3D scan based on correlating features of the object in the 2D images to the projected features of the shape defined by the CAD model.

18 . The computer processing platform of claim 15 , wherein

the CAD model defines shape of the object, and

the operations further comprise to:

correlate features of the shape of the object defined by the CAD model to corresponding features of the object in the 2D images;

estimate dimensions of the corresponding features of the object in the 2D images; and

estimate dimensions of the shape of the object defined by the CAD model based on the estimated dimensions of the corresponding features of the object in the 2D images.

Assignments (1)

Continuity (1)

References Cited (400)