IP Library › Granted Patent US 12,260,551
Granted Patent B2
US 12,260,551 · App. 17/755,558 · Granted Mar 25, 2025

Compensation of organ deformation for medical image registration

Inventors: Julian Krebs (Moers, DE); Sebastien Piat (Lawrence Township, NJ)
Assignee: Siemens Healthineers AG
G06T7/0012G06T7/11G06V10/25G06V10/761G06V10/7715G06V10/82G06T2207/20081G06T2207/20084G06V2201/03G06V2201/07
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,260,551
App. No.
17/755,558
Granted
Mar 25, 2025
Kind
B2
Abstract

Systems and methods for medical image registration are provided. A first input medical image and a second input medical image of one or more anatomical objects are received. For each respective anatomical object of the one or more anatomical objects, a region of interest comprising the respective anatomical object is detected in one of the first input medical image or the second input medical image, the region of interest is extracted from the first input medical image and from the second input medical image, and a motion distribution of the respective anatomical object is determined from one of the region of interest extracted from the first input medical image or the region of interest extracted from the second input medical image using a motion model specific to the respective anatomical object. The first input medical image and the second input medical image are registered based on the motion distribution of each respective anatomical object of the one or more anatomical objects to generate a fused image.

Claims (63)

1. A method, comprising:

receiving a first input medical image and a second input medical image of one or more anatomical objects;

for each respective anatomical object of the one or more anatomical objects:

detecting a region of interest comprising the respective anatomical object in one of the first input medical image or the second input medical image,

extracting the region of interest from the first input medical image and from the second input medical image, and

determining a motion distribution of the respective anatomical object from the region of interest extracted from one of the first input medical image or the region of interest extracted from the second input medical image using a motion model specific to the respective anatomical object; and

registering the first input medical image and the second input medical image based on the motion distribution of each respective anatomical object of the one or more anatomical objects.

2. The method of claim 1 , wherein registering the first input medical image and the second input medical image based on the motion distribution of each respective anatomical object of the one or more anatomical objects comprises:

determining a regularization term for the one or more anatomical objects based on distances between the motion distribution of each respective anatomical object and a prior distribution.

3. The method of claim 2 , wherein determining a regularization term for the one or more anatomical objects based on distances between the motion distribution of each respective anatomical object and a prior distribution comprises:

summing the distances between the motion distribution of each respective anatomical object and the prior distribution.

4. The method of claim 2 , wherein registering the first input medical image and the second input medical image based on the motion distribution of each respective anatomical object of the one or more anatomical objects comprises:

minimizing a loss function including the regularization term for the one or more anatomical objects.

5. The method of claim 1 , wherein the motion model specific to the respective anatomical object comprises a variational autoencoder comprising an encoder, and wherein determining a motion distribution of the respective anatomical object from one of the region of interest extracted from the first input medical image or the region of interest extracted from the second input medical image using a motion model specific to the respective anatomical object comprises:

determining the motion distribution of the respective anatomical object using the encoder.

6. The method of claim 1 , further comprising, for each respective anatomical object, learning the motion model specific to the respective anatomical object by:

receiving a first training image and a second training image of the respective anatomical object;

detecting a region of interest comprising the respective anatomical object in one of the first training image or the second training image;

extracting the region of interest from the first training image and the second training image; and

training a machine learning network to model motion of the respective anatomical object from the region of interest extracted from the first training image and the region of interest extracted from the second training image as the motion model specific to the respective anatomical object.

7. The method of claim 6 , wherein training a machine learning network to model motion of the respective anatomical object from the region of interest extracted from the first training image and the region of interest extracted from the second training image as the motion model specific to the respective anatomical object comprises:

training an encoder of a variational autoencoder to generate a code representing an encoding of deformation between the region of interest extracted from the first training image and the region of interest extracted from the second training image; and

training a decoder of the variational autoencoder to generate a deformation field from the code and the region of interest extracted from the first training image, the deformation field representing the deformation between the region of interest extracted from the first training image and the region of interest extracted from the second training image.

8. The method of claim 1 , wherein detecting a region of interest comprising the respective anatomical object in one of the first input medical image or the second input medical image comprises:

segmenting the respective anatomical object in one of the first input medical image or the second input medical image; and

centering the region of interest around the segmented respective anatomical object.

9. The method of claim 1 , wherein the one or more anatomical objects comprises one or more organs.

10. An apparatus, comprising:

means for receiving a first input medical image and a second input medical image of one or more anatomical objects;

means for, for each respective anatomical object of the one or more anatomical objects, detecting a region of interest comprising the respective anatomical object in one of the first input medical image or the second input medical image;

means for, for each respective anatomical object, extracting the region of interest from the first input medical image and from the second input medical image;

means for, for each respective anatomical object, determining a motion distribution of the respective anatomical object from the region of interest extracted from one of the first input medical image or the region of interest extracted from the second input medical image using a motion model specific to the respective anatomical object; and

means for registering the first input medical image and the second input medical image based on the motion distribution of each respective anatomical object of the one or more anatomical objects.

11. The apparatus of claim 10 , wherein the means for registering the first input medical image and the second input medical image based on the motion distribution of each respective anatomical object of the one or more anatomical objects comprises:

means for determining a regularization term for the one or more anatomical objects based on distances between the motion distribution of each respective anatomical object and a prior distribution.

12. The apparatus of claim 11 , wherein the means for determining a regularization term for the one or more anatomical objects based on distances between the motion distribution of each respective anatomical object and a prior distribution comprises:

means for summing the distances between the motion distribution of each respective anatomical object and the prior distribution.

13. The apparatus of claim 11 , wherein the means for registering the first input medical image and the second input medical image based on the motion distribution of each respective anatomical object of the one or more anatomical objects comprises:

means for minimizing a loss function including the regularization term for the one or more anatomical objects.

14. The apparatus of claim 10 , wherein the motion model specific to the respective anatomical object comprises a variational autoencoder comprising an encoder, and wherein the means for determining a motion distribution of the respective anatomical object from the region of interest extracted from one of the first input medical image or the region of interest extracted from the second input medical image using a motion model specific to the respective anatomical object comprises:

means for determining the motion distribution of the respective anatomical object using the encoder.

15. A non-transitory computer readable medium storing computer program instructions, the computer program instructions when executed by a processor cause the processor to perform operations comprising:

receiving a first input medical image and a second input medical image of one or more anatomical objects;

for each respective anatomical object of the one or more anatomical objects:

detecting a region of interest comprising the respective anatomical object in one of the first input medical image or the second input medical image,

extracting the region of interest from the first input medical image and from the second input medical image, and

determining a motion distribution of the respective anatomical object from the region of interest extracted from one of the first input medical image or the region of interest extracted from the second input medical image using a motion model specific to the respective anatomical object; and

registering the first input medical image and the second input medical image based on the motion distribution of each respective anatomical object of the one or more anatomical objects.

16. The non-transitory computer readable medium of claim 15 , wherein registering the first input medical image and the second input medical image based on the motion distribution of each respective anatomical object of the one or more anatomical objects comprises:

determining a regularization term for the one or more anatomical objects based on distances between the motion distribution of each respective anatomical object and a prior distribution.

17. The non-transitory computer readable medium of claim 16 , wherein registering the first input medical image and the second input medical image based on the motion distribution of each respective anatomical object of the one or more anatomical objects comprises:

minimizing a loss function including the regularization term for the one or more anatomical objects.

18. The non-transitory computer readable medium of claim 15 , further comprising, for each respective anatomical object, learning the motion model specific to the respective anatomical object by:

receiving a first training image and a second training image of the respective anatomical object;

detecting a region of interest comprising the respective anatomical object in one of the first training image or the second training image;

extracting the region of interest from the first training image and the second training image; and

training a machine learning network to model motion of the respective anatomical object from the region of interest extracted from the first training image and the region of interest extracted from the second training image as the motion model specific to the respective anatomical object.

19. The non-transitory computer readable medium of claim 18 , wherein training a machine learning network to model motion of the respective anatomical object from the region of interest extracted from the first training image and the region of interest extracted from the second training image as the motion model specific to the respective anatomical object comprises:

training an encoder of a variational autoencoder to generate a code representing an encoding of deformation between the region of interest extracted from the first training image and the region of interest extracted from the second training image; and

training a decoder of the variational autoencoder to generate a deformation field from the code and the region of interest extracted from the first training image, the deformation field representing the deformation between the region of interest extracted from the first training image and the region of interest extracted from the second training image.

20. The non-transitory computer readable medium of claim 15 , wherein detecting a region of interest comprising the respective anatomical object in one of the first input medical image or the second input medical image comprises:

segmenting the respective anatomical object in one of the first input medical image or the second input medical image; and

centering the region of interest around the segmented respective anatomical object.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 20, 2023
From: SIEMENS HEALTHCARE GMBH
To: SIEMENS HEALTHINEERS AG
Reel/Frame 066267/0346 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 4, 2022
From: SIEMENS MEDICAL SOLUTIONS USA, INC.
To: SIEMENS HEALTHCARE GMBH
Reel/Frame 059804/0150 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 2, 2022
From: KREBS, JULIAN; PIAT, SEBASTIEN
To: SIEMENS MEDICAL SOLUTIONS USA, INC.
Reel/Frame 059781/0303 →
Continuity (1)
Related Publication 20220270256A1 · Aug 25, 2022
References Cited (26)
US 11154196B2 · Zhang · 2021 [cited by applicant]
US 20110085716A1 · Chefd'Hotel et al. · 2011 [cited by applicant]
US 20150055842A1 · Codella et al. · 2015 [cited by applicant]
US 20150055942A1 · Codella et al. · 2015 [cited by applicant]
US 20150065859A1 · Hwang · 2015 [cited by examiner]
US 20170337682A1 · Liao et al. · 2017 [cited by applicant]
US 20190205766A1 · Krebs et al. · 2019 [cited by applicant]
CN 109102532B · 2022 [cited by applicant]
EP 3246875A2 · 2017 [cited by applicant]
Yang et al., “Quicksilver: Fast Predictive Image Registration—A Deep Learning Approach”, Neuroimage, 2017, vol. 158, pp. 378-396. [cited by applicant]
Rohe et al., “SVF-Net: Learning Deformable Image Registration Using Shape Matching”, International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2017, pp. 266-274. [cited by applicant]
Sokooti et al., “Nonrigid Image Registration Using Multi-scale 3D Convolutional Neural Networks”, International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2017, pp. 232-239. [cited by applicant]
de Vos et al., “End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network”, Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer, 2017, … [cited by applicant]
Dalca et al., “Unsupervised Learning for Fast Probabilistic Diffeomorphic Registration”, International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2018, pp. 729-738. [cited by applicant]
Krebs et al., “Learning a Probabilistic Model for Diffeomorphic Registration”, IEEE Transactions on Medical Imaging, 2019, pp. 1-12. [cited by applicant]
Hu et al., “Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration”, Medical Image Analysis, 2018, vol. 49, pp. 1-13. [cited by applicant]
Balakrishnan et al., “VoxelMorph: a Learning Framework for Deformable Medical Image Registration”, IEEE Transactions on Medical Imaging, 2019, pp. 1-13. [cited by applicant]
Krebs et al., “Probabilistic Motion Modeling from Medical Image Sequences: Application to Cardiac Cine-MRI”, STACOM 2019: Statistical Atlases and Computational Models of the Heart, 2019, 10 pgs. [cited by applicant]
Xu et al., “Efficient Multi-Atlas Abdominal Segmentation on Clinically Acquired CT with SIMPLE Context Learning”, Med Image Anal., 2015, vol. 24, No. 1, pp. 18-27. [cited by applicant]
Krebs et al., “Unsupervised Probabilistic Deformation Modeling for Robust Diffeomorphic Registration”, Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, 2018, pp. 101-109. [cited by applicant]
International Search Report mailed Jul. 23, 2020 in corresponding International Patent Application No. PCT/US2019/066141. [cited by applicant]
Ferrante Enzo et al.; 2018; “Weakly-Supervised Learning of Metric Aggregations for Deformable Image Registration”, Arxiv.Org, Cornell University. [cited by applicant]
Shan Siyuan et al: “Unsupervised End-to-end Learning for Deformable Medical Image Registration”; URL: https://arxiv.org/pdf/1711.08608.pdf / Nov. 23, 2017. [cited by applicant]
Preiswerk Frank et al.:; “Model guided respiratory organ motion prediction of the liver from 2D ultrasound”; 2014; ELSEVIER; pp. 740-751. [cited by applicant]
Ferrante, Enzo et al; “Weakly Supervised Learning of Metric Aggregations for Deformable Image Registration”; IEEE Journal of Biomedical and Health Informatics; vol. 23; No. 4. [cited by applicant]
Shan Siyuan et al; “Unsupervised End-to-end Learning for Deformable Medical Image Registration”; Nov. 23, 2017; vol. 14; No. 8; Journal of Latex Class Files. [cited by applicant]