IP Library › Granted Patent US 12,373,961
Granted Patent B2
US 12,373,961 · App. 17/654,323 · Granted Jul 29, 2025

Automatic MR-US prostate image fusion through semi-supervised constrained learning

Inventors: Sureerat Reaungamornrat (Havertown, PA); Mamadou Diallo (Plainsboro, NJ); Ali Kamen (Skillman, NJ)
Assignee: Siemens Healthineers AG
G06T7/33G06T7/11G06T2207/10028G06T2207/10088G06T2207/10132G06T2207/20081G06T2207/20084G06T2207/30081
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,373,961
App. No.
17/654,323
Granted
Jul 29, 2025
Kind
B2
Abstract

Systems and methods for automatically registering a first input medical image and a second input medical image are provided. The first input medical image in a first modality and the second input medical image in a second modality are received. One or more objects of interest are segmented from the first input medical image to generate a first segmentation map and one or more objects of interest are segmented from the second input medical image to generate a second segmentation map. A first point cloud is extracted from the first segmentation map and a second point cloud is extracted from the second segmentation map. A transformation for aligning the first point cloud and the second point cloud is determined to register the first input medical image and the second input medical image. The transformation is output.

Claims (42)

1. A computer-implemented method comprising:

receiving a first input medical image in a first modality and a second input medical image in a second modality;

segmenting one or more objects of interest from the first input medical image to generate a first segmentation map;

segmenting one or more objects of interest from the second input medical image to generate a second segmentation map;

extracting a first point cloud from the first segmentation map;

extracting a second point cloud from the second segmentation map;

determining a transformation for aligning the first point cloud and the second point cloud using a machine learning based point registration network based on a density of points in the first point cloud and the second point cloud to register the first input medical image and the second input medical image, the machine learning based point registration network comprising 1) one or more point embedding networks for generating embeddings based on the first input medical image and the second input medical image, 2) one or more attention modules for generating permutation invariant features based on the embeddings, and 3) a transform regressor network for generating the transformation based on the permutation invariant features; and

outputting the transformation.

2. The computer-implemented method of claim 1 , wherein determining a transformation for aligning the first point cloud and the second point cloud using a machine learning based point registration network based on a density of points in the first point cloud and the second point cloud to register the first input medical image and the second input medical image comprises:

determining a forward transformation to align the first point cloud to the second point cloud; and

determining a backward transformation to align the second point cloud to the first point cloud.

3. The computer-implemented method of claim 1 , wherein the first point cloud and the second point cloud have a different number of points.

4. The computer-implemented method of claim 1 , wherein the transformation is not determined based on a correspondence between points in the first point cloud and the second point cloud.

5. The computer-implemented method of claim 1 , wherein the first modality is ultrasound and the second modality is magnetic resonance.

6. The computer-implemented method of claim 1 , wherein the one or more objects of interest segmented from the first input medical image and the one or more objects of interest segmented from the second input medical image comprise a prostate of a patient.

7. An apparatus comprising:

means for receiving a first input medical image in a first modality and a second input medical image in a second modality;

means for segmenting one or more objects of interest from the first input medical image to generate a first segmentation map;

means for segmenting one or more objects of interest from the second input medical image to generate a second segmentation map;

means for extracting a first point cloud from the first segmentation map;

means for extracting a second point cloud from the second segmentation map;

means for determining a transformation for aligning the first point cloud and the second point cloud using a machine learning based point registration network based on a density of points in the first point cloud and the second point cloud to register the first input medical image and the second input medical image, the machine learning based point registration network comprising 1) one or more point embedding networks for generating embeddings based on the first input medical image and the second input medical image, 2) one or more attention modules for generating permutation invariant features based on the embeddings, and 3) a transform regressor network for generating the transformation based on the permutation invariant features; and

means for outputting the transformation.

8. The apparatus of claim 7 , wherein the means for determining a transformation for aligning the first point cloud and the second point cloud using a machine learning based point registration network based on a density of points in the first point cloud and the second point cloud to register the first input medical image and the second input medical image comprises:

means for determining a forward transformation to align the first point cloud to the second point cloud; and

means for determining a backward transformation to align the second point cloud to the first point cloud.

9. The apparatus of claim 7 , wherein the first point cloud and the second point cloud have a different number of points.

10. The apparatus of claim 7 , wherein the transformation is not determined based on a correspondence between points in the first point cloud and the second point cloud.

11. A non-transitory computer readable medium storing computer program instructions, the computer program instructions when executed by a processor cause the processor to perform operations comprising:

receiving a first input medical image in a first modality and a second input medical image in a second modality;

segmenting one or more objects of interest from the first input medical image to generate a first segmentation map;

segmenting one or more objects of interest from the second input medical image to generate a second segmentation map;

extracting a first point cloud from the first segmentation map;

extracting a second point cloud from the second segmentation map;

determining a transformation for aligning the first point cloud and the second point cloud using a machine learning based point registration network based on a density of points in the first point cloud and the second point cloud to register the first input medical image and the second input medical image, the machine learning based point registration network comprising 1) one or more point embedding networks for generating embeddings based on the first input medical image and the second input medical image, 2) one or more attention modules for generating permutation invariant features based on the embeddings, and 3) a transform regressor network for generating the transformation based on the permutation invariant features; and

outputting the transformation.

12. The non-transitory computer readable medium of claim 11 , wherein determining a transformation for aligning the first point cloud and the second point cloud using a machine learning based point registration network based on a density of points in the first point cloud and the second point cloud to register the first input medical image and the second input medical image comprises:

determining a forward transformation to align the first point cloud to the second point cloud; and

determining a backward transformation to align the second point cloud to the first point cloud.

13. The non-transitory computer readable medium of claim 11 , wherein the first point cloud and the second point cloud have a different number of points.

14. The non-transitory computer readable medium of claim 11 , wherein the first modality is ultrasound and the second modality is magnetic resonance.

15. The non-transitory computer readable medium of claim 11 , wherein the one or more objects of interest segmented from the first input medical image and the one or more objects of interest segmented from the second input medical image comprise a prostate of a patient.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 20, 2023
From: SIEMENS HEALTHCARE GMBH
To: SIEMENS HEALTHINEERS AG
Reel/Frame 066267/0346 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 13, 2022
From: SIEMENS MEDICAL SOLUTIONS USA, INC.
To: SIEMENS HEALTHCARE GMBH
Reel/Frame 059897/0146 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 10, 2022
From: REAUNGAMORNRAT, SUREERAT; DIALLO, MAMADOU; KAMEN, ALI
To: SIEMENS MEDICAL SOLUTIONS USA, INC.
Reel/Frame 059226/0813 →
Continuity (1)
Related Publication 20230289984A1 · Sep 14, 2023
References Cited (41)
US 20070103460A1 · Zhang · 2007 [cited by examiner]
US 20130108127A1 · Boettger · 2013 [cited by examiner]
CN 112802073A · 2021 [cited by examiner]
CN 109949349B · 2021 [cited by examiner]
WO WO2022257345A1 · 2022 [cited by examiner]
Minghan Zhu, Correspondence-Free Point Cloud Registration with SO(3)-Equivariant Implicit Shape Representations, 5th Conference on Robot Learning (CoRL 2021), London, UK (Year: 2021). [cited by examiner]
Cornud et al., “TRUS-MRI image registration: a paradigm shift in the diagnosis of significant prostate cancer,” 2013, Abdom Imaging No. 38, pp. 1447-1463. [cited by applicant]
Kongnyuy et al., “Magnetic Resonance Imaging-Ultrasound Fusion-Guided Prostate Biopsy: Review of Technology, Techniques, and Outcomes,” 2016, Current Urology Reports No. 17, Article No. 32, 9 pgs. [cited by applicant]
Ward, “MRI-US Image Fusion Prostate Biopsy,” 2019, Accessed Feb. 2022, https://grandroundsinurology.com/mri-us-Image-fusion-prostate-biopsy/, 4 pgs. [cited by applicant]
Marks, “MRI/US Fusion Biopsy,” 2019, Accessed Feb. 2022. https://grandroundsinurology.com/mri-us-fusion-biopsy/, 6 pgs. [cited by applicant]
Sountoulides et al., “Micro-Ultrasound-Guided vs Multiparametric Magnetic Resonance Imaging-Targeted Biopsy in the Detection of Prostate Cancer: A Systematic Review and Meta-Analysis,” 2021, The Journal of Urology vol. … [cited by applicant]
Klotz, “Can high resolution micro-ultrasound replace MRI in the diagnosis of prostate cancer?,” 2020, European Urology Focus vol. 6, Issue 2, 15,pp. 419-423. [cited by applicant]
Iommi, “3D ultrasound guided navigation system with hybrid image fusion,” 2021, Scientific Reports vol. 11, Article No. 8838, 10 pgs. [cited by applicant]
Fu et al., “Biomechanically constrained non-rigid MR-TRUS prostate registration using deep learning based 3D point cloud matching,” 2021, Medical Image Analysis vol. 67, 30 pgs. [cited by applicant]
Sokolakis et al., “Usability and diagnostic accuracy of different MRI/ultrasound-guided fusion biopsy systems for the detection of clinically significant and insignificant prostate cancer: a prospective cohort study,” 2… [cited by applicant]
Liang et al., “PolyTransform: Deep Polygon Transformer for Instance Segmentation,” 2020, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9131-9140. [cited by applicant]
Li et al., “H-DenseUNet: Hybrid Densely Connected UNet for Liver and Tumor Segmentation From CT Volumes,” 2018, IEEE Transactions on Medical Imaging, vol. 37, Issue: 12, pp. 2663-2674. [cited by applicant]
Abdollahi et al., “VNet: An End-to-End Fully Convolutional Neural Network for Road Extraction From High-Resolution Remote Sensing Data,” 2020; IEEE Access. Vol. 8, pp. 179424-179436. [cited by applicant]
Caron et al., “Emerging Properties in Self-Supervised Vision Transformers,” 2021, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650-9660. [cited by applicant]
El-Nouby et al., “XCIT: Cross-Covariance Image Transformers,” 2021,35th Conference on Neural Information Processing Systems, 14 pgs. [cited by applicant]
Zbontar et al., “Barlow Twins: Self-Supervised Learning via Redundancy Reduction,” 2021, Proceedings of the 38th International Conference on Machine Learning, PMLR 139, pp. 12310-12320. [cited by applicant]
Zbontar et al., Supplementary PDF—“Barlow Twins: Self-Supervised Learning via Redundancy Reduction,” 2021, Proceedings of the 38th International Conference on Machine Learning, PMLR 139, 2 pgs. [cited by applicant]
He et al., “Momentum Contrast for Unsupervised Visual Representation Learning,” 2020, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729-9738. [cited by applicant]
Park et al., “Contrastive Learning for Unpaired Image-to-Image Translation,” 2020, Computer Vision—ECCV 2020, Lecture Notes in Computer Science, vol. 12354, pp. 319-345. [cited by applicant]
In et al., “Focal Loss for Dense Object Detection,” 2017, Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2980-2988. [cited by applicant]
Perez, “FILM: Visual Reasoning with a General Conditioning Layer,” 2018, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, No. 1, pp. 3942-3951. [cited by applicant]
Zhu et al., “SEAN: Image Synthesis With Semantic Region-Adaptive Normalization,” 2020, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5104-5113. [cited by applicant]
Myronenko et al., “Point Set Registration: Coherent Point Drift,” 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, No. 12, pp. 2262-2275. [cited by applicant]
Reaungamornrat et al., “Constrained Multiscale Registration of Non-Correspondent Incomplete Multi-Labeled Point Sets,” 2021, Prior Art Journal 2021 No. 21, 7 pgs. [cited by applicant]
Reaungamornrat et al., “Multi-Scale Frequency-Disentanglement Network for Smooth Shape-Constrained Multi-Organ Segmentation,” 2021, Prior Art Journal 2021 No. 15, 4 pgs. [cited by applicant]
Schroeder et al., “Flying edges: A high-performance scalable isocontouring algorithm,” 2015, IEEE 5th Symposium on Large Data Analysis and Visualization, pp. 33-40. [cited by applicant]
Qi et al., “PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space,” 2017, Advances in Neural Information Processing Systems 30 (NIPS 2017), 10 pgs. [cited by applicant]
Guo et al., “PCT: Point cloud transformer,” 2021, Computational Visual Media, vol. 7, No. 2, pp. 187-199. [cited by applicant]
Zhao et al., “Point Transformer,” 2021, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16259-16268. [cited by applicant]
Xiang et al., “SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution With Skip-Transformer,” 2021, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5499-5509. [cited by applicant]
Yu et al., “PoinTr: Diverse Point Cloud Completion With Geometry-Aware Transformers,” 2021, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12498-12507. [cited by applicant]
Seferbekov et al., “Feature Pyramid Network for Multi-Class Land Segmentation,” 2018, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 272-275. [cited by applicant]
Zhao et al., “M2Det: A Single-Shot Object Detector Based on Multi-Level Feature Pyramid Network,” 2019, Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), pp. 9259-9266. [cited by applicant]
Christ et al., “Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields,” 2016, Medical Image Computing and Computer-Assisted Intervention, pp. 4… [cited by applicant]
Zhang et al., “A Sparse-View CT Reconstruction Method Based on Combination of DenseNet and Deconvolution,” 2018, IEEE Transactions on Medical Imaging, vol. 37, No. 6, pp. 1407-1417. [cited by applicant]
Guo et al., “Underwater Image Enhancement Using a Multiscale Dense Generative Adversarial Network,” 2020, IEEE Journal of Oceanic Engineering, vol. 45, No. 3, pp. 862-870. [cited by applicant]