Apparatus, method, and non-transitory computer-readable storage medium for combining real-number-based and complex-number-based images
The present disclosure relates to a real-number-based neural network operating in combination with a complex-number-based neural network to perform image processing (e.g., using phase-based medical images). In one embodiment, a method includes, but is not limited to, applying, to inputs of a first trained neural network trained to process real-number-based images, first image data generated from real-number-based measurements obtained by imaging a subject; applying, to inputs of a second trained neural network trained to process complex-number-based images, second image data generated from complex-number-based measurements obtained by imaging the subject; and combining a first output of the first trained neural network and a second output of the second trained neural network to produce a combined image, based on the first image data and the second image data.
1. An apparatus for performing image processing, comprising:
processing circuitry configured to
apply, to inputs of a first trained neural network trained to process real-number-based images, first image data generated from real-number-based measurements obtained by imaging a subject;
apply, to inputs of a second trained neural network trained to process complex-number-based images, second image data generated from complex-number-based measurements obtained by imaging the subject;
apply, to inputs of a third trained neural network trained to process complex-number-based images, third image data generated from complex-number-based measurements obtained by imaging the subject; and
combine a first output of the first trained neural network, a second output of the second trained neural network, and a third output of the third trained neural network to produce a combined image, based on the first image data, the second image data, and the third image data.
2. The apparatus according to claim 1 , wherein the first, second, and third trained neural networks are deep learning-based neural networks.
3. The apparatus according to claim 1 , wherein the processing circuitry is further configured to produce the combined image by calculating a weighted combination of the first, second, and third outputs.
4. The apparatus according to claim 1 , wherein the processing circuitry is further configured to produce the combined image by applying the first and second outputs to a fourth neural network trained to produce the combined image from the real-number-based measurements and from the complex-number-based measurements.
5. The apparatus according to claim 1 , wherein the processing circuitry is further configured to apply, to inputs of a fourth trained neural network trained to process at least one of real-number-based images and complex-number-based images, fourth image data from measurements obtained by imaging the subject, and
wherein the processing circuitry is further configured to produce the combined image by combining a fourth output of the fourth trained neural network with the first, second, and third outputs.
6. The apparatus according to claim 1 , further comprising the first trained neural network and the second trained neural network, wherein the first trained neural network comprises a magnitude-based neural network trained to process magnitude-based images, and the second trained neural network comprises a phase-based neural network trained to process phase-based images.
7. The apparatus according to claim 6 , wherein the phase-based neural network trained to process phase-based images comprises a phase-difference-based neural network trained to process phase-difference-based images.
8. The apparatus according to claim 1 , wherein the first image data has a first resolution, the second image data has a second resolution, and the combined image has a third resolution, and
wherein the third resolution is greater than at least one of the first and second resolutions.
9. The apparatus according to claim 1 , wherein the processing circuitry is further configured to generate a mask image from the first image data, and
wherein the second image data is generated by applying the mask image to third image data generated from the complex-number-based measurements obtained by imaging the subject.
10. The apparatus according to claim 1 , wherein the combined image generated by the processing circuitry has fewer artifacts than at least one of the first image data and the second image data.
11. The apparatus according to claim 1 , wherein the combined image generated by the processing circuitry has less noise than at least one of the first image data and the second image data.
12. The apparatus according to claim 1 , wherein the first image data and the second image data are dual-echo sequence data acquired by a magnetic resonance imaging device.
13. The apparatus according to claim 1 , wherein the first image data and the second image data are data acquired by an ultrasound probe.
14. An image processing method, comprising:
applying, to inputs of a first trained neural network trained to process real-number-based images, first image data generated from real-number-based measurements obtained by imaging a subject;
applying, to inputs of a second trained neural network trained to process complex-number-based images, second image data generated from complex-number-based measurements obtained by imaging the subject;
applying to inputs of a third trained neural network trained to process complex-number-based images, third image data generated from complex-number-based measurements obtained by imaging the subject; and
combining a first output of the first trained neural network, a second output of the second trained neural network, and a third output of the third trained neural network to produce a combined image, based on the first image data, the second image data, and the third image data.
15. A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a computer, cause the computer to perform an image processing method, comprising:
applying, to inputs of a first trained neural network trained to process real-number-based images, first image data generated from real-number-based measurements obtained by imaging a subject;
applying, to inputs of a second trained neural network, trained to process complex-number-based images, second image data generated from complex-number-based measurements obtained by imaging the subject;
applying, to inputs of a third trained neural network trained to process complex-number-based images, third image data generated from complex-number-based measurements obtained by imaging the subject; and
combining a first output of the first trained neural network, a second output of the second trained neural network, and a third output of the third trained neural network to produce a combined image, based on the first image data, the second image data, and the third image data.