Granted Patent B2

US 12,400,762 · App. 18/333,795 · Granted Aug 26, 2025

Inventors: James Otis Hare, II (Singapore, SG); Su Ping Carolyn Lam (Singapore, SG); Yoran Hummel (Singapore, SG); Matthew Frost (Singapore, SG); Mathias Iversen (Singapore, SG); Sze Chi Lim (Singapore, SG); Weile Wayne Tee (Singapore, SG)

Assignee: EKO.AI PTE. LTD.

G16H50/20A61B8/14A61B8/488G06T7/0012G06T7/11G06T2207/20084G06T2207/30048

Abstract

Claims (63)

1. A computer-implemented method for automated diagnosis of cardiac amyloidosis (CA) and hypertrophic cardiomyopathy (HCM) performed by an automated workflow engine executed by at least one processor, the method comprising:

receiving, from a memory, a plurality of echocardiogram images a heart;

separating the plurality of echocardiogram (echo) images according to 2D images and Doppler modality images;

classifying the 2D images by view type, including apical 4-chamber (A4C) video;

segmenting, by a 2D convolutional neural networks (CNN), a left ventricle of the heart in image frames of the A4C video to produce segmented A4C images having a segmentation mask over the left ventricle;

performing automatic phase detection on the segmented A4C images to determine systole endpoints and diastole endpoints per cardiac cycle in the A4C video, wherein the images frames of the A4C video between the systole and diastole endpoints comprise beat-to-beat video images;

performing, by a 3D CNN, CA and HCM disease classification by receiving the beat-to-beat images for respective cardiac cycles, and outputting cardiac cycle probability scores per the respective cardiac cycles for at least one of three cardiac cycle outcomes: i) CA, ii) HCM, and iii) No CA or HCM;

aggregating the cardiac cycle probability scores generated for the respective cardiac cycles to generate a A4C video-level conclusion for at least one of three disease outcomes: i) CA, ii) HCM, and iii) No CA or HCM;

responsive to a patient study including a plurality of A4C videos, combining A4C video-level conclusions from the plurality of A4C videos to generate a patient-level conclusion for the at least one of three disease outcomes; and

outputting a report showing the patient-level conclusion for the at least one of three disease outcomes.

2. The method of claim 1 , wherein receiving, from a memory, the plurality of echo images, further comprises:

receiving, by the processor, the plurality of echo images directly from a local or remote source, including an ultrasound device;

storing the plurality of echo images in an image archive; and

opening the stored echo images in the memory for processing.

3. The method of claim 1 , wherein separating the echo images further comprises:

analyzing metadata incorporated in the echo images to distinguish between the 2D and the Doppler modality images;

separating the Doppler modality images into either pulse wave, continuous wave, PWTDI or m-mode groupings;

performing color flow analysis on extracted pixel data using a combination of the metadata and color content within the echo images to separate views that contain color from those that do not;

removing from the echo images any metatags that contain personal information and cropping the echo images to exclude any identifying information; and

extracting pixel data from the echo images and converting the pixel data to numpy arrays for further processing.

4. The method of claim 1 , wherein classifying the 2D images and the Doppler modality images is based on a majority voting scheme comprising:

dividing a video of a 2D image of a Doppler modality image into frames;

generating for the frames, classification labels that constitute votes; and

applying the classification label receiving a highest number of the votes as the classification of the video.

5. The method of claim 1 , further comprising: implementing the workflow engine to comprise a first set of one or more classification CNNs for view classification, a second set of one or more segmentation CNNs for chamber segmentation and waveform mask/trace, a third set of one or more prediction CNNs for disease prediction.

6. The method of claim 5 , wherein the one or more segmentation CNNs are trained from hand-labeled real images or artificial images generated by general adversarial networks (GANs).

7. The method of claim 1 , further comprising: for all non-filtered out data, selecting as best measurement data the measurements associated with cardiac chambers with largest volumes; and saving with the best measurement data, image location, classification, annotation and other measurement data associated with the best measurement data.

8. A system, comprising:

a memory storing a plurality of echocardiogram images taken by an ultrasound device of a heart;

at least one processor coupled to the memory; and

a workflow engine, which when executed by the at least one processor is configurable to:

receive, from the memory, a plurality of echocardiogram images of the heart;

separate the plurality of echocardiogram (echo) images according to 2D images and Doppler modality images;

classify the 2D images by view type, including apical 4-chamber (A4C) video;

segment, by a 2D CNN, a left ventricle of the heart in image frames of the A4C video to produce segmented A4C images having a segmentation mask over the left ventricle;

perform automatic phase detection on the segmented A4C images to determine systole endpoints and diastole endpoints per cardiac cycle in the A4C video, wherein the images frames of the A4C video between the systole and diastole endpoints comprise beat-to-beat video images;

perform, by a 3D CNN, CA and HCM disease classification by receiving the beat-to-beat images for respective cardiac cycles, and outputting cardiac cycle probability scores per the respective cardiac cycles for at least one of three cardiac cycle outcomes: i) CA, ii) HCM, and iii) No CA or HCM;

aggregate the cardiac cycle probability scores generated for the respective cardiac cycles to generate a A4C video-level conclusion for at least one of three disease outcomes: i) CA, ii) HCM, and iii) No CA or HCM;

responsive to a patient study including a plurality of A4C videos, combine the A4C video-level conclusions from the plurality of A4C videos to generate a patient-level conclusion for the at least one of three disease outcomes; and

output a report showing the patient-level conclusion for at least one of three disease outcomes.

9. The system of claim 8 , wherein the workflow engine receives from the plurality of echo images directly from a local or remote source, including an ultrasound device;

stores the plurality of echo images in an image archive; and

opens the stored echo images in the memory for processing.

10. The system of claim 8 , wherein the workflow engine separates the echo images uses metadata incorporated in the echo images to distinguish between the 2D and the Doppler modality images;

separates the Doppler modality images into either pulse wave, continuous wave, PWTDI or m-mode groupings;

performs color flow analysis on extracted pixel data using a combination of the metadata and color content within the echo images to separate views that contain color from those that do not;

removes from the echo images any metatags that contain personal information and cropping the echo images to exclude any identifying information; and

extracts pixel data from the echo images and converts the pixel data to numpy arrays for further processing.

11. The system of claim 8 , wherein the workflow engine classifies the 2D images and the Doppler modality images based on a majority voting scheme, wherein workflow engine divides a video of a 2D image of a Doppler modality image into frames;

generates for the frames, classification labels that constitute votes; and

applies the classification label receiving a highest number of the votes as the classification of the video.

12. The system of claim 8 , wherein the workflow engine is implemented to comprise a first set of one or more classification CNNs for view classification, a second set of one or more segmentation CNNs for chamber segmentation and waveform mask/trace, a third set of one or more prediction CNNs for disease prediction.

13. The system of claim 12 , wherein the one or more segmentation CNNs are trained from hand-labeled real images or artificial images generated by general adversarial networks (GANs).

14. An executable software product stored on a non-transitory computer-readable medium containing program instructions for implementing an automated workflow for diagnosis of cardiac amyloidosis (CA) and hypertrophic cardiomyopathy (HCM), which when executed by a set of one or more processors, are configurable to cause the set of one processors to perform operations comprising:

receiving, from a memory, a patient study comprising a plurality of echocardiogram images taken by an ultrasound device of a heart;

separating, by a filter, the plurality of echocardiogram (echo) images according to 2D images and Doppler modality images based on analyzing image metadata;

classifying the 2D images by view type, including apical 4-chamber (A4C) video;

segmenting, by a 2D CNN, a left ventricle of the heart in image frames of the A4C video to produce segmented A4C images having a segmentation mask over the left ventricle;

performing automatic phase detection on the segmented A4C images to determine systole endpoints and diastole endpoints per cardiac cycle in the A4C video, wherein the images frames of the A4C video between the systole and diastole endpoints comprise beat-to-beat video images;

performing, by a 3D CNN, CA and HCM disease classification by receiving the beat-to-beat images for respective cardiac cycles, and outputting cardiac cycle probability scores per the respective cardiac cycles for at least one of three cardiac cycle outcomes: i) CA, ii) HCM, and iii) No CA or HCM;

aggregating the cardiac cycle probability scores generated for the cardiac cycles to generate a A4C video-level conclusion for at least one of three disease outcomes: i) CA, ii) HCM, and iii) No CA or HCM;

responsive to a patient study including a plurality of A4C videos, combining the A4C video-level conclusions from the plurality of A4C videos to generate a patient-level conclusion for the at least one of three disease outcomes; and

outputting a report showing the patient-level conclusion for at least one of three disease outcomes.

Assignments (1)

Continuity (3)

References Cited (218)