System and method for machine-learning based sensor analysis and vascular tree segmentation
Methods for automated identification of vascular features are described. In some embodiments, one or more machine learning (ML)-based vascular classifiers are used, with their results being combined to with results of at least one other vascular classifier in order to produce the final results. Potentially advantages of this approach include the ability to combine certain strengths of ML classifiers with segmentation approaches based on more classical (“formula-based”) methods. These strengths may include particularly the identification of anatomically identified targets mixed within an image also showing similar looking but anatomically distinct targets.
1. A method implemented by a system of one or more computers, the method comprising:
accessing an image sequence comprising a plurality of vascular images, the vascular images depicting a portion of a heart from a particular viewpoint and the vascular images being associated with different times within a time range;
determining a subset of the vascular images which are associated with a particular cardiac phase, wherein determining comprises computing a forward pass through a machine learning model trained to classify vascular images based on cardiac phase;
generating segmentation masks associated with the subset, wherein the segmentation masks segment vessels included in the respective vascular images which form the subset, wherein mask scores are determined for the segmentation masks which are indicative of a size or length associated with a vessel included in a segmentation mask, and wherein the subset is filtered to remove one or more vascular images based on the mask scores; and
determining a particular vascular image included in the filtered subset, wherein the determination is based analyzing one or more image quality measures determined for each vascular image included in the filtered subset,
wherein the particular vascular image is configured for inclusion in an interactive user interface.
2. The method of claim 1 , wherein the vascular images are angiographic images.
3. The method of claim 1 , wherein determining a subset of cardiac phase images comprises:
determining whether the image sequence is associated with a left-side or a right-side of the heart,
wherein the machine learning model is selected from a plurality of machine learning models based on the determination.
4. The method of claim 1 , wherein the machine learning model is a convolutional neural network, and wherein determining the subset comprises:
obtaining, from the image sequence, a threshold number of the vascular images, wherein the images which form the threshold number are contiguous in the image sequence; and
providing the threshold number of the vascular images as input to the machine learning model, wherein the machine learning outputs a classification associated with a particular image in the threshold number.
5. The method of claim 1 , wherein the particular cardiac phase is end-diastolic.
6. The method of claim 1 , wherein the segmentation masks are generated based on a different machine learning model, wherein the different machine learning model is trained to segment vessels depicted in vascular images.
7. The method of claim 1 , wherein the image sequence is determined to be associated with a left-side or a right-side of the heart, and wherein for the right-side, a single segmentation mask is generated for each vascular image included in the subset.
8. The method of claim 1 , wherein the image sequence is determined to be associated with a left-side or a right-side of the heart, and wherein for the left-side, two or more segmentation masks are generated for each vascular image included in the subset.
9. The method of claim 1 , wherein a mask score is indicative of a length associated with a vessel or an area associated with the vessel.
10. The method of claim 1 , wherein determining a particular vascular image comprises:
registering the vascular images included in the filtered subset, wherein registering comprises aligning features depicted in the vascular images;
determining image quality measures comprising contrast scores for the vascular images included in the filtered subset, wherein a contrast score indicates an average grayscale value associated with vessels in the vascular images; and
determining the particular vascular image based on the contrast scores.
11. The method of claim 10 , wherein a unified segmentation mask is generated for the vascular images included in the filtered subset, and wherein the contrast scores compare contrast of same vessels included in the vascular images.
12. The method of claim 1 , wherein the interactive user interface is configured to enable selection of a different vascular image than the particular vascular image, and wherein the system is configured to update the machine learning model based on the selection.
13. A system comprising one or more computers and non-transitory computer storage media storing instructions that when executed by the one or more computers, cause the one or more computers to:
access an image sequence comprising a plurality of vascular images, the vascular images depicting a portion of a heart from a particular viewpoint and the vascular images being associated with different times within a time range;
determine a subset of the vascular images which are associated with a particular cardiac phase, wherein determining comprises computing a forward pass through a machine learning model trained to classify vascular images based on cardiac phase;
generate segmentation masks associated with the subset, wherein the segmentation masks segment vessels included in the respective vascular images which form the subset, wherein mask scores are determined for the segmentation masks which are indicative of a size or length associated with a vessel included in a segmentation mask, and wherein the subset is filtered to remove one or more vascular images based on the mask scores; and
determine a particular vascular image included in the filtered subset, wherein the determination is based analyzing one or more image quality measures determined for each vascular image included in the filtered subset,
wherein the particular vascular image is configured for inclusion in an interactive user interface.
14. The system of claim 13 , wherein to determine a subset of cardiac phase images, the instructions cause the one or more computers to:
determine whether the image sequence is associated with a left-side or a right-side of the heart,
wherein the machine learning model is selected from a plurality of machine learning models based on the determination.
15. The system of claim 13 , wherein the image sequence is determined to be associated with a left-side or a right-side of the heart,
wherein for the right-side, a single segmentation mask is generated for each vascular image included in the subset, or
wherein for the left-side, two or more segmentation masks are generated for each vascular image included in the subset.
16. The system of claim 13 , wherein to determine a particular vascular image, the instructions cause the one or more computers to:
register the vascular images included in the filtered subset, wherein registering comprises aligning features depicted in the vascular images;
determine image quality measures comprising contrast scores for the vascular images included in the filtered subset, wherein a contrast score indicates an average grayscale value associated with vessels in the vascular images; and
determine the particular vascular image based on the contrast scores.
17. Non-transitory computer storage media storing instructions that when executed by a system of one or more computers, cause the one or more computers to:
access an image sequence comprising a plurality of vascular images, the vascular images depicting a portion of a heart from a particular viewpoint and the vascular images being associated with different times within a time range;
determine a subset of the vascular images which are associated with a particular cardiac phase, wherein determining comprises computing a forward pass through a machine learning model trained to classify vascular images based on cardiac phase;
generate segmentation masks associated with the subset, wherein the segmentation masks segment vessels included in the respective vascular images which form the subset, wherein mask scores are determined for the segmentation masks which are indicative of a size or length associated with a vessel included in a segmentation mask, and wherein the subset is filtered to remove one or more vascular images based on the mask scores; and
determine a particular vascular image included in the filtered subset, wherein the determination is based analyzing one or more image quality measures determined for each vascular image included in the filtered subset,
wherein the particular vascular image is configured for inclusion in an interactive user interface.
18. The computer storage media of claim 17 , wherein to determine a subset of cardiac phase images, the instructions cause the one or more computers to:
determine whether the image sequence is associated with a left-side or a right-side of the heart,
wherein the machine learning model is selected from a plurality of machine learning models based on the determination.
19. The computer storage media of claim 17 , wherein the image sequence is determined to be associated with a left-side or a right-side of the heart,
wherein for the right-side, a single segmentation mask is generated for each vascular image included in the subset, or
wherein for the left-side, two or more segmentation masks are generated for each vascular image included in the subset.
20. The computer storage media of claim 17 , wherein to determine a particular vascular image, the instructions cause the one or more computers to:
register the vascular images included in the filtered subset, wherein registering comprises aligning features depicted in the vascular images;
determine image quality measures comprising contrast scores for the vascular images included in the filtered subset, wherein a contrast score indicates an average grayscale value associated with vessels in the vascular images; and
determine the particular vascular image based on the contrast scores.