Method and system for machine learning classification based on structure or material segmentation in an image

1. A computer-implemented method for training a classification machine learning model for classifying a structure or material in an image of a subject, the method comprising:

dividing annotated segmentation maps of one or more images and annotated non-image data pertaining to the subject into a training set and a testing set, the annotated segmentation maps generated from one or more segmentations formed by segmenting each of the one or more images into one or more segmentations that correspond to respective structures or materials in the respective image;

(a) implementing a classification machine learning model for classifying a structure or material in an image of a subject, including initializing parameters of the classification machine learning model;

(b) updating the parameters of the classification machine learning model by running a learning algorithm on the training data;

(c) testing the classification machine learning model on the testing set;

(d) evaluating whether the classification machine learning model has satisfactory performance;

repeating steps (a) to (d) when the performance is found in step (d) not to be satisfactory; and

outputting the classification machine learning model for deployment as a trained classification machine learning model, or flagging the classification machine learning model as a trained classification machine learning model;

wherein each of the training and testing sets includes at least one of the annotated segmentation maps of the image and at least some annotated non-image data; and

each of the annotated segmentation maps corresponding to a respective image represents the image.

2. A method as claimed in claim 1 , including, when the performance is found in step (d) not to be satisfactory, receiving and using more image and non-image data for training the classification machine learning model.

3. A method as claimed in claim 1 , wherein the classification machine learning model comprises a neural network having a plurality of layers comprising artificial neurons, wherein the parameters comprise layer numbers, neuron numbers, neuron weights, and neuron function parameters.

4. A method as claimed in claim 1 , including dividing the annotated segmentation maps and the annotated non-image data into the training set, a development set and the testing set, and using the development set to investigate the learning procedure and to tune the parameters.

5. A method as claimed in claim 1 , including segmenting each of the one or more images into the one or more segmentations, wherein generating the annotated segmentation maps comprises:

i) generating structure segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of structure categories,

ii) generating material segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of material categories,

iii) generating structure segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of structure categories with a structure segmentation machine learning model, and/or

iv) generating material segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of material categories with a material segmentation machine learning model.

6. A method as claimed in claim 1 , further comprising down-sampling or otherwise processing the segmentation maps.

7. A method as claimed in claim 1 , including segmenting each of the one or more images into the one or more segmentations, wherein generating the annotated segmentation maps comprises:

generating abnormality segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of abnormality or normality categories, or

generating abnormality segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of abnormality or normality categories with an abnormality segmentation model.

8. A method as claimed in claim 1 , including segmenting each of the one or more images into the one or more segmentations, wherein generating the annotated segmentation maps comprises:

i) generating structure segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of structure categories, and generating material segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of material categories, or

ii) generating structure segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of structure categories with a structure segmentation machine learning model, and generating material segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of material categories with a material segmentation machine learning model.

9. A system for classifying a structure or material in an image of a subject, comprising:

a segmenter configured to segment an image into one or more segmentations that correspond to respective structures or materials in the image, and to generate from the segmentations one or more segmentation maps of the image including categorizations of pixels or voxels of the segmentation maps assigned from one or more respective predefined sets of categories;

a classifier that implements a trained classification machine learning model configured to generate, based on the segmentations maps, one or more classifications and to assign to the classifications respective scores indicative of a likelihood that the structure or material, or the subject, falls into the respective classifications; and

an output for outputting a result indicative of the classifications and scores;

wherein the trained classification machine learning model is a model trained according to the method of claim 1 .

10. A classification machine learning model trained according to the method of claim 1 .

11. A computer-implemented method for classifying a structure or material in an image of a subject, comprising:

segmenting an image into one or more segmentations that correspond to respective structures or materials in the image;

generating from the segmentations one or more segmentation maps of the image including categorizations of pixels or voxels of the segmentation maps assigned from respective predefined sets of categories of the structure or material;

using a trained classification machine learning model to generate, based on the segmentations maps, one or more classifications and to assign to the classifications respective scores indicative of a likelihood that the structure or material, or the subject, falls into the respective classifications; and

outputting a result indicative of the classifications and scores;

wherein the trained classification machine learning model is a model trained according to the method of claim 1 .

12. A system for training a classification machine learning model for classifying a structure or material in an image of a subject, the system comprising a processor configured to:

divide annotated segmentation maps of one or more images and annotated non-image data pertaining to the subject into a training set and a testing set, the annotated segmentation maps obtained by segmenting each of the one or more images into one or more segmentations that correspond to respective structures or materials in the respective image;

(a) implement a classification machine learning model for classifying a structure or material in an image of a subject, including initializing parameters of the classification machine learning model;

(b) update the parameters of the classification machine learning model by running a learning algorithm on the training data;

(c) test the classification machine learning model on the testing set;

(d) evaluate whether the classification machine learning model has satisfactory performance;

repeat steps (a) to (d) when the performance is found in step (d) not to be satisfactory; and

output the classification machine learning model for deployment as a trained classification machine learning model, or flag the classification machine learning model as a trained classification machine learning model;

wherein each of the training and testing sets includes at least one of the annotated segmentation maps of the image and at least some annotated non-image data; and

each of the annotated segmentation maps corresponding to a respective image represents the image.

13. A system as claimed in claim 12 , further comprising a segmenter configured to perform said segmenting of the one or more images, the segmenter comprising:

i) a structure segmenter configured to generate structure segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of structure categories,

ii) a material segmenter configured to generate material segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of material categories,

iii) a structure segmenter configured to generate structure segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of structure categories with a structure segmentation machine learning model, and/or

iv) a material segmenter configured to generate material segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of material categories with a material segmentation machine learning model.

14. A system as claimed in claim 12 , further comprising a segmentation map processor configured to down-sample or otherwise process the segmentation maps.

15. A system as claimed in claim 12 , further comprising a segmenter configured to perform said segmenting of the one or more images, the segmenter comprising:

an abnormality segmenter configured to generate abnormality segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of abnormality or normality categories, or

an abnormality segmenter configured to generate abnormality segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of abnormality or normality categories with an abnormality segmentation model.

16. A system as claimed in claim 12 , further comprising a segmenter configured to perform said segmenting of the one or more images, the segmenter comprising:

i) a structure segmenter configured to generate structure segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of structure categories, and a material segmenter configured to generate material segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of material categories; or

ii) a structure segmenter configured to generate structure segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of structure categories with a structure segmentation machine learning model, and a material segmenter configured to generate material segmentation maps of the respective image including categorizations of the pixels or voxels assigned from a predefined set of material categories with a material segmentation machine learning model.