Interactive training of a machine learning model for tissue segmentation

1. A method performed by one or more data processing apparatus for training a machine learning model to segment magnified images of tissue samples, wherein the method comprises repeatedly performing operations including:

obtaining a magnified image of a tissue sample;

processing an input comprising: the image, features derived from the image, or both, in accordance with current values of model parameters of a machine learning model to generate an automatic segmentation of the image into a plurality of tissue classes, wherein the automatic segmentation specifies a respective tissue class corresponding to each pixel of the image;

providing, to a user through a user interface, an indication of: (i) the image, and (ii) the automatic segmentation of the image;

obtaining, from the user through the user interface, an input specifying one or more modifications to the automatic segmentation of the image;

determining an edited segmentation of the image, comprising applying the modifications specified by the user to the automatic segmentation of the image; and

determining updated values of the model parameters of the machine learning model based the edited segmentation of the image, comprising:

determining gradients of an objective function that characterizes a similarity between: (i) the automatic segmentation of the image, and (ii) the edited segmentation of the image; and

adjusting the values of the model parameters using the gradients.

2. The method of claim 1 , wherein the plurality of tissue classes comprise cancerous tissue and non-cancerous tissue.

3. The method of claim 1 , wherein the image is a whole slide image (WSI) captured using a microscope.

4. The method of claim 1 , wherein providing, to the user through the user interface, an indication of: (i) the image, and (ii) the automatic segmentation of the image comprises:

providing, to the user through the user interface, a visualization that depicts the automatic segmentation overlaid on the image.

5. The method of claim 4 , wherein the visualization that depicts the automatic segmentation overlaid on the image indicates the respective tissue class corresponding to each pixel of the image.

6. The method of claim 1 , wherein:

each modification to the automatic segmentation of the image indicates, for one or more pixels of the image, a change to the respective tissue class specified for the pixel by the automatic segmentation; and

applying the modifications specified by the user to the automatic segmentation of the image comprises changing the tissue classes corresponding to one or more pixels of the image in accordance with the modifications specified by the user.

7. The method of claim 1 , further comprising determining an expertise score that characterizes a predicted skill of the user in reviewing and editing tissue segmentations.

8. A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to repeatedly perform operations including:

obtaining a magnified image of a tissue sample;

processing an input comprising: the image, features derived from the image, or both, in accordance with current values of model parameters of a machine learning model to generate an automatic segmentation of the image into a plurality of tissue classes, wherein the automatic segmentation specifies a respective tissue class corresponding to each pixel of the image;

providing, to a user through a user interface, an indication of: (i) the image, and (ii) the automatic segmentation of the image;

obtaining, from the user through the user interface, an input specifying one or more modifications to the automatic segmentation of the image;

determining an edited segmentation of the image, comprising applying the modifications specified by the user to the automatic segmentation of the image; and

determining updated values of the model parameters of the machine learning model based the edited segmentation of the image, comprising:

determining gradients of an objective function that characterizes a similarity between: (i) the automatic segmentation of the image, and (ii) the edited segmentation of the image; and

adjusting the values of the model parameters using the gradients.

9. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to repeatedly perform operations comprising:

obtaining a magnified image of a tissue sample;

processing an input comprising: the image, features derived from the image, or both, in accordance with current values of model parameters of a machine learning model to generate an automatic segmentation of the image into a plurality of tissue classes, wherein the automatic segmentation specifies a respective tissue class corresponding to each pixel of the image;

providing, to a user through a user interface, an indication of: (i) the image, and (ii) the automatic segmentation of the image;

obtaining, from the user through the user interface, an input specifying one or more modifications to the automatic segmentation of the image;

determining an edited segmentation of the image, comprising applying the modifications specified by the user to the automatic segmentation of the image; and

determining updated values of the model parameters of the machine learning model based the edited segmentation of the image, comprising:

determining gradients of an objective function that characterizes a similarity between: (i) the automatic segmentation of the image, and (ii) the edited segmentation of the image; and

adjusting the values of the model parameters using the gradients.

10. The computer storage media of claim 9 , wherein the machine learning model comprises a neural network model.

11. The computer storage media of claim 10 , wherein the neural network model is a convolutional neural network model comprising one or more convolutional neural network layers.

12. The computer storage media of claim 10 , wherein the model parameters specify weight values of each layer of the neural network model.

13. The computer storage media of claim 9 , wherein the machine learning model comprises a random forest model.

14. The computer storage media of claim 13 , wherein the model parameters specify splitting functions used at nodes of decision trees of the random forest model.

15. The computer storage media of claim 9 , wherein providing, to the user through the user interface, an indication of: (i) the image, and (ii) the automatic segmentation of the image comprises:

providing, to the user through the user interface, a visualization that depicts the automatic segmentation overlaid on the image.

16. The computer storage media of claim 15 , wherein the visualization that depicts the automatic segmentation overlaid on the image indicates the respective tissue class corresponding to each pixel of the image.

17. The computer storage media of claim 9 , wherein:

each modification to the automatic segmentation of the image indicates, for one or more pixels of the image, a change to the respective tissue class specified for the pixel by the automatic segmentation; and

applying the modifications specified by the user to the automatic segmentation of the image comprises changing the tissue classes corresponding to one or more pixels of the image in accordance with the modifications specified by the user.

18. The computer storage media of claim 9 , further comprising determining an expertise score that characterizes a predicted skill of the user in reviewing and editing tissue segmentations.

19. A method performed by one or more data processing apparatus for training a machine learning model to segment magnified images of tissue samples, wherein the method comprises:

repeatedly performing operations including:

obtaining a magnified image of a tissue sample;

processing an input comprising: the image, features derived from the image, or both, in accordance with current values of model parameters of a machine learning model to generate an automatic segmentation of the image into a plurality of tissue classes, wherein the automatic segmentation specifies a respective tissue class corresponding to each pixel of the image;

providing, to a user through a user interface, an indication of: (i) the image, and (ii) the automatic segmentation of the image;

obtaining, from the user through the user interface, an input specifying one or more modifications to the automatic segmentation of the image;

determining an edited segmentation of the image, comprising applying the modifications specified by the user to the automatic segmentation of the image; and

determining updated values of the model parameters of the machine learning model based the edited segmentation of the image; and

determining an expertise score that characterizes a predicted skill of the user in reviewing and editing tissue segmentations.

20. A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

repeatedly performing operations including:

obtaining a magnified image of a tissue sample;

processing an input comprising: the image, features derived from the image, or both, in accordance with current values of model parameters of a machine learning model to generate an automatic segmentation of the image into a plurality of tissue classes, wherein the automatic segmentation specifies a respective tissue class corresponding to each pixel of the image;

providing, to a user through a user interface, an indication of: (i) the image, and (ii) the automatic segmentation of the image;

obtaining, from the user through the user interface, an input specifying one or more modifications to the automatic segmentation of the image;

determining an edited segmentation of the image, comprising applying the modifications specified by the user to the automatic segmentation of the image; and

determining updated values of the model parameters of the machine learning model based the edited segmentation of the image; and

determining an expertise score that characterizes a predicted skill of the user in reviewing and editing tissue segmentations.

21. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to repeatedly perform operations comprising:

repeatedly performing operations including:

obtaining a magnified image of a tissue sample;

processing an input comprising: the image, features derived from the image, or both, in accordance with current values of model parameters of a machine learning model to generate an automatic segmentation of the image into a plurality of tissue classes, wherein the automatic segmentation specifies a respective tissue class corresponding to each pixel of the image;

providing, to a user through a user interface, an indication of: (i) the image, and (ii) the automatic segmentation of the image;

obtaining, from the user through the user interface, an input specifying one or more modifications to the automatic segmentation of the image;

determining an edited segmentation of the image, comprising applying the modifications specified by the user to the automatic segmentation of the image; and

determining updated values of the model parameters of the machine learning model based the edited segmentation of the image; and

determining an expertise score that characterizes a predicted skill of the user in reviewing and editing tissue segmentations.