Shift invariant loss for deep learning based image segmentation

1. A method comprising:

receiving a first image of an unstained tissue sample;

receiving a second image of the unstained tissue sample;

generating, by a trained neural network, a first set of labels corresponding to first tissue features within the first image, and a second set of labels corresponding to second tissue features within the second image;

determining, by the trained neural network, an alignment result between the first and second images based at least in part on at least one of the first and second sets of labels;

generating, by the trained neural network, an aligned first image on the alignment result; and

applying a virtual stain to the aligned first image.

2. The method of claim 1 , wherein the trained neural network was trained by:

identifying an input data set;

identifying a label data set that identifies one or more first parts of the input data set that correspond to a particular label;

processing the input data set using a neural network to generate predicted label data set that identifies one or more second parts of the input data set predicted to correspond to the particular label;

determining an alignment result using the predicted label data set and the label data set;

determining, based on the alignment result, a transformation that results in a shift, rotation, scaling, and/or deformation of the one or more first parts of the input data set relative to the one or more second parts of the input data set;

computing a loss score using the transformation, label data and the predicted label data set; and

updating the neural network based on the loss score to generate the trained neural network.

3. The method of claim 1 , wherein generating the aligned first image comprises shifting, rotating, or scaling the first tissue features.

4. The method of claim 1 , wherein determining an alignment result comprises determining, using a warping neural network, a warping or deformation of at least one of the first or second images.

5. The method of claim 4 , wherein generating the aligned first image comprises warping the first tissue features based on the determined warping or deformation.

6. The method of claim 1 , further comprising:

dividing, by the trained neural network, the first image into a plurality of first image patches;

dividing, by the trained neural network, the second image into a plurality of second image patches; and

wherein:

generating the first and second sets of labels is based on the plurality of first image patches and the plurality of second image patches, respectively, and

generating the first and second sets of dense predictions is based on the plurality of first image patches and the plurality of second image patches, respectively.

7. The method of claim 1 , further comprising:

generating, by the trained neural network, a first set of dense predictions of third tissue features within the first image and a second set of dense predictions of fourth tissue features within the second image; and

wherein determining the alignment result is further based on the first and second sets of dense predictions.

8. A system comprising:

a non-transitory computer-readable medium; and

one or more processors communicatively coupled to the non-transitory computer-readable medium, the one or more processors configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to:

receive a first image of an unstained tissue sample;

receive a second image of the unstained tissue sample;

generate, by a trained neural network, a first set of labels corresponding to first tissue features within the first image, and a second set of labels corresponding to second tissue features within the second image;

determine, by the trained neural network, an alignment result between the first and second images based at least in part on at least one of the first and second sets of labels;

generate, by the trained neural network, an aligned first image on the alignment result; and

apply a virtual stain to the aligned first image.

9. The system of claim 8 , wherein the trained neural network was trained by:

identifying an input data set;

identifying a label data set that identifies one or more first parts of the input data set that correspond to a particular label;

processing the input data set using a neural network to generate predicted label data set that identifies one or more second parts of the input data set predicted to correspond to the particular label;

determining an alignment result using the predicted label data set and the label data set;

determining, based on the alignment result, a transformation that results in a shift, rotation, scaling, and/or deformation of the one or more first parts of the input data set relative to the one or more second parts of the input data set;

computing a loss score using the transformation, label data and the predicted label data set; and

updating the neural network based on the loss score to generate the trained neural network.

10. The system of claim 8 , wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to generate the aligned first image based on shifting, rotating, or scaling the first tissue features.

11. The system of claim 8 , wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to determine, using a warping neural network, a warping or deformation of at least one of the first or second images.

12. The system of claim 8 , wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to generate the aligned first image based on warping the first tissue features based on the determined warping or deformation.

13. The system of claim 8 , wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:

divide, by the trained neural network, the first image into a plurality of first image patches;

divide, by the trained neural network, the second image into a plurality of second image patches;

generate the first and second sets of labels based on the plurality of first image patches and the plurality of second image patches, respectively, and

generate the first and second sets of dense predictions based on the plurality of first image patches and the plurality of second image patches, respectively.

14. The system of claim 8 , wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:

generate, by the trained neural network, a first set of dense predictions of third tissue features within the first image and a second set of dense predictions of fourth tissue features within the second image; and

determine the alignment result further based on the first and second sets of dense predictions.

15. A non-transitory computer-readable medium comprising processor-executable instructions stored in the non-transitory computer-readable medium configured to cause one or more processors to:

receive a first image of an unstained tissue sample;

receive a second image of the unstained tissue sample;

generate, by a trained neural network, a first set of labels corresponding to first tissue features within the first image, and a second set of labels corresponding to second tissue features within the second image;

determine, by the trained neural network, an alignment result between the first and second images based at least in part on at least one of the first and second sets of labels;

generate, by the trained neural network, an aligned first image on the alignment result; and

apply a virtual stain to the aligned first image.

16. The non-transitory computer-readable medium of claim 15 , wherein the trained neural network was trained by:

identifying an input data set;

identifying a label data set that identifies one or more first parts of the input data set that correspond to a particular label;

processing the input data set using a neural network to generate predicted label data set that identifies one or more second parts of the input data set predicted to correspond to the particular label;

determining an alignment result using the predicted label data set and the label data set;

determining, based on the alignment result, a transformation that results in a shift, rotation, scaling, and/or deformation of the one or more first parts of the input data set relative to the one or more second parts of the input data set;

computing a loss score using the transformation, label data and the predicted label data set; and

updating the neural network based on the loss score to generate the trained neural network.

17. The non-transitory computer-readable medium of claim 15 , further comprising processor-executable instructions stored in the non-transitory computer-readable medium configured to cause the one or more processors to generate the aligned first image based on shifting, rotating, or scaling the first tissue features.

18. The non-transitory computer-readable medium of claim 15 , further comprising processor-executable instructions stored in the non-transitory computer-readable medium configured to cause the one or more processors to:

determine, using a warping neural network, a warping or deformation of at least one of the first or second images; and

generate the aligned first image based on warping the first tissue features based on the determined warping or deformation.

19. The non-transitory computer-readable medium of claim 15 , further comprising processor-executable instructions stored in the non-transitory computer-readable medium configured to cause the one or more processors to generate the aligned first image based on warping the first tissue features based on the determined warping or deformation.

20. The non-transitory computer-readable medium of claim 15 , further comprising processor-executable instructions stored in the non-transitory computer-readable medium configured to cause the one or more processors to:

divide, by the trained neural network, the first image into a plurality of first image patches;

divide, by the trained neural network, the second image into a plurality of second image patches;

generate the first and second sets of labels based on the plurality of first image patches and the plurality of second image patches, respectively, and

generate the first and second sets of dense predictions based on the plurality of first image patches and the plurality of second image patches, respectively.

21. The non-transitory computer-readable medium of claim 15 , further comprising processor-executable instructions stored in the non-transitory computer-readable medium configured to cause the one or more processors to:

generate, by the trained neural network, a first set of dense predictions of third tissue features within the first image and a second set of dense predictions of fourth tissue features within the second image; and

determine the alignment result further based on the first and second sets of dense predictions.