System and method for generating a stained image
A system and method for generating a stained image including the steps of obtaining a first image of a key sample section; and processing the first image with a multi-modal stain learning engine arranged to generate at least one stained image, wherein the at least one stained image represents the key sample section stained with at least one stain.
1. A method for generating a stained image comprising the steps of:
obtaining a first image of a tissue sample section; and
processing the first image with a multi-modal stain learning engine arranged to virtually stain the first image with at least one stain to generate at least one generated stained image;
wherein the multi-modal stain learning engine comprises a machine learning network arranged to generate the at least one generated stained image; the machine learning network trained using a training image of each of at least three adjacent sample sections including a key sample section and adjacent sample sections obtained in proximity to the key sample section; the multi-modal stain learning engine further arranged to correct spatial mismatches among the training image representing the key sample section and the training images of the adjacent sample sections with an extrapolation process based on a predetermined section thickness and a predetermined ordering of the key sample section and the adjacent sample sections.
2. The method for generating a stained image in accordance with claim 1 , wherein the tissue sample section is physically unstained or physically stained with an input stain.
3. The method for generating a stained image in accordance with claim 1 , wherein one of the at least three adjacent sample sections is stained with an output stain; and wherein the remaining adjacent sample sections are all stained with an input stain.
4. The method for generating a stained image in accordance with claim 3 , wherein the at least one generated stained image represents the key sample section stained with the output stain.
5. The method for generating a stained image in accordance with claim 3 , wherein the remaining adjacent sample sections are stained with multiple types of the input stain in an alternating manner.
6. The method for generating a stained image in accordance with claim 1 , wherein the machine learning network includes a generator network arranged to generate the at least one generated stained image over a plurality of cycles and a discriminator network arranged to analyze the at least one generated stained image to provide feedback to the generator network on each of the plurality of cycles.
7. The method for generating a stained image in accordance with claim 6 , wherein the machine learning network is a generative adversarial network.
8. The method for generating a stained image in accordance with claim 7 , wherein the generative adversarial network is trained with the training images of the at least three adjacent sample sections.
9. The method for generating a stained image in accordance with claim 8 , wherein the generative adversarial network is further trained with the training images of the at least three adjacent sample sections imaged with different illumination conditions.
10. The method for generating a stained image in accordance with claim 9 , wherein the illumination conditions include brightfield and autofluorescence.
11. The method for generating a stained image in accordance with claim 3 , wherein each of the input stain and the output stain includes at least one of hematoxylin and eosin (H&E) and a special stain.
12. The method for generating a stained image in accordance with claim 11 , wherein the special stain includes immunostains.
13. The method for generating a stained image in accordance with claim 12 , wherein the at least three adjacent sample sections are frozen.
14. The method for generating a stained image in accordance with claim 13 , wherein the at least three adjacent sample sections are formalin-fixed and embedded in paraffin.
15. The method for generating a stained image in accordance with claim 1 , wherein the training image representing the key sample section is stained with an output stain and the training images of the adjacent sample sections are stained with an input stain.
16. The method for generating a stained image in accordance with claim 15 , wherein the multi-modal stain learning engine is further arranged to align all training images of the adjacent sample sections stained with the input stain to the training image representing the key sample section stained with the output stain using extrapolation.
17. The method for generating a stained image in accordance with claim 1 , wherein the extrapolation process further includes a step of applying an image extrapolation transformation to generate at least one estimated input image based on the training images of the adjacent sample sections stained with the input stain.
18. A system for generating a stained image comprising:
an image gateway comprising an input device arranged to obtain a first image of a tissue sample section; and
an image generator comprising a multi-modal stain learning engine which further comprises a machine learning network; the image generator arranged to process the first image with the multi-modal stain learning engine arranged to virtually stain the first image with at least one stain to generate at least one generated stained image;
wherein the multi-modal stain learning engine comprises a machine learning network arranged to generate the at least one generated stained image; the machine learning network trained using a training image of each of at least three adjacent sample sections including a key sample section and adjacent sample sections obtained in proximity to the key sample section; the multi-modal stain learning engine further arranged to correct spatial mismatches among the training image representing the key sample section and the training images of the adjacent sample sections with an extrapolation process based on a predetermined section thickness and a predetermined ordering of the key sample section and the adjacent sample sections.
19. The system for generating a stained image in accordance with claim 18 , wherein the tissue sample section is physically unstained or physically stained with an input stain.
20. The system for generating a stained image in accordance with claim 18 , wherein one of the at least three adjacent sample sections is stained with an output stain; and wherein the remaining adjacent sample sections are all stained with an input stain.
21. The system for generating a stained image in accordance with claim 20 , wherein the at least one generated stained image represents the key sample section stained with the output stain.
22. The system for generating a stained image in accordance with claim 20 , wherein the remaining adjacent sample sections are stained with multiple types of the input stain in an alternating manner.
23. The system for generating a stained image in accordance with claim 18 , wherein the machine learning network includes a generator network arranged to generate the at least one generated stained image over a plurality of cycles and a discriminator network arranged to analyze the at least one generated stained image to provide feedback to the generator network on each of the plurality of cycles.
24. The system for generating a stained image in accordance with claim 23 , wherein the machine learning network is a generative adversarial network.
25. The system for generating a stained image in accordance with claim 24 , wherein the generative adversarial network is trained with the training images of the at least three adjacent sample sections.
26. The system for generating a stained image in accordance with claim 25 , wherein the generative adversarial network is further trained with the training images of the at least three adjacent sample sections imaged with different illumination conditions.
27. The system for generating a stained image in accordance with claim 26 , wherein the illumination conditions include brightfield and autofluorescence.
28. The system for generating a stained image in accordance with claim 20 , wherein each of the input stain and the output stain includes at least one of hematoxylin and eosin (H&E) and a special stain.
29. The system for generating a stained image in accordance with claim 28 , wherein the special stain includes immunostains.
30. The system for generating a stained image in accordance with claim 29 , wherein the at least three adjacent sample sections are frozen.
31. The system for generating a stained image in accordance with claim 30 , wherein the at least three adjacent sample sections are formalin-fixed or embedded in paraffin.
32. The system for generating a stained image in accordance with claim 18 , wherein the training image representing the key sample section is stained with an output stain and the training images of the adjacent sample sections are stained with an input stain.
33. The system for generating a stained image in accordance with claim 32 , wherein the multi-modal stain learning engine is arranged to align all training images of the adjacent sample sections stained with the input stain to the training image representing the key sample section stained with the output stain using extrapolation.
34. The system for generating a stained image in accordance with claim 33 , wherein the multi-modal stain learning engine is further arranged to apply an image transformation to generate at least one estimated input image based on the training images of the adjacent sample sections stained with the input stain.