Systems and methods for visualizing elastin and collagen in tissue
Systems and methods for visualizing, and/or determining the amount of, collagen and elastin in tissue are provided. Training data can be generated using Mueller matrix polarimetry microscopy data, combined with second harmonic generation (SHG) and/or two photon excitation fluorescence (TPEF) microscopy data as ground truth. The SHG and/or TPEF data can be used to train a neural network for feature extraction, and classification can be performed. The components and decompositions of the Mueller matrix data can be arranged as individual channels of information, forming one voxel per sample.
1. A system for visualizing elastin and collagen in tissue, the system comprising:
a processor; and
a machine-readable medium in operable communication with the processor and having instructions stored thereon that, when executed by the processor, perform the following steps:
receiving imaging data of the tissue;
extracting from the imaging data a ground truth for collagen in the tissue and a ground truth for elastin in the tissue;
extracting Mueller matrix data from the imaging data to generate a voxel of the imaging data;
utilizing a neural network on the voxel to extract features and generate a feature matrix of the imaging data; and
applying at least one classifier to the feature matrix to generate a predicted image to visualize the elastin and the collagen in the tissue.
2. The system according to claim 1 , the instructions when executed further performing the following step:
before utilizing the neural network, training the neural network using the ground truth for collagen, the ground truth for elastin, and the voxel.
3. The system according to claim 1 , the instructions when executed further performing the following step:
before applying the at least one classifier, training the at least one classifier using the ground truth for collagen and the ground truth for elastin.
4. The system according to claim 1 , the instructions when executed further performing the following steps:
transforming the ground truth for collagen and the ground truth for elastin into a ground truth vector;
before utilizing the neural network, training the neural network using the ground truth vector and the voxel; and
before applying the at least one classifier, training the at least one classifier using the ground truth vector.
5. The system according to claim 1 , the at least one classifier comprising a machine vision classifier.
6. The system according to claim 1 , the at least one classifier comprising a K-nearest neighbor classifier and a semantic segmentation neural network.
7. The system according to claim 1 , the neural network being a convolutional neural network (CNN).
8. The system according to claim 1 , the instructions when executed further performing the following step:
comparing the visualized elastin and collagen in the tissue to the ground truth for collagen and the ground truth for elastin, using at least one of mean square error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM).
9. The system according to claim 1 , the extracting of the ground truth for collagen and the ground truth for elastin comprising using second harmonic generation (SHG) and two photon excitation fluorescence (TPEF).
10. The system according to claim 9 , the SHG being used to extract the ground truth for collagen, and the TPEF being used to extract the ground truth for elastin.
11. The system according to claim 1 , the instructions when executed further performing the following step:
before utilizing the neural network and before applying the at least one classifier, removing outliers from the voxel, the ground truth for collagen, and the ground truth for elastin.
12. The system according to claim 1 , the instructions when executed further performing the following step:
before utilizing the neural network and before applying the at least one classifier, normalizing the voxel, the ground truth for collagen, and the ground truth for elastin.
13. The system according to claim 1 , the voxel comprising data channels, and the data channels of the voxel respectively comprising diattenuation of the Mueller matrix data, depolarization coefficient of the Mueller matrix data, linear retardation of the Mueller matrix data, differential diattenuation of the Mueller matrix data, orientation of the Mueller matrix data, differential depolarization of the Mueller matrix data, and all Muller Matrix elements of the Mueller matrix data except M 22 , M 33 , and M 44 .
14. A method for visualizing elastin and collagen in tissue, the system comprising:
receiving, by a processor, imaging data of the tissue;
extracting, by the processor, from the imaging data a ground truth for collagen in the tissue and a ground truth for elastin in the tissue;
extracting, by the processor, Mueller matrix data from the imaging data to generate a voxel of the imaging data;
utilizing, by the processor, a neural network on the voxel to extract features and generate a feature matrix of the imaging data; and
applying, by the processor, at least one classifier to the feature matrix to generate a predicted image to visualize the elastin and the collagen in the tissue.
15. The method according to claim 14 , further comprising:
transforming the ground truth for collagen and the ground truth for elastin into a ground truth vector;
before utilizing the neural network, training the neural network using the ground truth vector and the voxel; and
before applying the at least one classifier, training the at least one classifier using the ground truth vector.
16. The method according to claim 14 , the at least one classifier comprising a K-nearest neighbor classifier and a semantic segmentation neural network, and
the neural network being a convolutional neural network (CNN).
17. The method according to claim 14 , the extracting of the ground truth for collagen and the ground truth for elastin comprising using second harmonic generation (SHG) and two photon excitation fluorescence (TPEF),
the SHG being used to extract the ground truth for collagen, and
the TPEF being used to extract the ground truth for elastin.
18. The method according to claim 14 , further comprising:
before utilizing the neural network and before applying the at least one classifier, removing outliers from the voxel, the ground truth for collagen, and the ground truth for elastin; and
before utilizing the neural network and before applying the at least one classifier but after removing the outliers, normalizing the voxel, the ground truth for collagen, and the ground truth for elastin.
19. The method according to claim 14 , the voxel comprising data channels, and the data channels of the voxel respectively comprising diattenuation of the Mueller matrix data, depolarization coefficient of the Mueller matrix data, linear retardation of the Mueller matrix data, differential diattenuation of the Mueller matrix data, orientation of the Mueller matrix data, differential depolarization of the Mueller matrix data, and all Muller Matrix elements of the Mueller matrix data except M 22 , M 33 , and M 44 .
20. A system for visualizing elastin and collagen in tissue, the system comprising:
a processor; and
a machine-readable medium in operable communication with the processor and having instructions stored thereon that, when executed by the processor, perform the following steps:
receiving imaging data of the tissue;
extracting from the imaging data a ground truth for collagen in the tissue and a ground truth for elastin in the tissue;
extracting Mueller matrix data from the imaging data to generate a voxel of the imaging data;
removing outliers from the voxel, the ground truth for collagen, and the ground truth for elastin to generate a post-outlier voxel, a post-outlier ground truth for collagen, and a post-outlier ground truth for elastin;
normalizing the post-outlier voxel, the post-outlier ground truth for collagen, and the post-outlier ground truth for elastin to generate a normalized voxel, a normalized ground truth for collagen, and a normalized ground truth for elastin;
transforming the normalized ground truth for collagen and the normalized ground truth for elastin into a ground truth vector;
training a neural network using the ground truth vector and the normalized voxel;
utilizing the neural network on the normalized voxel to extract features and generate a feature matrix of the imaging data;
training at least one classifier using the ground truth vector;
applying the at least one classifier to the feature matrix to generate a predicted image to visualize the elastin and the collagen in the tissue; and
comparing the visualized elastin and collagen in the tissue to the normalized ground truth for collagen and the normalized ground truth for elastin, using at least one of mean square error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM),
the at least one classifier comprising a K-nearest neighbor classifier and a semantic segmentation neural network,
the neural network being a convolutional neural network (CNN),
the extracting of the ground truth for collagen and the ground truth for elastin comprising using second harmonic generation (SHG) and two photon excitation fluorescence (TPEF),
the SHG being used to extract the ground truth for collagen,
the TPEF being used to extract the ground truth for elastin,
the voxel comprising data channels,
the data channels of the voxel respectively comprising diattenuation of the Mueller matrix data, depolarization coefficient of the Mueller matrix data, linear retardation of the Mueller matrix data, differential diattenuation of the Mueller matrix data, orientation of the Mueller matrix data, differential depolarization of the Mueller matrix data, and all Muller Matrix elements of the Mueller matrix data except M 22 , M 33 , and M 44 , and
the tissue being cervical tissue.