Granted Patent
B2
US 12,608,768 · App. 18/423,669 · Granted Apr 21, 2026
Hyperspectral learning for instantaneous spatiospectral imaging of hemodynamics
Inventors:
Young L Kim (West Lafayette, IN); Yuhyun Ji (West Lafayette, IN); Sang Mok Park (West Lafayette, IN); Semin Kweon (West Lafayette, IN); Jungwoo Leem (West Lafayette, IN)
Assignee:
Purdue Research Foundation
G06T5/50G01N21/31G01N33/4833G06T7/0012H04N23/10G06T2207/10016G06T2207/10024G06T2207/10056G06T2207/20081G06T2207/20084G06T2207/30024
View Patent ↗
Loading inventors, assignments & file history…
Abstract
A method of generating an image or video of a field of interest of a sample which includes obtaining i) a first RGB image from a field of interest of a sample, and ii) hyperspectral data from a subarea of the field of interest, extracting an RGB image of the subarea from the first RGB image of the field of interest, applying the hyperspectral data of the subarea to conduct a spectroscopic analysis of a sample thereby generating spectral parameters, inputting i) the spectral parameters, and ii) the first RGB image, collectively as training input data to a deep learning model (DLM), training the DLM with the training input data thus generating a trained DLM, obtaining and inputting a second RGB image of about the field of interest to the trained DLM, and outputting from the trained DLM a spectral map for the field of interest.
Claims (30)
1 . A method of generating an image or video of a field of interest of a sample, comprising:
obtaining: i) a first Red-Green-Blue (RGB) image from about a field of interest of a sample, and ii) hyperspectral data from a subarea of the field of interest;
extracting an RGB image of the subarea from the first RGB image of the field of interest;
applying the hyperspectral data of the subarea to conduct a spectroscopic analysis of a sample, thereby generating spectral parameters for the subarea;
inputting i) the generated spectral parameters, and ii) the extracted RGB image of the subarea, collectively as training input data to a deep learning model;
training the deep learning model with the training input data thus generating a trained deep learning model;
obtaining a second RGB image about the field of interest including areas outside of the subarea;
inputting the second RGB image of the field of interest to the trained deep learning model;
outputting from the trained deep learning model a spectral map for the field of interest; and
extracting hemodynamic parameters from the outputted spectral map of the field of interest; and
determining oxygen saturation (sPO 2 ).
2 . The method of claim 1 , wherein the deep learning model is a neural network.
3 . The method of claim 1 , wherein the first and second RGB images of the field of interest are obtained as a photograph.
4 . The method of claim 1 , wherein the first and second RGB images of the field of interest are obtained from a video frame.
5 . The method of claim 4 , wherein the video has a frame rate of between about 960 frames per second and about 1920 frames per second.
6 . The method of claim 1 , wherein the first and second RGB images are obtained from a trichromatic camera.
7 . The method of claim 1 , wherein the hyperspectral data is obtained from an imaging spectrograph.
8 . The method of claim 1 , wherein the hyperspectral data is obtained from a spectrometer.
9 . The method of claim 6 , wherein the trichromatic camera is a smartphone camera.
10 . The method of claim 6 , wherein light from the sample is provided to the trichromatic camera via a mirror and a first plurality of lenses.
11 . The method of claim 7 , wherein the mirror includes a photometric slit adapted to provide light from the subarea.
12 . The method of claim 7 , wherein light exiting the photometric slit is diffracted by a diffraction grating and supplied to the imaging spectrograph via a second plurality of lenses.
13 . The method of claim 7 , position of the photometric slit on the mirror is selectable.
14 . The method of claim 10 , wherein the first and second RGB images from the field of interest are dividable into two or more subareas and for each said subarea, position of the photometric slit on the mirror is selectable.
15 . The method of claim 1 , wherein the first and second RGB images are obtained from a microscope combined with a fiber optics spectrometer via a beam-splitter thereby enabling obtaining hyperspectral and RGB data in the subarea and an RGB image of the field of interest.
16 . The method of claim 1 , wherein the spectroscopic analysis of the sample is based on a sample optics domain knowledge model to generate the spectral parameters from the hyperspectral data.
17 . The method of claim 14 , wherein the sample optics domain knowledge model is based on one or more of theory of radiative transport, robust approximations, and Monte Carlo simulations.
18 . The method of claim 15 , wherein the robust approximations are based on one or more of diffusion, Born, and empirical modeling.
19 . The method of claim 1 , wherein the spectral parameters include hemodynamic parameters.
20 . The method of claim 1 , wherein the first RGB image is same as the second RGB image.
Continuity (2)
Provisional Application
63444522
· Feb 9, 2023
References Cited (108)
WO WO2023205631A2
· 2023
[cited by examiner]
WO WO2025144823A1
· 2025
[cited by examiner]
French et al., “Speckle-based hyperspectral imaging combining multiple scattering and compressive sensing in nanowire mats,” Optics Letters 42, 1820-1823, 9, (2017).
[cited by applicant]
Deng et al., “Snapshot hyperspectral imaging via spectral basis multiplexing in Fourier domain,” Optics Express 26, 32509-32521, 25, (2018).
[cited by applicant]
Antipa et al., “DiffuserCam: Lensless single-exposure 3D imaging,” Optica 5, 1-9, 1, (2018).
[cited by applicant]
Wang et al., “HyperReconNet: Joint coded aperture optimization and image reconstruction for compressive hyperspectral imaging,” IEEE Transactions on Image Processing 28, 2257-2270, 5, (2019).
[cited by applicant]
Toivonen et al., “Snapshot hyperspectral imaging using wide dilation networks,” Machine Vision and Applications 32, 1, (2020).
[cited by applicant]
Bacca et al., “Compressive spectral image reconstruction using deep prior and low-rank tensor representation,” Applied Optics 60, 4197-4207, 14, (2021).
[cited by applicant]
Lin et al., “Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks,” Medical Image Analysis 48, 162, (2021).
[cited by applicant]
Brown et al., “Neural network-based on-chip spectroscopy using a scalable plasmonic encoder,” ACS Nano 15, 6305-6315, 4, (2021).
[cited by applicant]
Kudenov et al., “Practical spectral photography II: Snapshot spectral imaging using linear retarders and microgrid polarization cameras,” Optics Express 30, 12337-12352, 8, (2022).
[cited by applicant]
Xie et al., “Dual camera snapshot hyperspectral imaging system via physics-informed learning,” Optics and Lasers in Engineering 154, (2022).
[cited by applicant]
Xiong et al., “Dynamic brain spectrum acquired by a real-time ultraspectral imaging chip with reconfigurable metasurfaces,” Optica 9, 461-468, 5, (2022).
[cited by applicant]
Lu et al., “Medical hyperspectral imaging: A review,” Journal of Biomedical Optics 19, 1, (2014).
[cited by applicant]
Pichette et al., “Intraoperative video-rate hemodynamic response assessment in human cortex using snapshot hyperspectral optical imaging,” Neurophotonics 3, 045003, 4, (2016).
[cited by applicant]
Mori et al., “Intraoperative visualization of cerebral oxygenation using hyperspectral image data: A two-dimensional mapping method,” International Journal of Computer Assisted Radiology and Surgery 9, 1059-1072, 6, (20…
[cited by applicant]
Chen et al., “Modified Wiener estimation of diffuse reflectance spectra from RGB values by the synthesis of new colors for tissue measurements,” Journal of Biomedical Optics 17, 3, (2012).
[cited by applicant]
Yoshida et al., “Multispectral imaging of absorption and scattering properties of in vivo exposed rat brain using a digital red-green-blue camera,” Journal of Biomedical Optics 20, 5, (2015).
[cited by applicant]
Yoon et al., “Hyperspectral imaging using RGB color for foodborne pathogen detection,” Journal of Electronic Imaging 24, 4, (2015).
[cited by applicant]
Galliani et al., Learned spectral super-resolution, arXiv preprint arXiv:1703.09470, (2017).
[cited by applicant]
Koundinya et al., “2D-3D CNN based architectures for spectral reconstruction from RGB images,” IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 957-964, (2018).
[cited by applicant]
Khansari et al., “Inter-visit variability of conjunctival microvascular hemodynamic measurements in healthy and diabetic retinopathy subjects,” Microvascular Research 118, 7-11, (2018).
[cited by applicant]
Watanabe et al., “Hemoglobin phase of oxygenation and deoxygenation in early brain development measured using fNIRS,” National Academy of Sciences of the United States of America 114, E1737-E1744, 9, (2017).
[cited by applicant]
Liang et al., “Symbolic time series analysis of fNIRS signals in brain development assessment,” Journal of Neural Engineering 15, 6, (2018).
[cited by applicant]
Zhang, “Film-like images with super-fine details using Foveon X3 Technology,” Imaging Systems and Applications, IM3E. 4, (2017).
[cited by applicant]
Zhao et al., “Perfect RGB-IR color routers for sub-wavelength size CMOS image sensor pixels,” Advanced Photonics Research 2, 2000048, 3, (2021).
[cited by applicant]
Zou et al., “Pixel-level Bayer-type colour router based on metasurfaces,” Nature Communications 13, 1-7, 1, (2022).
[cited by applicant]
Royle et al., “Parental investment and egg yolk lipid composition in gulls,” Functional Ecology 13, 298-306, 3, (1999).
[cited by applicant]
Van Veen et al., “Determination of visible near-IR absorption coefficients of mammalian fat using time- and spatially resolved diffuse reflectance and transmission spectroscopy,” Journal of Biomedical Optics 10, 054004,…
[cited by applicant]
Lagarias et al., “Convergence properties of the Nelder-Mead simplex method in low dimensions,” SIAM Journal on Optimization 9, 112-147, 1, (1998).
[cited by applicant]
Kruse et al., “The Spectral Image-Processing System (Sips)—Interactive visualization and analysis of imaging spectrometer data,” Remote Sensing of Environment 44, 145-163, 2-3, (1993).
[cited by applicant]
Arrieta et al., “Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI,” Information Fusion 58, 82-115, (2020).
[cited by applicant]
Karniadakis et al., “Physics-informed machine learning,” Nature Reviews Physics 3, 422-440, 6, (2021).
[cited by applicant]
Cuomo et al., “Scientific machine learning through physics-informed neural networks: Where we are and what's next,” Journal of Scientific Computing 92, 3, (2022).
[cited by applicant]
Skopal et al., “Early evolution of the extraordinary Nova Delphini 2013 (V339 Del),” Astronomy & Astrophysics 569, A112, (2014).
[cited by applicant]
Finlayson et al., “Rank-based camera spectral sensitivity estimation,” Journal of the Optical Society of America A 33, 589-599, 4, (2016).
[cited by applicant]
Ji et al., “Compressive recovery of smartphone RGB spectral sensitivity functions,” Optics Express 29, 11947-11961, 8, (2021).
[cited by applicant]
Finlayson et al., “Finding a colour filter to make a camera colorimetric by optimisation,” Computational Color Imaging 11418, 53-62, (2019).
[cited by applicant]
Yoon et al., “Hyperspectral image recovery using a color camera for detecting colonies of foodborne pathogens on agar plate,” Journal of Biosystems Engineering 44, 169-185, 3, (2019).
[cited by applicant]
Flock et al., “Monte Carlo modeling of light-propagation in highly scattering tissues. 1. Model predictions and comparison with diffusion-theory,” IEEE Transactions on Biomedical Engineering 36, 1162-1168, 12, (1989).
[cited by applicant]
Finlay et al., “Effect of pigment packaging on diffuse reflectance spectroscopy of samples containing red blood cells,” Optics Letters 29, 965-967, 9, (2004).
[cited by applicant]
Amelink et al., “Confidence intervals on fit parameters derived from optical reflectance spectroscopy measurements,” Journal of Biomedical Optics 13, 5, (2008).
[cited by applicant]
Rajaram et al., “Experimental validation of the effects of microvasculature pigment packaging on in vivo diffuse reflectance spectroscopy,” Lasers in Surgery and Medicine 42, 680-688, 7, (2010).
[cited by applicant]
Boas et al., Handbook of biomedical optics, p. 787 (CRC Press, Boca Raton, 2011).
[cited by applicant]
Simonyan et al., “Deep inside convolutional networks: Visualising image classification models and saliency maps,” arXiv preprint arXiv:1312.6034, (2013).
[cited by applicant]
Selvaraju et al., “Grad-cam: Visual explanations from deep networks via gradient-based localization,” IEEE International Conference on Computer Vision, 618-626, (2017).
[cited by applicant]
Perry, “A complete culture system for the chick-embryo,” Nature 331, 70-72, 6151, (1988).
[cited by applicant]
Yoon et al., “A background correction method to compensate illumination variation in hyperspectral imaging,” Plos One 15, 3, (2020).
[cited by applicant]
Abraham et al., “Continuous monitoring of tissue pH with a fiberoptic conjunctival sensor,” Annals of Emergency Medicine 14, 840-844, 9, (1985).
[cited by applicant]
Lin et al., “Melanocyte biology and skin pigmentation,” Nature 445, 843-850, 7130, (2007).
[cited by applicant]
Yan et al., “Accurate spectral super-resolution from single RGB image using multi-scale CNN,” Pattern Recognition and Computer Vision 11257, 206-217, (2018).
[cited by applicant]
Stiebel et al., “Reconstructing spectral images from RGB-images using a convolutional neural network,” IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1061-1066, (2018).
[cited by applicant]
Arad et al., “NTIRE 2018 challenge on spectral reconstruction from RGB images,” IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1042-1051, (2018).
[cited by applicant]
Kaya et al., “Towards spectral estimation from a single RGB image in the wild,” IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 3546-3555, (2019).
[cited by applicant]
Park et al., “mHealth spectroscopy of blood hemoglobin with spectral super-resolution,” Optica 7, 563-573, 6, (2020).
[cited by applicant]
Zou et al., “Cluster-based deep convolutional networks for spectral reconstruction from RGB images,” Neurocomputing 464, 342-351, (2021).
[cited by applicant]
Lin et al., “Physically plausible spectral reconstruction from RGB Images,” IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2257-2266, (2020).
[cited by applicant]
Kwak et al., “A pearl spectrometer,” Nano Letters 21, 921-930, 2, (2021).
[cited by applicant]
Lin et al., “On the optimization of regression-based spectral reconstruction,” Sensors 21, 16, (2021).
[cited by applicant]
Arad et al., “NTIRE 2020 challenge on spectral reconstruction from an RGB image,” IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1806-1822, (2020).
[cited by applicant]
Munos et al., “Mobile health: The power of wearables, sensors, and apps to transform clinical trials,” Annals of the New York Academy of Sciences 1375, 3-18, Special Issue: Annals Reports, (2016).
[cited by applicant]
Wood et al., “Taking connected mobile-health diagnostics of infectious diseases to the field,” Nature 566, 467-474, 7745, (2019).
[cited by applicant]
Mathews et al., “Digital health: A path to validation,” npj Digital Medicine 2, 1-9, (2019).
[cited by applicant]
Hunt et al., “Smartphone-based imaging systems for medical applications: A critical review,” Journal of Biomedical Optics 26, 4, (2021).
[cited by applicant]
Hussain et al., “Smartphone-based optical spectroscopic platforms for biomedical applications: A review,” Biomedical Optics Express 12, 1974-1998, 4, (2021).
[cited by applicant]
Hussain et al., “A multi-channel smartphone-based spectroscopic system for high-throughput biosensing in low-resource settings,” Analyst 147, 3007-3016, 13, (2022).
[cited by applicant]
Sedhom et al., “Mobile app validation: A digital health scorecard approach,” npj Digital Medicine 4, 1, (2021).
[cited by applicant]
Von Rueden et al., “Informed machine learning—A taxonomy and survey of integrating knowledge into learning systems,” arXiv preprint arXiv:1903.12394, (2019).
[cited by applicant]
Iten et al., “Discovering physical concepts with neural networks,” Physical Review Letters 124, 1, (2020).
[cited by applicant]
Zhang et al., “Deeply learned broadband encoding stochastic hyperspectral imaging,” Light-Science & Applications 10, 1, (2021).
[cited by applicant]
Sun et al., “pHSCNN: CNN-based hyperspectral recovery from a pair of RGB images,” Optics Express 30, 24862-24873, 14, (2022).
[cited by applicant]
Du et al., Label-free hyperspectral imaging and deep-learning prediction of retinal amyloid β-protein and phosphorylated tau, PNAS nexus 1, 164, 4, (2022).
[cited by applicant]
Wei et al., “Fast fusion of multi-band images based on solving a Sylvester equation,” IEEE Transactions on Image Processing 24, 4109-4121, 11, (2015).
[cited by applicant]
Kanatsoulis et al., “Hyperspectral super-resolution: A coupled tensor factorization approach,” IEEE Transactions on Signal Processing 66, 6503-6517, 24, (2018).
[cited by applicant]
Li et al., “Hyperspectral image super-resolution using deep convolutional neural network,” Neurocomputing 266, 29-41, (2017).
[cited by applicant]
Wang et al., Single-shot on-chip spectral sensors based on photonic crystal slabs, Nature Communications 10, 1-6, (2019).
[cited by applicant]
Wang et al., “Snapshot channeled imaging spectrometer using geometric phase holograms,” Optics Express 27, 15444-15455, 11, (2019).
[cited by applicant]
He et al., “Analysis of skin morphological features and real-time monitoring using snapshot hyperspectral imaging,” Biomedical Optics Express 10, 5625-5638, 11, (2019).
[cited by applicant]
McClung et al., “Snapshot spectral imaging with parallel metasystems,” Science Advances 6, 38, (2020).
[cited by applicant]
Altaqui et al., “Mantis shrimp-inspired organic photodetector for simultaneous hyperspectral and polarimetric imaging,” Science Advances 7, 10, (2021).
[cited by applicant]
Monakhova et al., “Spectral DiffuserCam: Lensless snapshot hyperspectral imaging with a spectral filter array,” Optica 7, 1298-1307, 10, (2020).
[cited by applicant]
Yako et al., “Video-rate hyperspectral camera based on a CMOS-compatible random array of Fabry-Pérot filters,” Nature Photonics, (2023).
[cited by applicant]
Bouchard et al., “Ultra-fast multispectral optical imaging of cortical oxygenation, blood flow, and intracellular calcium dynamics,” Optics Express 17, 15670-15678, 18, (2009).
[cited by applicant]
He et al., “Hyperspectral imaging enabled by an unmodified smartphone for analyzing skin morphological features and monitoring hemodynamics,” Biomedical optics express 11, 895-910, 2, (2020).
[cited by applicant]
Johnson et al., “Snapshot hyperspectral imaging in ophthalmology,” Journal of Biomedical Optics 12, 1, (2007).
[cited by applicant]
Boniface et al., “Rapid broadband characterization of scattering medium using hyperspectral imaging,” Optica 6, 274-279, 3, (2019).
[cited by applicant]
Mu et al., “Snapshot hyperspectral imaging polarimetry with full spectropolarimetric resolution,” Optics and Lasers in Engineering 148, 106767, (2022).
[cited by applicant]
Wagadarikar et al., “Video rate spectral imaging using a coded aperture snapshot spectral imager,” Optics Express 17, 6368-6388, 8, (2009).
[cited by applicant]
Yokoya et al., “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” IEEE Geoscience and Remote Sensing Magazine 5, 29-56, 2, (2017).
[cited by applicant]
Huang et al., “Spectral imaging with deep learning,” Light-Science & Applications 11, 1, (2022).
[cited by applicant]
Yang et al., “Miniaturization of optical spectrometers,” Science 371, 480, 6528, (2021).
[cited by applicant]
Redding et al., “Compact spectrometer based on a disordered photonic chip,” Nature Photonics 7, 746-751, 9, (2013).
[cited by applicant]
Bao et al., “A colloidal quantum dot spectrometer,” Nature 523, 67, 7558, (2015).
[cited by applicant]
Ghamisi et al., “Advances in hyperspectral image and signal processing: A comprehensive overview of the state of the art,” IEEE Geoscience and Remote Sensing Magazine 5, 37-78, 4, (2017).
[cited by applicant]
Khan et al., “Modern trends in hyperspectral image analysis: A Review,” IEEE Access 6, 14118-14129, (2018).
[cited by applicant]
Yang et al., “Single-nanowire spectrometers,” Science 365, 1017, 6457, (2019).
[cited by applicant]
Jaiswal et al., “Critical insights into modern hyperspectral image applications through deep learning,” Wiley Interdisciplinary Reviews-Data Mining and Knowledge Discovery 11, 6, (2021).
[cited by applicant]
Reichstein et al., “Deep learning and process understanding for data-driven earth system science,” Nature 566, 195-204, 7743, (2019).
[cited by applicant]
Manifold et al., “A versatile deep learning architecture for classification and label-free prediction of hyperspectral images,” Nature Machine Intelligence 3, 306, 4, (2021).
[cited by applicant]
Zhang et al., “A survey on computational spectral reconstruction methods from RGB to hyperspectral imaging,” Scientific Reports 12, 11905, 1, (2022).
[cited by applicant]
Tsagkatakis et al., “Survey of deep-learning approaches for remote sensing observation enhancement,” Sensors 19, 3929, 18, (2019).
[cited by applicant]
Hong et al., “Interpretable hyperspectral artificial intelligence: When nonconvex modeling meets hyperspectral remote sensing,” IEEE Geoscience and Remote Sensing Magazine 9, 52-87, 2, (2021).
[cited by applicant]
Kirchner et al., “Snapshot hyperspectral imaging (SHI) for revealing irreversible and heterogeneous plasmonic processes,” Journal of Physical Chemistry C 122, 6865-6875, 12, (2018).
[cited by applicant]
Hadoux et al., “Non-invasive in vivo hyperspectral imaging of the retina for potential biomarker use in Alzheimer's disease,” Nature Communications 10, 1-12, (2019).
[cited by applicant]
Yoon et al., “A clinically translatable hyperspectral endoscopy (HySE) system for imaging the gastrointestinal tract,” Nature Communications 10, 1-13, (2019).
[cited by applicant]
Zhang et al., “Unsupervised adaptation learning for hyperspectral imagery super-resolution,” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 3070-3079, (2020).
[cited by applicant]
Wang et al., “DNU: Deep non-local unrolling for computational spectral imaging,” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1658-1668, (2020).
[cited by applicant]