IP Library › Granted Patent US 12,226,188
Granted Patent B2
US 12,226,188 · App. 18/118,013 · Granted Feb 18, 2025

Active illumination and time-of-flight camera system to evaluate facial blood flow, eye movements and physiological parameters

Inventor: Mohammed N. Islam (Ann Arbor, MI)
Assignee: OMNI MEDSCI, INC.
A61B5/0088A61B5/0013A61B5/0022A61B5/0075A61B5/0086A61B5/14532A61B5/14546A61B5/1455A61B5/4547A61B5/6801A61B5/7203A61B5/7257A61B5/7405A61B5/742A61C19/04G01J3/02G01J3/0218G01J3/108G01J3/14G01J3/28G01J3/2823G01J3/42G01J3/453G01N21/35G01N21/3504G01N21/3563G01N21/359G01N21/39G01N21/88G01N33/02G01N33/025G01N33/15G01N33/442G01N33/49G16H40/67G16Z99/00A61B5/0024A61B2562/0233A61B2562/0238A61B2562/146A61B2576/02A61C1/0046G01J2003/104G01J2003/1208G01J3/1838G01J2003/2826G01M3/38G01N2021/3513G01N2021/3595G01N2021/399G01N21/85G01N21/9508G01N2201/061G01N2201/06113G01N2201/062G01N2201/08G01N2201/12G01N2201/129H01S3/0092H01S3/06758H01S3/302Y02A90/10
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,226,188
App. No.
18/118,013
Granted
Feb 18, 2025
Kind
B2
Abstract

A measurement system comprising one or more semiconductor diodes configured to penetrate tissue comprising skin. The detection system comprising a camera, which may also include a direct or indirect time-of-flight sensor. The detection system synchronized to the pulsing of the semiconductor diodes, and the camera further coupled to a processor. The detection system non-invasively measuring blood within the skin, measuring hemoglobin absorption between 700 to 1300 nm, and the processor deriving physiological parameters and comparing properties between different spatial locations and variation over time. The semiconductor diodes may comprise vertical cavity surface emitting lasers, and the detection system may comprise single photon avalanche photodiodes. The measurement system may be used to observe eye parameters and differential blood flow. The system may be used with photo-bio-modulation therapy, or it may be used in advanced driver monitoring systems for multiple functions including head pose, eye tracking, facial authentication, and smart restraint control systems.

Claims (22)

1. A measurement system comprising:

one or more semiconductor diodes to operate in a pulsed mode and to generate a pulsed light having one or more optical wavelengths that includes at least one near-infrared wavelength, wherein at least a portion of the pulsed light generated by the one or more semiconductor diodes is configured to penetrate tissue comprising skin;

a detection system comprising a camera, wherein the detection system is configured to be synchronized to the pulsed light of the one or more semiconductor diodes, and wherein the detection system further comprises one or more lenses in front of at least a part of the detection system;

the camera configured to receive at least a portion of the pulsed light reflected from the tissue from at least a part of the one or more semiconductor diodes, wherein the camera is configured to generate data based at least in part on the portion of the pulsed light received, and the camera is further coupled to a processor;

wherein the measurement system including the processor is configured to generate an image using at least part of the data from the detection system;

the detection system configured to non-invasively measure blood in blood vessels within or below a dermis layer within the skin based at least in part on reflection from the skin;

the detection system configured to measure absorption of hemoglobin in the near-infrared wavelength between 700 nanometers and 1300 nanometers;

wherein the processor is configured to compare the absorption of hemoglobin between different spatial locations of tissue;

wherein the measurement system including the processor is configured to measure one or more physiological parameters based at least in part on the non-invasively measured blood within the tissue, and wherein the measurement system including the processor is configured to measure over a period of time a variation in the blood within the tissue or the one or more physiological parameters; and

wherein the measurement system including the processor is further configured to be coupled to an active remote sensing system comprising:

one or more laser diodes configured to generate laser light having an initial light intensity and one or more second optical wavelengths, wherein at least a portion of the one or more second optical wavelengths is between 700 nanometers and 2500 nanometers, wherein the one or more laser diodes comprises one or more Bragg reflectors, wherein the one or more laser diodes is further configured to be modulated with a pulsed output with a pulse duration of approximately 0.5 to 2 nanoseconds and a pulse repetition rate between about 10 Megahertz and 1 Gigahertz, wherein the one or more laser diodes is further configured to be coupled to driver electronics, and wherein the laser light from the one or more laser diodes is configured to be directed to the tissue; and

a second detection system comprising a photodiode array, wherein the second detection system further comprises one or more second lenses and one or more spectral filters in front of at least a part of the photodiode array, wherein the photodiode array is further coupled to the processor, and wherein the photodiode array comprises a plurality of pixels coupled to CMOS transistors;

wherein the second detection system is configured to receive at least a portion of laser light from the one or more laser diodes reflected from the tissue, and wherein the second detection system is further configured to be synchronized to the one or more laser diodes comprising Bragg reflectors;

wherein the second detection system is further configured to perform a time-of-flight measurement based on a time difference between a first time in which the one or more laser diodes generate laser light and a second time in which the photodiode array receives the at least a portion of laser light from the one or more laser diodes reflected from the tissue; and

wherein the second detection system is further configured to perform the time-of-flight measurement at least in part by measuring a temporal distribution of photons in the received portion of laser light from the one or more laser diodes reflected from the tissue.

2. The measurement system of claim 1 , wherein the one or more laser diodes comprises vertical cavity surface emitting lasers, wherein the photodiode array comprises single-photon avalanche photodiodes, and wherein at least some of the one or more second optical wavelengths are near 850 nanometers or near 940 nanometers.

3. The measurement system of claim 2 , wherein the processor analyzes information from one or both of the detection system or the second detection system using artificial intelligence, machine learning, deep learning, or generative artificial intelligence.

4. The measurement system of claim 3 , wherein depth information from the time-of-flight measurement is used at least in part to compensate for motion artifacts.

5. The measurement system of claim 4 , wherein the measurement system including the processor is configured to compare differential blood flow between a forehead region and a nose region or between the forehead region and a cheek region on a user; or

wherein the measurement system including the processor is further coupled to a therapy or diagnostic system comprising additional light sources configured to emit additional light between approximately 700 and 1100 nanometers, wherein the additional light from the additional light sources is configured to be directed toward the forehead region on the user, and wherein the additional light sources comprise vertical cavity surface emitting lasers or light emitting diodes.

6. The measurement system of claim 4 , wherein the measurement system including the processor is configured to be used in an advance driver assistance system or a driver monitoring system and configured to perform functions selected from the group consisting of (i) head pose, (ii) eye parameters, (iii) facial authentication, (iv) vital sign monitoring, (v) facial blood flow for drowsiness or impaired or drunk driver detection, and (vi) smart restraint control systems.

7. The measurement system of claim 3 , wherein the camera is configured to measure near an eye region of a user, and wherein the processor is configured to determine eye parameters selected from the group consisting of eye blink rate, percent of eye closure, pupil dilation, eye gaze direction, eye saccade and eye fixation.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 7, 2023
From: ISLAM, MOHAMMED N.
To: OMNI MEDSCI, INC.
Reel/Frame 062908/0324 →
Continuity (41)
Continuation In Part 17666518 · Feb 7, 2022
Continuation 17135233 · Dec 28, 2020
Continuation 16669794 · Oct 31, 2019
Continuation 16284514 · Feb 25, 2019
Continuation 16016649 · Jun 24, 2018
Continuation 15860065 · Jan 2, 2018
Continuation 15686198 · Aug 25, 2017
Continuation 15357136 · Nov 21, 2016
Continuation 14651367
Continuation 16506885 · Jul 9, 2019
Continuation 16272069 · Feb 11, 2019
Continuation 16029611 · Jul 8, 2018
Continuation 15888052 · Feb 4, 2018
Continuation 15212549 · Jul 18, 2016
Continuation 14650897
Continuation 16004359 · Jun 9, 2018
Continuation 14109007 · Dec 17, 2013
Continuation 16188194 · Nov 12, 2018
Continuation 16004154 · Jun 8, 2018
Continuation 15855201 · Dec 27, 2017
Continuation 15711907 · Sep 21, 2017
Division 15357225 · Nov 21, 2016
Continuation 14650981
Continuation 16241628 · Jan 7, 2019
Continuation 16015737 · Jun 22, 2018
Continuation 15594053 · May 12, 2017
Continuation 14875709 · Oct 6, 2015
Continuation 14108986 · Dec 17, 2013
Continuation 16284514 · Feb 25, 2019
Continuation 16016649 · Jun 24, 2018
Continuation 15860065 · Jan 2, 2018
Continuation 15686198 · Aug 25, 2017
Continuation 15357136 · Nov 21, 2016
Continuation 14651367
Provisional Application 61747477 · Dec 31, 2012
Provisional Application 61754698 · Jan 21, 2013
Provisional Application 61747472 · Dec 31, 2012
Provisional Application 61747553 · Dec 31, 2012
Provisional Application 61747485 · Dec 31, 2012
Provisional Application 61747487 · Dec 31, 2012
Related Publication 20230277065A1 · Sep 7, 2023
References Cited (83)
US 8587771B2 · Xu et al. · 2013 [cited by applicant]
US 8749796B2 · Pesach et al. · 2014 [cited by applicant]
US 8928893B2 · Findlay et al. · 2015 [cited by applicant]
US 9151829B2 · Campbell · 2015 [cited by applicant]
US 9171985B2 · Dutton et al. · 2015 [cited by applicant]
US 9236519B2 · Mazzillo et al. · 2016 [cited by applicant]
US 9265456B2 · Kirenko et al. · 2016 [cited by applicant]
US 9273846B1 · Rossi et al. · 2016 [cited by applicant]
US 9316735B2 · Baxter · 2016 [cited by applicant]
US 9417734B2 · Raynor et al. · 2016 [cited by applicant]
US 9430093B2 · Harrison et al. · 2016 [cited by applicant]
US 9770213B2 · Kirenko et al. · 2017 [cited by applicant]
US 9860522B2 · Lapstun · 2018 [cited by applicant]
US 9891309B2 · Hudman et al. · 2018 [cited by applicant]
US 9915726B2 · Bailey et al. · 2018 [cited by applicant]
US 10117588B2 · Lee et al. · 2018 [cited by applicant]
US 10359505B2 · Buettgen et al. · 2019 [cited by applicant]
US 10376192B2 · Lee et al. · 2019 [cited by applicant]
US 10441173B2 · Shan et al. · 2019 [cited by applicant]
US 10646167B2 · De Haan · 2020 [cited by applicant]
US 10694988B2 · Lee et al. · 2020 [cited by applicant]
US 10705191B2 · Ryu et al. · 2020 [cited by applicant]
US 10806354B2 · Hutchinson et al. · 2020 [cited by applicant]
US 10824837B2 · Yeke Yazdandoost et al. · 2020 [cited by applicant]
US 10838206B2 · Fortin-Deschenes et al. · 2020 [cited by applicant]
US 10922395B2 · Morestin et al. · 2021 [cited by applicant]
US 10952683B2 · Jones et al. · 2021 [cited by applicant]
US 11403754B2 · Hutchinson et al. · 2022 [cited by applicant]
US 11625098B2 · Beyhs · 2023 [cited by applicant]
US 20160181325A1 · Johnson et al. · 2016 [cited by applicant]
US 20190294868A1 · Martinez · 2019 [cited by applicant]
US 20210096726A1 · Faulkner et al. · 2021 [cited by applicant]
US 20220225006A1 · Allec et al. · 2022 [cited by applicant]
US 20230000376A1 · Maman et al. · 2023 [cited by applicant]
WO WO2021149048A1 · 2021 [cited by applicant]
WO WO2022074652A1 · 2022 [cited by applicant]
WO WO2022084991A1 · 2022 [cited by applicant]
Carlos F. Benitez-Quiroz, Ramprakah Srinivasan, and Aleix M. Martinez, “Facial color is an efficient mechanism to visually transmit emotion,” Proceedings of the National Academy of Science PNAS, Apr. 3, 2018, vol. 115, … [cited by applicant]
Kaiwen Guo, Tianqu Zhai, Elton Pashollari, Christopher J. Varlamos, Aymaan Ahmed, and Mohammed N. Islam, “Contactless Vital Sign Monitoring System for Heart and Respiratory Rate Measurements with Motion Compensation Usi… [cited by applicant]
Kaiwen Guo, Tianqu Zhai, Manoj H. Purushothama, Alexander Dobre, Shawn Meah, Elton Pashollari, Aabhaas Vaish, Carl DeWilde and Mohammed N. Islam, “Contactless Vital Sign Monitoring System for In-Vehicle Driver Monitorin… [cited by applicant]
“The 6 best LiDAR Apps and Games for iPhone 12,” http://www.mickeysworkshop.com/product-design/2021/1/23 (printed from web on Aug. 8, 2022). [cited by applicant]
Samuel Kim, Irfan Wisanggeni, Ryan Ros, Rania Hussein, “Detecting Fatigue Driving through Perclos: A Review,” International Journal of Image Processing (IJIP), vol. 14, Issue 1, 2020, pp. 1-7. [cited by applicant]
Tereza Soukupova and Jan Cech, “Real-Time Eye Blink Detection using Facial Landmarks,” 21st Computer Vision Winter Workshop, Luka Cehovin, Rok Mandeljc, Vitormir Struc (eds), Rimske Toplice, Slovenia, Feb. 3-5, 2016. [cited by applicant]
C. Fabian Benitez-Quiroz, Ramprakash Srinivasan and Aleix M. Martinez, “Discriminant Functional Learning of Color Features for the Recognition of Facial Action Units and Their Intensities,” IEEE Transactions on Pattern … [cited by applicant]
Ramprakash Srinivasan, Julie D. Golomb and Aleix M. Martinez, “A Neural Basis of Facial Action Recognition in Humans,” The Journal of Neuroscience, Apr. 20, 2016, vol. 36, No. 16, pp. 4434-4442. [cited by applicant]
Suhandi Junaedi and Habibullah Akbar, “Driver Drowsiness Detection Based on Face Feature and PERCLOS,” Journal of Physics: Conference Series, 1090, 012037 (2018). [cited by applicant]
“Camera Phone,” on Wikipedia, https://en.wikipedia.org/wiki/Camera_Phone, printed Aug. 19, 2022. [cited by applicant]
Preethi Padmanabhan, Chao Zhang, and Edoardo Charbon, “Modeling and Analysis of a Direct Time-of-Flight Sensor Architecture for LiDAR Applications,” Sensors, 2019, 19, 5464; doi: 10.3390/s19245464, www.mdpi.com/journal/… [cited by applicant]
Application Note: Time-of-flight Basics, Melexis, AUH-2022, pp. 1-14. [cited by applicant]
Dingkang Wang, Connor Watkins and Huikai Xie, “MEMS Mirrors for LiDAR: A Review,” Micromachines 2020, 11, 456; doi:10.3390/mil1050456, www.mdpi.com/journal/micromachines. [cited by applicant]
Karl Joachim Ebeling, Rainer Michalzik and Holger Moench, “Vertical-cavity surface-emitting laser technology applications with focus on sensor and three-dimensional imaging,” Japanese Journal of Applied Physics, 57, 08P… [cited by applicant]
Stella C. Gerdemann, Katherine McAuliffe, Peter R. Blake, Daniel B.M. Haun and Robert Hepach, “the ontogeny of children's social emotions in response to (un)fairness,” 2022. Royal Society Open Science 9:191456. https://… [cited by applicant]
Robert Hepach, Amrisha Vaish and Michael Tomasello, “Novel paradigms to measure variability of behavior in early childhood: posture, gaze and pupil dilation,” (2015), Frontiers in Psychology 6:858. doi: 10.3389/fpsyg.20… [cited by applicant]
Andrew Barszczyk and Kang Lee, “Measuring Blood Pressure: from Cuff to Smartphone,” Current Hypertension Reports. (2019) 21:84 https://doi.org/10.1007/s11906-019-0990-3. [cited by applicant]
Deye Yang, Guomin Xiao, Jing Wei and Hong Luo, “Preliminary assessment of video-based blood pressure measurement according to ANSI/AAMI/ISO81060-2:2013 guideline accuracy criteria: Anura smartphone app with transdermal … [cited by applicant]
Andrew Barszczyk, Weihong Zhou and Kang Lee, “Aim and Transdermal Optical Imaging,” Springer Nature Switzerland AG 2021, N. Lidstromer, H. Ashrafian (eds.), Artificial Intelligence in Medicine, https://doi.org/10.1007/9… [cited by applicant]
Hudda Salih, Si Jia Wu, Evgueni Kabakov, Kang Lee and Weihong Zhou, “Smartphone-based Identification of critical levels of glycated hemoglobin A1c using transdermal optical imaging,” Primary Research Article published i… [cited by applicant]
“Effects of diabetes on the body and organs,” https://www.medicalnewstoday.com/articles/317483#faq (printed from the web Oct. 13, 2022). [cited by applicant]
Jiehui Sun, Keqin Han, Miao Xu, Lujuan Li, Jin Qian, Li Li and Xuejin Li, “Blood viscosity in subjects with type 2 diabetes Mellitus: Roles of hyperglycemia and elevated plasma fibrinogen,” 2022, Frontiers in Physiology… [cited by applicant]
Ying Li, Xiu-Xia Tian, Tiemin Liu, Rui-Tao Wang, “Association between whole blood viscosity and arterial stiffness in patients with type 2 diabetes mellitus,” Springer published on-line Oct. 14, 2014, Endocrine, DOI: 10… [cited by applicant]
G.E. McVeigh, “Arterial Compliance in Hypertension and Diabetes mellitus, ” American Journal of Nephrology. 1996; 16:217-222. [cited by applicant]
“Compliance (physiology),” from Wikipedia, https://en.wikipedia.org/wiki/Compliance_(physiology), printed from web Jan. 10, 2023. [cited by applicant]
Chenguang Zhao, Dongwei Li, Yuanjun Kong, Hongyu Liu, Yiqing Hu, Haijing Niu, Ole Jensen, Kiaoli Li, Hanli Liu, Yan Song, “Transcranial photobiomodulation enhances visual working memory capacity in humans,” Scientific A… [cited by applicant]
Fabrizio Dos Santos Cardoso, Farzad Salehpour, Norberto Cysne Coimbra, Francisco Gonzalez-Lima and Sergio Gomes Da Silva, “Photobiomodulation for the treatment of neuroinflammation: A systematic review of controlled lab… [cited by applicant]
Fabrizio Dos Santos Cardoso, Douglas W. Barrett, Zachary Wade, Sergio Gomes Da Silva and F. Gonzalez-Lima, “Photobiomodulation for the aging brain,” Ageing Research Reviews. 70 (2021) 101415. [cited by applicant]
Fabrizio Dos Santos Cardoso, Farzad Salehpour, Norberto Cysne Coimbra, Francisco Gonzalez-Lima and Sergio Gomes Da Silva, “Photobiomodulation of cytochrome c oxidase by chronic transcranial laser in young and aged brain… [cited by applicant]
Agnes S. Chan, Tsz-Lok Lee, Michael R. Hamblin, Mei-Chun Cheung, “Photoneuromodulation makes a difficult cognitive task less arduous,” www.nature.com/scientificreports. Science Reports (2021) 11:13688; https://doi.org/1… [cited by applicant]
Celeste L. Saucedo, Emily C. Courtois, Zachary S. Wade, Meghan N. Kelley, Nusha Kheradbin, Douglas W. Barrett, F. Gonzalez-Lima, “Transcranial laser simulation: Mitochondrial and cerebrovascular effects in younger and o… [cited by applicant]
“The science behind how laser therapy works.” https://lightforcemedical.com/photobiomodulation-therapy-pbm/, printed from the web Jan. 18, 2023. [cited by applicant]
Meeri Kim, “Can low-power light improve working memory?” https://www.optica-opn.org/home/newsroom/2022/december/can_low-power_light_improve_working_memory/, printed from web Jan. 18, 2023. [cited by applicant]
Christoph F. Geissler, Jorn Schneider, Christian Frings, “Shedding light on the prefrontal correlates of mental workload in simulated driving: a functional near-infrared spectroscopy study,” www.nature.com/scientificrep… [cited by applicant]
Marie Cheour, “Parts of the brain used while driving,” written Dec. 5, 2018, https://healthfully.com/parts-of-the-brain-involved-with-hearing-4113334.html, printed from web Jan. 22, 2023. [cited by applicant]
Hannah J. Foy, Patrick Runham, Peter Chapman, “Prefrontal cortex activation and young driver behavior: a fNIRS study,” (2016) PLoS One 11 (5): e0156512. doi: 10.1371/journal.pone.0156512. [cited by applicant]
Kayoko Yoshino, Noriyuki Oka, Kouji Yamamoto, Hideki Takahashi, and Toshinori Kato, “Correlation of prefrontal cortical activation with changing vehicle speeds in actual driving: a vector-based functional near-infrared … [cited by applicant]
Fangqing Zhengren, George Chernyshov, Dingding Zheng, “Cognitive load assessment from facial temperature using smart eyewear,” MDR, Keio University (2020). [cited by applicant]
Naomi P. Friedman and Trevor W. Robbins, “The role of prefrontal cortex in cognitive control and execution function,” Neuropsychopharmacology (2022) 47:72-89; https://doi.org/10.1038/s441386-021-01132-0. [cited by applicant]
Kanji Matsukawa, Ryota Asahara, Miho Yoshikawa and Kana Endo, “Deactivation of the prefrontal cortex during exposure to pleasantly-charged emotional challenge,” www.nature.com/scientificreports. (2018) 8:14540; doi: 10.… [cited by applicant]
Hugo Mitre-Hernandez, Jorge Sanchez-Rodriguez, Sergio Nava-Munoz and Carlos Lara-Alvarez, “Classifying the difficulty levels of working memory tasks by using pupillary response,” PeerJ 10:e12864; http://doi.org/10.7717/… [cited by applicant]
Krzysztof Krejtz, Andrew T. Duchowski, Anna Niedzielska, Cezary Biele, Izabela Krejtz, “Eye tracking cognitive load using pupil diameter and microsaccades with fixed gaze,” (2018) PLoS One 13 (9): e0203629. https://doi.… [cited by applicant]
Hamidur Rahmah, Mobyen Uddin Ahmed, Shaibal Barua, Peter Funk and Shahina Begum, “Vision-based driver's cognitive load classification considering eye movement using machine learning and deep learning,” Sensors, 2021, 21… [cited by applicant]
Product Document by AMS OSRAM on “Belago 1.1 Dot-Pattern Infrared Illuminator” Datasheet DS000676, Apr. 21, 2021. [cited by applicant]
Inter Partes Review No. 2021-00453; Decision Granting Institution of Inter Partes Review of U.S. Pat. No. 10,517,484, [cited by applicant]
Inter Partes Review No. IPR2020-00175; Judgement Final Written Decision of U.S. Patent No. 10, 188,299; [cited by applicant]