IP Library › Granted Patent US 12,376,766
Granted Patent B2
US 12,376,766 · App. 17/451,470 · Granted Aug 5, 2025

System and method for determining heart beat features

Inventors: Andrey Viktorovich Filimonov (Kamenki, RU); Ivan Sergeevich Shishalov (Nizhniy Novgorod, RU); Roman Aleksandrovich Ershov (Nizhny Novgorod, RU); Andrey Sergeevich Shilov (Nizhny Novgorod, RU)
Assignee: HARMAN BECKER AUTOMOTIVE SYSTEMS GMBH
A61B5/1176A61B5/0077A61B5/02108A61B5/02405A61B5/026A61B5/18
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,376,766
App. No.
17/451,470
Granted
Aug 5, 2025
Kind
B2
Abstract

Method for determining one or more heart beat features, the method comprising: capturing a series of visible light images of a face of a user by a visible light camera; capturing, during capturing the series of the visible light images, at least one infrared image of the face by an infrared camera; detecting, on the at least one infrared image, one or more first face landmarks indicative of a location of the face; locating one or more skin patches on the visible light image series; and determining one or more heart beat features based on the skin patches; characterized in that locating one or more skin patches comprises determining a position of the skin patches relative to the one or more first face landmarks.

Claims (30)

1. A system for determining one or more heart beat features, the system comprising:

a visible light camera;

an infrared camera; and

a computing device;

wherein:

the visible light camera is operable to capture a series of visible light images of a face of a user;

the infrared camera is operable to capture, during capturing the series of the visible light images, at least one infrared image of the face; and

the computing device implemented in hardware is operable to:

detect, on the at least one infrared image, one or more first face landmarks;

detect, on at least one of the visible light images, one or more second face landmarks;

determine a first confidence value of the one or more first face landmarks;

determine a second confidence value of the one or more second face landmarks, wherein the second confidence value is expressed on the same scale as the first confidence value;

compare the first confidence value to the second confidence value;

in response to the first confidence value being greater than the second confidence value, determine a location of one or more skin patches on the series of visible light images based on the one or more first face landmarks;

in response to the second confidence value being equal to or greater than the first confidence value, determine the location of the one or more skin patches on the series of visible light images based on the one or more second face landmarks; and

determine the one or more heart beat features based on the determined location of the one or more skin patches.

2. The system of claim 1 ,

wherein the second confidence value is indicative of a confidence of the one or more second face landmarks on one of the one or more visible light images relative to a confidence of the one or more first face landmarks on one of the at least one infrared image.

3. The system of claim 1 ,

wherein the second confidence value is indicative of a confidence of the one or more second face landmarks on a sequence of visible light images of the one or more visible light images relative to a confidence of the one or more first face landmarks on a sequence of infrared images of the at least one infrared image.

4. The system of claim 1 , wherein, to determine the location of the one or more skin patches on the series of visible light images based on the one or more second face landmarks, the computing device is further operable to transform the one or more first face landmarks into a reference frame of the visible light camera.

5. The system of claim 1 , wherein determining the one or more heart beat features comprises determining a blood flow intensity value.

6. The system of claim 1 , wherein determining the one or more heart beat features comprises determining a blood pulse wave.

7. The system of claim 6 , wherein the blood pulse wave is determined by remote photoplethysmography (rPPG) based on a time series of values obtained from the one or more visible light images.

8. The system of claim 1 , wherein determining the one or more heart beat features comprises determining one or more of a heart inter-beat interval, a heart rate, and a heart rate variability.

9. The system of claim 1 , wherein detecting the one or more first face landmarks comprises:

detecting, on a first infrared image, one or more first preliminary face landmarks;

detecting, on a second infrared image recorded before or after the first infrared image, one or more second preliminary face landmarks; and

determining the one or more first face landmarks by interpolating between the one or more first preliminary face landmarks and the one or more second preliminary face landmarks.

10. The system of claim 1 , wherein a frame rate of the visible light camera is equal to a frame rate of the infrared camera.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 19, 2021
From: FILIMONOV, ANDREY VIKTOROVICH; SHISHALOV, IVAN SERGEEVICH; ERSHOV, ROMAN ALEKSANDROVICH; SHILOV, ANDREY SERGEEVICH
To: HARMAN BECKER AUTOMOTIVE SYSTEMS GMBH
Reel/Frame 057839/0326 →
Priority Claims (1)
WO PCT/RU2020/000555 · Oct 19, 2020 · international
Continuity (1)
Related Publication 20220117517A1 · Apr 21, 2022
References Cited (60)
US 6090051A · Marshall · 2000 [cited by applicant]
US 6102870A · Edwards · 2000 [cited by applicant]
US 6496594B1 · Prokoski · 2002 [cited by examiner]
US 7344251B2 · Marshall · 2008 [cited by applicant]
US 7435227B2 · Farbos · 2008 [cited by applicant]
US 7438418B2 · Marshall · 2008 [cited by applicant]
US 7938785B2 · Aguilar et al. · 2011 [cited by applicant]
US 8977347B2 · Mestha et al. · 2015 [cited by applicant]
US 9642536B2 · Kashef et al. · 2017 [cited by applicant]
US 9646046B2 · Sadowsky et al. · 2017 [cited by applicant]
US 9723992B2 · Senechal et al. · 2017 [cited by applicant]
US 9763573B2 · Distasi et al. · 2017 [cited by applicant]
US 9934425B2 · el Kaliouby et al. · 2018 [cited by applicant]
US 10111611B2 · el Kaliouby et al. · 2018 [cited by applicant]
US 10667723B2 · Jacquel et al. · 2020 [cited by applicant]
US 20070066916A1 · Lemos · 2007 [cited by applicant]
US 20080150734A1 · Johns · 2008 [cited by applicant]
US 20140316293A1 · Ahmad · 2014 [cited by examiner]
US 20150297142A1 · De Jaam · 2015 [cited by examiner]
US 20160228069A1 · Derkx et al. · 2016 [cited by applicant]
US 20180055391A1 · Murakami et al. · 2018 [cited by applicant]
US 20180096119A1 · Yun · 2018 [cited by examiner]
US 20180125356A1 · Yamada · 2018 [cited by applicant]
US 20180125405A1 · Yamada · 2018 [cited by applicant]
US 20180125406A1 · Yamada · 2018 [cited by applicant]
US 20180153422A1 · Watanabe · 2018 [cited by examiner]
US 20190350471A1 · Marks et al. · 2019 [cited by applicant]
US 20200214614A1 · Rundo et al. · 2020 [cited by applicant]
CN 112052830A · 2020 [cited by examiner]
CN 111611905B · 2023 [cited by examiner]
WO 2006024129A1 · 2006 [cited by applicant]
WO 2008107832A1 · 2008 [cited by applicant]
WO 2015116832A1 · 2015 [cited by applicant]
European Patent Office, Extended European Search Report Issued in Application No. 21203425.0, Mar. 9, 2022, Germany, 5 pages. [cited by applicant]
Malik, M. et al., “Heart rate variability and clinical cardiology,” British Heart Journal, vol. 71, No. 1, Jan. 1994, 4 pages. [cited by applicant]
Carney, R. et al., “Change in Heart Rate and Heart Rate Variability During Treatment for Depression in Patients with Coronary Heart Disease,” Psychosomatic Medicine, vol. 62, No. 5, Sep. 2000, 9 pages. [cited by applicant]
“Driver Workload Metrics Project—Task 2 Final Report,” NHTSA Website, Available Online at https://www.nhtsa.gov/document/driver-workload-metrics-project-task-2-final-report, Nov. 2006, 460 pages. [cited by applicant]
Sforza, E. et al., “Cardiac Variability and Heart-Rate Increment as a Marker of Sleep Fragmentation in Patients With a Sleep Disorder: a Preliminary Study,” Sleep, vol. 30, No. 1, Jan. 2007, 9 pages. [cited by applicant]
Patil, S. et al., “Extraction of Significant Patterns from Heart Disease Warehouses for Heart Attack Prediction,” IJCSNS International Journal of Computer Science and Network Security, vol. 9, No. 2, Feb. 2009, 8 pages. [cited by applicant]
Estrada, E. et al., “EEG and HRV Signal Features for Automatic Sleep Staging and Apnea Detection,” Proceedings of the 2010 20th International Conference on Electronics Communications and Computers (CONIELECOMP), Feb. 22… [cited by applicant]
Mohebbi, M. et al., “Prediction of paroxysmal atrial fibrillation based on non-linear analysis and spectrum and bispectrum features of the heart rate variability signal,” Computer Methods and Programs in Biomedicine, vo… [cited by applicant]
Huikun, H,et al., “Heart Rate Variability in Risk Stratification of Cardiac Patients,” Progress in Cardiovascular Diseases, vol. 56, No. 2, Sep. 2013, Available Online Aug. 12, 2013, 7 pages. [cited by applicant]
Tobaldini, E. et al., “Heart rate variability in normal and pathological sleep,” Frontiers in Physiology, vol. 4, No. 294, Oct. 16, 2013, 12 pages. [cited by applicant]
Moon, E. et al., “Comparative Study of Heart Rate Variability in Patients with Schizophrenia, Bipolar Disorder, Post-traumatic Stress Disorder, or Major Depressive Disorder,” Clinical Psychopharmacology and Neuroscience… [cited by applicant]
Ebrahimzadeh, E. et al., “A Novel Approach to Predict Sudden Cardiac Death (SCD) Using Nonlinear and Time-Frequency Analyses from HRV Signals,” PLOS ONE, vol. 9, No. 2, Feb. 4, 2014, 14 pages. [cited by applicant]
Li, X. et al., “Remote Heart Rate Measurement From Face Videos Under Realistic Situations,” Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 23, 2014, Columbus, Ohio, 8 pages. [cited by applicant]
Wang, W. et al., “Exploiting Spatial Redundancy of Image Sensor for Motion Robust rPPG,” IEEE Transactions on Biomedical Engineering, vol. 62, No. 2, Feb. 2015, 11 pages. [cited by applicant]
Rubin, J. et al., “Towards a Mobile and Wearable System for Predicting Panic Attacks,” UbiComp '15: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Sep. 7, 2015, Osaka, … [cited by applicant]
Lam, A. et al., “Robust Heart Rate Measurement from Video Using Select Random Patches,” Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Dec. 7, 2015, Santiago, Chile, 9 pages. [cited by applicant]
Fujiwara, K. et al., “Epileptic Seizure Prediction Based on Multivariate Statistical Process Control of Heart Rate Variability Features,” IEEE Transactions on Biomedical Engineering, vol. 63, No. 6, Jun. 2016, 11 pages. [cited by applicant]
Koenig, J. et al., “Depression and resting state heart rate variability in children and adolescents—A systematic review and meta-analysis,” Clinical Psychology Review, vol. 46, Jun. 2016, 15 pages. [cited by applicant]
Wang, W. et al., “A Novel Algorithm for Remote Photoplethysmography: Spatial Subspace Rotation,” IEEE Transactions on Biomedical Engineering, vol. 63, No. 9, Sep. 2016, 11 pages. [cited by applicant]
Monkaresi, H. et al., “Automated Detection of Engagement using Video-Based Estimation of Facial Expressions and Heart Rate,” IEEE Transactions on Affective Computing, vol. 8, No. 1, Jan. 1, 2017, 14 pages. [cited by applicant]
Wang, W. et al., “Algorithmic Principles of Remote-PPG,” IEEE Transactions on Biomedical Engineering, vol. 64, No. 79, Jul. 2017, 12 pages. [cited by applicant]
Hassan, M. et al., “Heart rate estimation using facial video: A review,” Biomedical Signal Processing and Control, vol. 38, Sep. 2017, 15 pages. [cited by applicant]
Myers, K. et al., “Heart rate variability in epilepsy: A potential biomarker of sudden unexpected death in epilepsy risk,” Epilepsia, vol. 59, No. 7, Jun. 6, 2018, 9 pages. [cited by applicant]
Balakrishnan, G. et al., “Detecting Pulse from Head Motions in Video,” Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 23, 2018, Portland, Oregon, 8 pages. [cited by applicant]
Chen, Z. et al., “Estimating Depth from RGB and Sparse Sensing,” Proceedings of the European Conference on Computer Vision (ECCV), Sep. 8, 2018, Munich, Germany, 16 pages. [cited by applicant]
“European New Car Assessment Programme (Euro NCAP) Assessment Protocol—Adult Occupant Protection,” Euro NCAP Website, Available Online at https://cdn.euroncap.com/media/43372/euro-ncap-assessment-protocol-aop-v901.pdf, … [cited by applicant]
Harford, M. et al., “Availability and performance of image-based, non-contact methods of monitoring heart rate, blood pressure, respiratory rate, and oxygen saturation: a systematic review,” Physiological Measurement, v… [cited by applicant]