IP Library › Granted Patent US 11,471,084
Granted Patent B2
US 11,471,084 · App. 16/729,879 · Granted Oct 18, 2022

Image processing method and corresponding system

Inventors: Francesco Rundo (Gravina di Catania, IT); Francesca Trenta (Camporotondo Etneo, IT); Sabrina Conoci (Tremestieri Etneo, IT); Sebastiano Battiato (Aci Catena, IT)
Assignee: STMicroelectronics S.R.L.
A61B5/18A61B5/0255A61B5/02405A61B5/02416A61B5/6893A61B5/6898A61B5/7267A61B5/7278A61B5/746B60W40/08B60W50/14B60W60/0051G06N3/0445G06N3/08G06V20/597G06V40/165G06V40/166G06V40/167A61B5/0077A61B2576/02B60W2050/143G06K9/6262G06N3/0454
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,471,084
App. No.
16/729,879
Granted
Oct 18, 2022
Kind
B2
Abstract

A method includes receiving a video signal that comprises a time series of images of a face of a human, wherein the images in the time series of images comprise a set of landmark points in the face, applying tracking processing to the video signal to reveal variations over time of at least one image parameter at the set of landmark points in the human face, generating a set of variation signals indicative of variations revealed at respective landmark points in the set of landmark points, applying processing to the set of variation signals, the processing comprising artificial neural network processing to produce a reconstructed PhotoPletysmoGraphy (PPG) signal, and estimating a heart rate variability of a variable heart rate of the human as a function of the reconstructed PPG signal.

Claims (34)

1. A method, comprising:

receiving a video signal that comprises a time series of images of a face of a human, wherein the images in the time series of images comprise a set of landmark points in the face;

applying tracking processing to the video signal to reveal variations over time of at least one image parameter at the set of landmark points in the human face;

generating a set of variation signals indicative of variations revealed at respective landmark points in the set of landmark points, the set of variation signals being indicative of heart pulsatile activity occurring with a variable heart rate;

applying processing to the set of variation signals, the processing comprising artificial neural network processing to produce a reconstructed PhotoPletysmoGraphy (PPG) signal, wherein applying the processing to the set of variation signals comprises training at least one artificial neural network over a training set of signals produced while sampling a sample set of PPG signals, wherein the training comprises reducing mismatch between a set of reconstructed signals and the sample set of PPG signals; and

estimating a heart rate variability of a variable heart rate of the human as a function of the reconstructed PPG signal.

2. The method of claim 1 , wherein the artificial neural network processing of the set of variation signals comprises long short-term memory processing.

3. The method of claim 1 , further comprising calculating a driver attention level indicator from the estimated heart rate variability.

4. The method of claim 3 , wherein the method is implemented on board a vehicle driven by a driver and wherein the video signal received comprises a time series of images of the face of the driver.

5. The method of claim 4 , wherein the video signal is indicative of the heart pulsatile activity of the heart of the driver and the driver attention level indicator is indicative of drowsiness of the driver.

6. The method of claim 5 , further comprising issuing a drowsiness alert to the driver as a function of the driver attention level indicator.

7. The method of claim 5 , further comprising causing a driver assistance system to take over control of the vehicle as a function of the driver attention level indicator.

8. The method of claim 1 , wherein reducing the mismatch comprises reducing the mismatch between minimum points of respective signals.

9. The method of claim 1 , wherein applying processing to the set of variation signals, processing comprises obtaining a relative variation of the at least one image parameter between subsequent images in the video signal.

10. A non-transitory storage medium storing a computer program loadable in a memory of at least one processing circuit, the computer program including software code for executing the steps of the method of claim 1 when executed by at least one processing circuit.

11. A system, comprising:

a video-recording device configured to record a video signal that comprises a time series of images of a face of a human, wherein the images in the time series of images comprise a set of landmark points in the face; and

processing circuitry coupled to the video-recording device to receive the video signal therefrom, the processing circuitry comprising artificial neural network processing circuits and being configured to estimate heart rate variability of a variable heart rate as a function of a reconstructed PhotoPletysmoGraphy (PPG) signal by applying tracking processing to the video signal to reveal variations over time of at least one image parameter at the set of landmark points in the human face, generating a set of variation signals indicative of variations revealed at respective landmark points in the set of landmark points, the set of variation signals being indicative of heart pulsatile activity occurring with a variable heart rate, applying artificial neural network processing to the set of variation signals to produce the reconstructed PPG signal, and estimating a heart rate variability of a variable heart rate of the human as a function of the reconstructed PPG signal, wherein applying the artificial neural network processing comprises training at least one artificial neural network using a training set of signals, wherein the training comprises reducing mismatch between a set of reconstructed signals produced by the artificial neural network during the training and a sample set of PPG signals corresponding to the training set of signals.

12. The system of claim 11 , comprising PPG sensing circuitry configured to sense the sample set of PPG signals included in the training set of signals, the sample set of PPG signals sensed over a given time duration indicative of heart pulsatile activity occurring with a variable heart rate, the PPG sensing circuitry configured to be disabled at the expiry of a given time duration.

13. The system of claim 12 , wherein the video-recording device for recording the video signal is a smart-phone comprising a camera.

14. The system of claim 13 , wherein the camera comprises a low frame-rate camera.

15. The system of claim 11 , wherein the system is installed in a vehicle with a driver assistance device that is configured to operate as a function of the estimated heart rate variability.

16. The system of claim 11 , wherein the processing circuitry is further configured to calculate, based on the reconstructed PPG signal, an indicator indicative of drowsiness of the human.

17. A system, comprising:

a video-recording device configured to record a video signal that comprises a time series of images of a face of a human, wherein the images in the time series of images comprise a set of landmark points in the face;

processing circuitry; and

a non-transitory storage medium storing a computer program that includes software code that, when executed by the processing circuitry, causes the processing circuitry to perform a method comprising:

applying tracking processing to the video signal to reveal variations over time of at least one image parameter at the set of landmark points in the human face;

generating a set of variation signals indicative of variations revealed at respective landmark points in the set of landmark points, the variation signals being indicative of heart pulsatile activity occurring with a variable heart rate;

applying processing to the set of variation signals, the processing comprising artificial neural network processing to produce a reconstructed PhotoPletysmoGraphy (PPG) signal, wherein applying the processing to the set of variation signals comprises training an artificial neural network using a training set of signals, wherein the training comprises reducing mismatch between a set of reconstructed signals produced by the artificial neural network during the training and a sample set of PPG signals corresponding to the training set of signals; and

estimating a heart rate variability of a variable heart rate of the human as a function of the reconstructed PPG signal.

18. The system of claim 17 , wherein the artificial neural network processing of the set of variation signals comprises long short-term memory processing.

19. The system of claim 17 , further comprising calculating a driver attention level indicator from the estimated heart rate variability.

20. The system of claim 17 , wherein applying processing to the set of variation signals, processing comprises obtaining a relative variation of the at least one image parameter between subsequent images in the video signal.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 30, 2019
From: RUNDO, FRANCESCO; TRENTA, FRANCESCA; CONOCI, SABRINA; BATTIATO, SEBASTIANO
To: STMICROELECTRONICS S.R.L.
Reel/Frame 051388/0504 →
Priority Claims (1)
IT 102019000000133 · Jan 7, 2019 · national
Continuity (1)
Related Publication 20200214614A1 · Jul 9, 2020
Cited By (1)
US 12,521,050