Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions
Recent studies in computer vision have shown that, while practically invisible to a human observer, skin color changes due to blood flow can be captured on face videos and, surprisingly, be used to estimate the heart rate (HR). While considerable progress has been made in the last few years, still many issues remain open. In particular, state-of-the-art approaches are not robust enough to operate in natural conditions (e.g. in case of spontaneous movements, facial expressions, or illumination changes). Opposite to previous approaches that estimate the HR by processing all the skin pixels inside a fixed region of interest, we introduce a strategy to dynamically select face regions useful for robust HR estimation. The present approach, inspired by recent advances on matrix completion theory, allows us to predict the HR while simultaneously discover the best regions of the face to be used for estimation. Thorough experimental evaluation conducted on public benchmarks suggests that the proposed approach significantly outperforms state-of-the-art HR estimation methods in naturalistic conditions.
1. A method of determining heart rate through observation of a human face, comprising:
acquiring with at least one automated camera, a time series of images of a human face, wherein the time series of images are subject to variations between respective images of the time series in illumination and facial movements;
adaptively selecting, with the at least one automated processor, a subset of the regions of interest of respective images of the time series of images of the human face, that exhibit a more statistically reliable heart-rate-determined variation than a non-selected subset of regions of the respective images of the human face;
based at least on the adaptively selected subset of regions of interest of the respective images of the time series of the human face that exhibit the reliable heart-rate-determined variation, determining a heart rate, and updating the adaptively selected subset of the regions of interest that exhibit the reliable heart-rate-determined variation; and
outputting a signal corresponding to the determined heart rate.
2. The method according to claim 1 , wherein the regions of interest are selected according to at least matrix completion theory.
3. The method according to claim 1 , wherein the heart rate is determined based on at least matrix completion theory.
4. The method according to claim 1 , wherein the selected subset is selected dependent on at least a noise parameter of respective features of the time series of images.
5. The method according to claim 1 , wherein the selected subset is selected dependent on at least a movement of the human face represented in the time series of images.
6. The method according to claim 1 , wherein the selected subset is selected dependent on at least changes represented in the time series of images which represent human facial expressions.
7. The method according to claim 1 , further comprising tracking the face in the time series of images to follow rigid head movements.
8. The method according to claim 1 , further comprising detecting chrominance features from the time series of images comprising video images, and assessing the heart rate-determined variation based on the detected chrominance features.
9. The method according to claim 1 , wherein the adaptively selected subset of the regions of interest exhibit the reliable heart-rate-determined variation through an entire period of heart rate estimation.
10. The method according to claim 1 , wherein the reliable heart-rate-determined variation is a variation in chrominance.
11. The method according to claim 1 , wherein the heart rate is determined in a process employing a cardiac cycle responsive filter.
12. The method according to claim 1 , further comprising simultaneously recovering an unknown low-rank matrix and an underlying data mask, corresponding to most statistically reliable heart-rate-determined variation observations of the human face according to a reliability statistic.
13. A method of determining heart rate from video images, comprising:
processing, with the at least one automated processor, a stream of video images of a face from at least one automated camera, to extract a plurality of face regions;
computing chrominance features of the plurality of face regions, with at least one automated processor;
jointly estimating an underlying low-rank feature matrix and a mask of a selected subset of the plurality of face regions which have a higher statistical reliability than a non-selected subset of the plurality of face regions, using a self-adaptive matrix completion algorithm, with the at least one automated processor; and
computing the heart rate from a signal estimate provided by the self-adaptive matrix completion algorithm, with the at least one automated processor.
14. The method according to claim 13 , wherein said processing comprises warping a representation of the face into rectangles using a piece-wise linear warping procedure, and dividing rectangles into a grid containing the plurality of face regions.
15. The method according to claim 14 , the selected subset of the plurality of face regions being further selected to be robust to facial movements and expressions, while being sufficiently discriminant to account for changes in skin color responsive to cardiac cycle variation for said computing the heart rate.
16. The method according to claim 13 , wherein said computing chrominance features comprises:
for each pixel, computing a chrominance signal C as a linear combination of two signals X f and Y f , such that C=X f −αY f , where
α
=
σ
(
X
f
)
σ
(
Y
f
)
and σ(X f ), σ(Y f ) denote the standard deviations of X f , Y f ;
band-pass filtering signals the signals X and Y to obtain X f , Y f respectively, where X=3R n −2G n , Y=1.5R n +G n −1.5B n and R n , G n and B n are the normalized values of the individual color channels, wherein the color combination coefficients to derive X and Y are computed using a skin-tone standardization approach;
and, for each region r=1, . . . ,R, computing the final chrominance features averaging the values of the chrominance signals over all the pixels.
17. The method according to claim 13 , wherein said jointly estimating comprises enforcing a detection of chrominance feature variations that occur within a heart-rate frequency range.
18. The method according to claim 13 , wherein said jointly estimating comprises masking extracted regions of the plurality of face regions dependent on at least facial movement dependent changes.
19. The method according to claim 13 , wherein said jointly estimating comprises determining a local standard deviation over time of each extracted region of the plurality of face regions.
20. The method according to claim 13 , wherein said jointly estimating comprises employing, by the at least one automated processor, an alternating direction method of multipliers (ADMM), which solves an optimization problem by alternating a direction of the optimization while keeping other directions fixed.
21. The method according to claim 20 , wherein the solving of the optimization problem comprises repetitively performing the following three steps until convergence:
E/M-step
with fixed F and Z, obtaining optimal values of E and M by solving:
min
E
v
E
*
+
ρ
2
E
-
F
+
ρ
-
1
Z
ℱ
2
.
(
6
)
min
M
M
°
(
F
-
C
)
ℱ
2
-
β
M
1
+
μ
M
-
M
~
ℱ
2
(
8
)
F-step
with fixed E, Z and M, determining the optimal value of F by solving:
min
F
M
°
(
F
-
C
)
ℱ
2
+
γ
Tr
(
FLF
⊤
)
+
ρ
2
F
-
E
-
ρ
-
1
Z
ℱ
2
(
11
)
Z-step
determining value of Z:
Z*=Z +ρ( E−F ), (14)
where the right-hand side represent the current values.
22. The method according to claim 21 , further comprising determining the largest singular value of E, which encodes the heart rate information, by the at least one automated processor.
23. A system for determining cardiac contraction timing from video images, comprising:
an input port configured to receive a time sequence of images of a human face from an automated camera;
at least one automated processor, configured to:
process the time sequence of images of the human face to extract a plurality of facial regions;
compute heartbeat-induced time-varying features of the respective plurality of facial regions;
determine a respective statistical parameter for heartbeat-induced time-varying features of the respective plurality of facial regions;
adaptively select, based on the determined respective statistical parameter, a dynamically changing subset of the plurality of facial regions having a higher statistical reliability than a non-selected subset; and
compute a cardiac contraction timing based on at least the respective heartbeat-induced time-varying features of the respective selected subset of the plurality of facial regions; and
an output port configured to convey a signal responsive to the cardiac contraction timing.