Remote heart rate estimation
For remote heart rate estimation, a method detects an object of interest (OOI) in each image of a video data and tracks the OOI in each image of the video data. The method identifies a region of interest (ROI) within the OOI and generates a plurality of super pixels from a plurality of pixels in each ROI. The method further generates a super-pixel time series from the plurality of super pixels in each image and removes interfering signals from the super-pixel time series. The method further models the super-pixel time series as a super-pixel model and calculates a heart beat signal from the super-pixel model. The method calculates heart characteristics from the heart beat signal. The heart characteristics include one or more of a heart rate, an inter-beat interval, and a heart rate variability.
1. A method comprising:
detecting, by use of a processor, an object of interest (OOI) in each image of a video data;
tracking the OOI in each image of the video data;
identifying a region of interest (ROI) within the OOI;
generating a plurality of super pixels from a plurality of pixels in each ROI;
generating a super-pixel time series from the plurality of super pixels in each image;
removing interfering signals from the super-pixel time series;
modeling the super-pixel time series as a super-pixel model;
calculating a heart beat signal from the super-pixel model, wherein an ith super pixel for a time t is modeled as y i (t)=u i (t)+h i (t)+n i (t), where y i (t) is a measured pixel value, u i (t) is a background signal representing contributions to the measured pixel value due to movement and lighting variations, h i (t) is the heart beat signal, and n i (t) is a sensor noise signal representing contributions to the measured pixel value due to sensor noise; and
calculating heart characteristics from the heart beat signal, the heart characteristics comprising one or more of a heart rate, an inter-beat interval, and a heart rate variability.
2. The method of claim 1 , wherein the sensor noise signal n i (t) is assumed to be independent, identically distributed Gaussian noise.
3. The method of claim 1 , wherein the background signal u i (t) is assumed to be smooth.
4. The method of claim 3 , wherein the background signal u i (t) is modeled as a first order Markov random process.
5. The method of claim 1 , wherein the heart beat is assumed to be the same in each super pixel, wherein h i (t)=h(t) for all i.
6. The method of claim 1 , wherein the heart beat signal and a background signal are further calculated by optimizing
min
u
i
(
t
)
,
h
(
t
)
∑
i
∑
t
y
i
(
t
)
-
u
i
(
t
)
-
h
(
t
)
2
+
λ
1
[
H
|
h
]
*
+
λ
2
∑
i
∑
t
u
i
(
t
+
1
)
-
u
i
(
t
)
2
subject to [H] (i,j) =h(2L+i−j) and [h] i =h(L+T+i−1), where the sum on i is over the plurality of super pixels, and the sum on t is over the plurality of super pixels in the super pixel time series, λ 1 and λ 2 are user parameters, H is an (M+1)×(2L+1) Toeplitz matrix having (i,j) th element h(2L+i−j) for i=1, 2, . . . , M+1 and j=1, 2, . . . , 2L+1, h is an (M+1)×1 vector having i th element h(L+T+i−1) for i=1, 2, . . . , M+1, and ∥•∥ * is a nuclear norm.
7. The method of claim 1 , wherein the heart beat signal and a background signal are further calculated by optimizing
min
u
,
h
[
y
0
0
]
-
[
I
I
α
D
0
0
β
P
]
[
u
h
]
where
D
=
[
-
1
1
-
1
1
⋱
⋱
-
1
1
]
P
=
[
p
L
⋯
p
-
L
0
0
-
1
0
0
0
⋱
⋱
⋱
⋱
⋱
⋱
0
0
0
p
L
⋯
p
-
L
0
0
-
1
]
and α and β are user selectable constants selected to generate a smoothness of the background signal and a predictability of the heart beat signal, the vector u contains samples of the background signal and the vector h contains samples of the heart beat signal, the prediction coefficients p L , . . . , p −L are interpolation coefficients derived from a hypothesized period of the heart beat signal and a placement of the −1 in the P matrix is also dependent on a hypothesized period of the heart beat signal, and wherein an optimization is repeated for a series of different heart beat periods and a first heart beat period giving the smallest objective value is chosen as a period of the heart beat signal.
8. The method of claim 1 , wherein the OOI is detected using cascaded object detection on red/green/blue (RGB) pixels of the video data and the OOI is tracked with sub-pixel resolution using spatial correlation-based methods.
9. The method of claim 1 , wherein the OOI is detected and tracked using infrared band information.
10. The method of claim 1 , wherein the video data is captured by one or more of a plurality of cameras, 3-color channel camera, a multispectral n-channel camera, an infrared camera, a depth camera, a 1 pixel sensor, and a servo controlled camera.
11. The method of claim 1 , wherein each super pixel comprises one or more measured pixel values representing an average value of pixels in the ROI.
12. The method of claim 1 , wherein the heart rate is calculated using one of a machine learning analysis of the heart beat signal, a peak of a Fourier transform of the heart beat signal, a power spectral density of the heart beat signal, a zero crossing rate of the heart beat signal, and a sliding correlation analysis of the heart beat signal.
13. The method of claim 1 , wherein removing the interfering signals from the super-pixel time series comprises de-trending by modeling a background signal as a Gaussian process.
14. The method of claim 1 , wherein removing the interfering signals from the super-pixel time series comprises de-trending by decorrelating the super-pixel time series with auxiliary signals derived from a position of a facebox and from other regions in the video data.
15. The method of claim 1 , wherein removing the interfering signals comprises band pass filtering to remove signals outside a frequency band of a normal heart rate.
16. An apparatus comprising:
a camera;
a processor;
a memory storing code executable by the processor to perform:
detecting an object of interest (OOI) in each image of a video data captured by the camera;
tracking the OOI in each image of the video data;
identifying a region of interest (ROI) within the OOI;
generating a plurality of super pixels from a plurality of pixels in each ROI;
generating a super-pixel time series from the plurality of super pixels in each image;
removing interfering signals from the super-pixel time series;
modeling the super-pixel time series as a super-pixel model;
calculating a heart beat signal from the super-pixel model, wherein an ith super pixel for a time t is modeled as y i (t)=u i (t)+h i (t)+n i (t), where y i (t) is a measured pixel value, u i (t) is a background signal representing contributions to the measured pixel value due to movement and lighting variations, h i (t) is the heart beat signal, and n i (t) is a sensor noise signal representing contributions to the measured pixel value due to sensor noise; and
calculating heart characteristics from the heart beat signal, the heart characteristics comprising one or more of a heart rate, an inter-beat interval, and a heart rate variability.
17. The apparatus of claim 16 , wherein the heart beat signal and a background signal are further calculated by optimizing
min
u
i
(
t
)
,
h
(
t
)
∑
i
∑
t
y
i
(
t
)
-
u
i
(
t
)
-
h
(
t
)
2
+
λ
1
[
H
|
h
]
*
+
λ
2
∑
i
∑
t
u
i
(
t
+
1
)
-
u
i
(
t
)
2
subject to [H] (i,j) =h(2L+i−j) and [h] i =h(L+T+i−1), where the sum on i is over the plurality of super pixels, and the sum on t is over the plurality of super pixels in the super pixel time series, λ 1 and λ 2 are user parameters, H is an (M+1)×(2L+1) Toeplitz matrix having (i,j) th element h(2L+i−j) for i=1, 2, . . . , M+1 and j=1, 2, . . . , 2L+1, h is an (M+1)×1 vector having i th element h(L+T+i−1) for i=1, 2, . . . , M+1, and ∥•∥ * is a nuclear norm.
18. The apparatus of claim 16 , wherein the heart beat signal and a background signal are further calculated by optimizing
min
u
,
h
[
y
0
0
]
-
[
I
I
α
D
0
0
β
P
]
[
u
h
]
where
D
=
[
-
1
1
-
1
1
⋱
⋱
-
1
1
]
P
=
[
p
L
…
p
-
L
0
0
-
1
0
0
0
⋱
⋱
⋱
⋱
⋱
⋱
0
0
0
p
L
…
p
-
L
0
0
-
1
]
and α and β are user selectable constants selected to generate a smoothness of the background signal and a predictability of the heart beat signal, the vector u contains samples of the background signal and the vector h contains samples of the heart beat signal, the prediction coefficients p L , . . . , p −L are interpolation coefficients derived from a hypothesized period of the heart beat signal and a placement of the −1 in the P matrix is also dependent on a hypothesized period of the heart beat signal, and wherein an optimization is repeated for a series of different heart beat periods and a first heart beat period giving the smallest objective value is chosen as a period of the heart beat signal.
19. A program product comprising a non-transitory computer readable storage medium storing code executable by a processor to perform:
detecting an object of interest (OOI) in each image of a video data captured by the camera;
tracking the OOI in each image of the video data;
identifying a region of interest (ROI) within the OOI;
generating a plurality of super pixels from a plurality of pixels in each ROI;
generating a super-pixel time series from the plurality of super pixels in each image;
removing interfering signals from the super-pixel time series;
modeling the super-pixel time series as a super-pixel model;
calculating a heart beat signal from the super-pixel model, wherein an ith super pixel for a time t is modeled as y i (t)=u i (t)+h i (t)+n i (t), where y i (t) is a measured pixel value, u i (t) is a background signal representing contributions to the measured pixel value due to movement and lighting variations, h i (t) is the heart beat signal, and n i (t) is a sensor noise signal representing contributions to the measured pixel value due to sensor noise; and
calculating heart characteristics from the heart beat signal, the heart characteristics comprising one or more of a heart rate, an inter-beat interval, and a heart rate variability.
20. The program product of claim 19 , wherein the heart beat signal and a background signal are further calculated by optimizing
min
u
i
(
t
)
,
h
(
t
)
∑
i
∑
t
y
i
(
t
)
-
u
i
(
t
)
-
h
(
t
)
2
+
λ
1
[
H
|
h
]
*
+
λ
2
∑
i
∑
t
u
i
(
t
+
1
)
-
u
i
(
t
)
2
subject to [H] (i,j) =h(2L+i−j) and [h] i =h(L+T+i−1), where the sum on i is over the plurality of super pixels, and the sum on t is over the plurality of super pixels in the super pixel time series, λ 1 and λ 2 are user parameters, H is an (M+1)×(2L+1) Toeplitz matrix having (i,j) th element h(2L+i−j) for i=1, 2, . . . , M+1 and j=1, 2, . . . , 2L +1, h is an (M+1)×1 vector having i th element h(L+T+i−1) for i=1, 2, . . . , M+1, and ∥•∥ * is a nuclear norm.