Detecting abnormalities in vital signs of subjects of videos
View Patent ↗Detecting abnormalities in vital signs of subjects of videos is provided. Aspects of the present disclosure include methods, apparatuses, and systems to detect and measure vital sign information of one or more human subjects of a video and detect abnormalities in the vital sign information. In some examples, such abnormalities can be used to indicate video data is likely altered or fraudulent. In this regard, imaging photophlethysmography (IPPG) and advanced signal processing techniques, including adaptive color beamforming, can be used to extract the vital signs of the video subjects.
1. A method for detecting abnormalities in video data, the method comprising:
receiving video data comprising a plurality of color channels, the plurality of color channels comprising a red color channel, a green color channel, and a blue color channel;
beamforming each of the plurality of color channels in a region of interest on skin of a subject into a color time-series to enhance vital sign information, the beamforming based on an equation:
[
R
j
i
;
G
j
i
;
B
j
i
]
=
[
∑
ROI
,
k
pixel
k
r
e
d
❘
"\[LeftBracketingBar]"
ROI
❘
"\[RightBracketingBar]"
;
∑
ROI
,
k
pixel
k
g
r
e
e
n
❘
"\[LeftBracketingBar]"
ROI
❘
"\[RightBracketingBar]"
;
∑
ROI
,
k
pixel
k
b
l
u
e
❘
"\[LeftBracketingBar]"
ROI
❘
"\[RightBracketingBar]"
]
,
where R J i denotes a red color channel value at an i-th processing interval at a j-th index; G j i denotes a green color channel value at the i-th processing interval at the j-th index; B j i denotes a blue color channel value at the i-th processing interval at the j-th index; and ROI denotes a number of pixels in the region of interest;
extracting the vital sign information of the subject from the color time-series, the extracting including applying color beamforming weights to the color time-series, wherein the color beamforming weights are based on a spectral energy distribution of the color time-series within a frequency range of interest using a constructed spectral covariance matrix;
analyzing the vital sign information; and
determining whether an abnormality occurs in the analyzed vital sign information.
2. The method of claim 1 , wherein the video data comprises a plurality of frames.
3. The method of claim 2 , further comprising tracking movement of the region of interest across the plurality of frames.
4. The method of claim 2 , further comprising spatially averaging changes in each of the plurality of color channels in the region of interest across the plurality of frames.
5. The method of claim 4 , further comprising adaptively beamforming the spatially averaged changes in each of the plurality of color channels to enhance indications of a pulse of the subject.
6. The method of claim 1 , further comprising applying a face recognition algorithm to the video data to determine the region of interest.
7. The method of claim 6 , wherein the face recognition algorithm is a multiple-face recognition algorithm which determines additional regions of interest.
8. The method of claim 1 , wherein the vital sign information comprises a heart rate of the subject and the frequency range of interest is a frequency range of the heart rate.
9. The method of claim 8 , wherein analyzing the vital sign information comprises analyzing variations in the heart rate of the subject.
10. The method of claim 9 , wherein determining whether the abnormality occurs in the analyzed vital sign information comprises determining whether the heart rate of the subject is abnormally constant or has an abnormal variation in beat to beat intervals.
11. The method of claim 1 , wherein the vital sign information comprises a respiration rate of the subject and the frequency range of interest is a frequency range of the respiration rate.
12. A fraud detector, comprising:
a memory; and
a signal processor coupled to the memory and configured to:
obtain video data stored in the memory and comprising a plurality of color channels, the plurality of color channels comprising a red color channel, a green color channel, and a blue color channel;
beamform each of the plurality of color channels in a region of interest on skin of a subject into a color time series to enhance vital sign information, the beamforming based on an equation:
[
R
j
i
;
G
j
i
;
B
j
i
]
=
[
∑
ROI
,
k
pixel
k
r
e
d
❘
"\[LeftBracketingBar]"
ROI
❘
"\[RightBracketingBar]"
;
∑
ROI
,
k
pixel
k
g
r
e
e
n
❘
"\[LeftBracketingBar]"
ROI
❘
"\[RightBracketingBar]"
;
∑
ROI
,
k
pixel
k
b
l
u
e
❘
"\[LeftBracketingBar]"
ROI
❘
"\[RightBracketingBar]"
]
,
where denotes a red color channel value at an i-th processing interval at a j-th index; G J i denotes a green color channel value at the i-th processing interval at the j-th index; B j i denotes a blue color channel value at the i-th processing interval at the j-th index; and ROI denotes a number of pixels in the region of interest;
extract the vital sign information of the subject from the color time-series, the extracting comprising applying color beamforming weights to the color time-series, wherein the color beamforming weights are based on a spectral energy distribution of the color time-series within a frequency range of interest using a constructed spectral covariance matrix;
analyze the vital sign information to detect an abnormality; and
if the abnormality is detected, determine the video data is fraudulent.
13. The fraud detector of claim 12 , wherein the signal processor is further configured to apply a face recognition algorithm to the video data to determine and track the region of interest over time.
14. The fraud detector of claim 13 , wherein the signal processor is further configured to extract the vital sign information by:
tracking changes in each of the plurality of color channels in the region of interest over time; and
adaptively beamforming the changes in each of the plurality of color channels to produce the vital sign information.
15. The fraud detector of claim 12 , wherein the signal processor is further configured to analyze the vital sign information to detect the abnormality by analyzing variations in the vital sign information of the subject.
16. A method for detecting fraudulent video data, the method comprising:
receiving video data comprising a plurality of color channels, the plurality of color channels comprising a red color channel, a green color channel, and a blue color channel;
beamforming each color channel in the plurality of color channels in a region of interest on skin of a subject into a color time-series to enhance vital sign information, the beamforming based on an equation:
[
R
j
i
;
G
j
i
;
B
j
i
]
=
[
∑
ROI
,
k
pixel
k
r
e
d
❘
"\[LeftBracketingBar]"
ROI
❘
"\[RightBracketingBar]"
;
∑
ROI
,
k
pixel
k
g
r
e
e
n
❘
"\[LeftBracketingBar]"
ROI
❘
"\[RightBracketingBar]"
;
∑
ROI
,
k
pixel
k
b
l
u
e
❘
"\[LeftBracketingBar]"
ROI
❘
"\[RightBracketingBar]"
]
,
where R J i denotes a red color channel value at an i-th processing interval at a j-th index; G j i denotes a green color channel value at the i-th processing interval at the j-th index; B j i denotes a blue color channel value at the i-th processing interval at the j-th index; and ROI denotes a number of pixels in the region of interest;
extracting the vital sign information of the subject from the color time-series, the extracting comprising applying color beamforming weights to the color time-series, wherein the color beamforming weights are based on a spectral energy distribution of the color time-series within a frequency range of interest using a constructed spectral covariance matrix;
analyzing the vital sign information to detect an abnormality; and
determining the video data is fraudulent based on the abnormality.
17. The method of claim 16 , wherein extracting the vital sign information of the subject comprises:
tracking changes in each of the plurality of color channels in the region of interest over time; and
adaptively beamforming the changes in each of the plurality of color channels to produce the vital sign information.
18. The method of claim 17 , wherein determining the video data is fraudulent is further based on background color changes.
19. The method of claim 17 , wherein determining the video data is fraudulent is further based on at least one of edge detection or edge motion.