VIDEO BASED DETECTION OF PULSE WAVEFORM
The video based detection of pulse waveform includes systems, devices, methods, and computer-readable instructions for capturing a video stream including a sequence of frames, processing each frame of the video stream to spatially locate a region of interest, cropping each frame of the video stream to encapsulate the region of interest, processing the sequence of frames, by a 3-dimensional convolutional neural network, to determine the spatial and temporal dimensions of each frame of the sequence of frames and to produce a pulse waveform point for each frame of the sequence of frames, and generating a time series of pulse waveform points to generate the pulse waveform of the subject for the sequence of frames.
1 . (canceled)
2 . A computer-implemented method for generating a pulse waveform, the computer-implemented method comprising:
detecting, within a video stream comprising a sequence of frames, a region of interest corresponding to exposed skin of a subject;
extracting image data from the region of interest across a plurality of frames of the video stream;
preprocessing the extracted image data to normalize pixel values within the region of interest;
inputting the preprocessed image data into a three-dimensional convolutional neural network trained to extract physiological signals, wherein the three-dimensional convolutional neural network processes spatial and temporal features of the image data;
generating, by the three-dimensional convolutional neural network, a pulse amplitude value for each frame of the plurality of frames; and
constructing a time series from the pulse waveform points to form the pulse waveform representing the subject's physiological pulse signal.
3 . The computer-implemented method according to claim 2 , wherein the video stream includes one or more of a visible-light video stream, a near-infrared video stream, and a thermal video stream of a subject.
4 . The computer-implemented method according to claim 3 , further comprising:
combining at least two of the visible-light video stream, the near-infrared video stream, and the thermal video stream into a fused video stream.
5 . The computer-implemented method according to claim 4 , wherein the visible-light video stream, the near-infrared video stream, and/or the thermal video stream are combined according to a synchronization device.
6 . The computer-implemented method according to claim 2 , wherein the region of interest includes each frame being downsized by bi-cubic interpolation to reduce the number of image pixels.
7 . The computer-implemented method according to claim 2 , wherein the region of interest includes a face or a plurality of body parts.
8 . The computer-implemented method according to claim 2 , further comprising modifying the temporal feature of at least one frame with one or more dilations.
9 . The computer-implemented method according to claim 2 , further comprising:
partitioning the sequence of frames into partially overlapping subsequences, wherein a first subsequence of frames overlaps with a second subsequence of frames.
10 . The computer-implemented method according to claim 9 , further comprising:
applying a Hann function to each subsequence; and
adding the overlapping subsequences to generate the pulse waveform.
11 . The computer-implemented method according to claim 2 , further comprising:
calculating a heart rate or heart rate variability based on the pulse waveform.
12 . A system for generating a pulse waveform, the system comprising:
a processor; and
a memory storing one or more programs for execution by the processor, the one or more programs including instructions for:
detecting, within a video stream comprising a sequence of frames, a region of interest corresponding to exposed skin of a subject;
extracting image data from the region of interest across a plurality of frames of the video stream;
preprocessing the extracted image data to normalize pixel values within the region of interest;
inputting the preprocessed image data into a three-dimensional convolutional neural network trained to extract physiological signals, wherein the three-dimensional convolutional neural network processes spatial and temporal features of the image data;
generating, by the three-dimensional convolutional neural network, a pulse amplitude value for each frame of the plurality of frames; and
constructing a time series from the pulse waveform points to form the pulse waveform representing the subject's physiological pulse signal.
13 . The system according to claim 12 , wherein the video stream includes one or more of a visible-light video stream, a near-infrared video stream, and a thermal video stream of a subject.
14 . The system according to claim 13 , further comprising:
combining at least two of the visible-light video stream, the near-infrared video stream, and the thermal video stream into a fused video stream.
15 . The system according to claim 14 , wherein the visible-light video stream, the near-infrared video stream, and/or the thermal video stream are combined according to a synchronization device.
16 . The system according to claim 12 , wherein the region of interest includes each frame being downsized by bi-cubic interpolation to reduce the number of image pixels.
17 . The system according to claim 12 , wherein the region of interest includes a face or a plurality of body parts.
18 . The system according to claim 12 , further comprising:
partitioning the sequence of frames into partially overlapping subsequences, wherein a first subsequence of frames overlaps with a second subsequence of frames.
19 . The system according to claim 18 , further comprising:
applying a Hann function to each subsequence; and
adding the overlapping subsequences to generate the pulse waveform.
20 . The system according to claim 12 , further comprising:
calculating a heart rate or heart rate variability based on the pulse waveform.
21 . A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to generate a pulse waveform, the instructions comprising:
detecting, within a video stream comprising a sequence of frames, a region of interest corresponding to exposed skin of a subject;
extracting image data from the region of interest across a plurality of frames of the video stream;
preprocessing the extracted image data to normalize pixel values within the region of interest;
inputting the preprocessed image data into a three-dimensional convolutional neural network trained to extract physiological signals, wherein the three-dimensional convolutional neural network processes spatial and temporal features of the image data;
generating, by the three-dimensional convolutional neural network, a pulse amplitude value for each frame of the plurality of frames; and
constructing a time series from the pulse waveform points to form the pulse waveform representing the subject's physiological pulse signal.