Video based detection of pulse waveform
The video based detection of pulse waveform includes systems, devices, methods, and computer-readable instructions for capturing a video stream including a sequence of frames, processing each frame of the video stream to spatially locate a region of interest, cropping each frame of the video stream to encapsulate the region of interest, processing the sequence of frames, by a 3-dimensional convolutional neural network, to determine the spatial and temporal dimensions of each frame of the sequence of frames and to produce a pulse waveform point for each frame of the sequence of frames, and generating a time series of pulse waveform points to generate the pulse waveform of the subject for the sequence of frames.
1. A computer-implemented method for generating a pulse waveform, the computer-implemented method comprising:
capturing a video stream including a sequence of frames;
processing each frame of the video stream to spatially locate a region of interest;
cropping each frame of the video stream to encapsulate the region of interest;
processing the sequence of frames, by a 3-dimensional convolutional neural network, to determine the spatial and temporal dimensions of each frame of the sequence of frames and to produce a pulse waveform point for each frame of the sequence of frames;
modifying the temporal dimension of at least one frame with one or more dilations; and
generating a time series of pulse waveform points to generate the pulse waveform of a subject for the sequence of frames.
2. The computer-implemented method according to claim 1 , wherein the video stream includes one or more of a visible-light video stream, a near-infrared video stream, and a thermal video stream of a subject.
3. The computer-implemented method according to claim 2 , further comprising:
combining at least two of the visible-light video stream, the near-infrared video stream, and the thermal video stream into a fused video stream.
4. The computer-implemented method according to claim 3 , wherein the visible-light video stream, the near-infrared video stream, and/or the thermal video stream are combined according to a synchronization device.
5. The computer-implemented method according to claim 1 , wherein the cropping includes each frame being downsized by bi-cubic interpolation to reduce the number of image pixels.
6. The computer-implemented method according to claim 1 , wherein the region of interest includes a face.
7. The computer-implemented method according to claim 1 , wherein the region of interest includes two or more body parts.
8. The computer-implemented method according to claim 1 , further comprising:
partitioning the sequence of frames into partially overlapping subsequences,
wherein a first subsequence of frames overlaps with a second subsequence of frames.
9. The computer-implemented method according to claim 8 , further comprising:
applying a Hann function to each subsequence;
adding the overlapping subsequences to generate the pulse waveform.
10. The computer-implemented method according to claim 1 , further comprising:
calculating a heart rate or heart rate variability based on the pulse waveform.
11. A system for generating a pulse waveform, the system comprising:
a processor; and
a memory storing one or more programs for execution by the processor, the one or more programs including instructions for:
capturing a video stream including a sequence of frames;
processing each frame of the video stream to spatially locate a region of interest;
cropping each frame of the video stream to encapsulate the region of interest;
processing the sequence of frames, by a 3-dimensional convolutional neural network, to determine the spatial and temporal dimensions of each frame of the sequence of frames and to produce a pulse waveform point for each frame of the sequence of frames;
modifying the temporal dimension of at least one frame with one or more dilations; and
generating a time series of pulse waveform points to generate the pulse waveform of a subject for the sequence of frames.
12. The system according to claim 11 , wherein the video stream includes one or more of a visible-light video stream, a near-infrared video stream, and a thermal video stream of a subject.
13. The system according to claim 12 , further comprising:
combining at least two of the visible-light video stream, the near-infrared video stream, and the thermal video stream into a fused video stream.
14. The system according to claim 13 , wherein the visible-light video stream, the near-infrared video stream, and/or the thermal video stream are combined according to a synchronization device.
15. The system according to claim 11 , wherein the cropping includes each frame being downsized by bi-cubic interpolation to reduce the number of image pixels.
16. The system according to claim 11 , wherein the region of interest includes a face.
17. The system according to claim 11 , wherein the region of interest includes two or more body parts.
18. The system according to claim 11 , further comprising:
partitioning the sequence of frames into partially overlapping subsequences, wherein a first subsequence of frames overlaps with a second subsequence of frames.
19. The system according to claim 18 , further comprising:
applying a Hann function to each subsequence;
adding the overlapping subsequences to generate the pulse waveform.
20. The system according to claim 11 , further comprising:
calculating a heart rate or heart rate variability based on the pulse waveform.
21. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to generate a pulse waveform, the instructions comprising:
capturing a video stream including a sequence of frames;
processing each frame of the video stream to spatially locate a region of interest;
cropping each frame of the video stream to encapsulate the region of interest;
processing the sequence of frames, by a 3-dimensional convolutional neural network, to determine the spatial and temporal dimensions of each frame of the sequence of frames and to produce a pulse waveform point for each frame of the sequence of frames;
modifying the temporal dimension of at least one frame with one or more dilations; and
generating a time series of pulse waveform points to generate the pulse waveform of a subject for the sequence of frames.