Systems and methods for non-contact heart rate sensing
View Patent ↗An embodiment generally relates to systems and methods for estimating heart rates of individuals using non-contact imaging. A processing module can process multi-spectral video images of individuals and detect skin blobs within different images of the multi-spectral video images. The skin blobs can be converted into time series signals and processed with a band pass filter. Further, the time series signals can be processed to separate pulse signals from unnecessary signals. The heart rate of the individual can be estimated according to the resulting time series signal processing.
1. A method of processing data, the method comprising:
processing, by a processor, a video of an individual, wherein the video was captured using one or more of a multi-spectral video camera, a multi-spectral near infrared video camera, or a multi-spectral short wave Infrared video camera, and wherein the processing comprises:
separating the video into a sequence of image data cubes each comprising a set of more than three entries, wherein each entry of each set of entries corresponds to a spectral frequency at which the video was captured, wherein the spectral frequency of each entry of the set of entries is in a range of 700 nanometers (nm) to 1700 nm;
identifying, from the image data cubes, a set of skin blobs corresponding to the set of entries;
processing the set of skin blobs to remove non-skin areas;
generating, from the set of skin blobs, a set of time series signals;
processing the set of time series signals to estimate a heart rate of the individual; and
providing the heart rate to a device.
2. The method of claim 1 , wherein the processing further comprises:
processing the set of time series signals to identify a source signal that corresponds to a heart beat.
3. The method of claim 1 , wherein identifying the set of skin blobs comprises:
identifying the set of skin blobs from a first frame of the image data cubes; and
tracking the set of skin blobs in successive frames of the image data cubes using motion-based video tracking.
4. The method of claim 1 , wherein the processing further comprises:
processing the set of time series signals to separate independent source signals that contain a heart beat.
5. The method of claim 1 , further comprising:
capturing the video.
6. The method of claim 1 , wherein the processing further comprises:
processing the sequence of image data cubes to estimate the heart rate on a continuous basis.
7. The method of claim 1 , wherein the processing further comprises:
prefiltering the set of time series signals by a band pass filter.
8. The method of claim 1 , wherein generating the set of time series signals comprises:
computing a channel average of pixels in each skin blob of the set of skin blobs;
computing a global channel average based on the channel average of pixels in each skin blob; and
generating the set of time series signals by subtracting the global channel average from the channel average of pixels, and dividing a result of the subtraction by a global channel standard deviation.
9. The method of claim 1 , wherein processing the set of time series signals comprises performing an independent component analysis (ICA) on the set of time series signals.
10. The method of claim 1 , wherein estimating the heart rate of the individual comprises:
applying a Fast Fourier Transform (FFT) on a source signal from the set of time series signals to generate an FFT source signal; and
calculating a square of the FFT source signal; and
estimating the heart rate of the individual based on the calculation of the square of the FFT source signal.
11. The method of claim 10 , wherein estimating the heart rate of the individual based on the calculation of the square of the FFT source signal comprises:
identifying a greatest magnitude of the square of the FFT source signal within a frequency band of interest.
12. The method of claim 1 , wherein providing the heart rate to the device comprises:
outputting the heart rate of the individual to a display monitor.
13. The method of claim 1 , wherein the processing further comprises:
outputting a source signal used to estimate the heart rate on a display monitor.
14. A system for processing data, the system comprising:
one or more filters configured to filter images of an individual into multi-spectral images;
a capture device configured to capture, from the one or more filters, the multi-spectral images as a video, wherein the capture device comprises one or more of a multi-spectral video camera, a multi-spectral near infrared video camera, or a multi-spectral short wave Infrared video camera; and
a processor coupled to the capture device and configured to perform actions comprising:
processing the video, wherein the processing comprises:
separating the video into a sequence of image data cubes each comprising a set of more than three entries, wherein each entry of each set of entries corresponds to a spectral frequency at which the multi-spectral video was captured, wherein the spectral frequency of each entry of the set of entries is in a range of 700 nanometers (nm) to 1700 nm;
identifying, from the image data cubes, a set of skin blobs corresponding to the set of entries;
processing the set of skin blobs to remove non-skin areas;
generating, from the set of skin blobs, a set of time series signals;
processing the set of time series signals to estimate a heart rate of the individual; and
providing the heart rate to a device.
15. The system of claim 14 , wherein identifying the set of skin blobs comprises:
identifying the set of skin blobs from a first frame of the image data cubes; and
tracking the set of skin blobs in successive frames of the image data cubes using motion-based video tracking.
16. The system of claim 14 , wherein the processing further comprises:
processing the set of time series signals to identify a source signal that corresponds to a heart beat.
17. The system of claim 14 , wherein the processing further comprises:
processing the set of time series signals to separate independent source signals that contain a heart beat.
18. The system of claim 14 , wherein the processing further comprises:
processing the sequence of image data cubes to estimate the heart rate on a continuous basis.
19. The system of claim 14 , wherein the processing further comprises:
prefiltering the set of time series signals by a band pass filter.
20. The system of claim 14 , wherein generating the set of time series signals comprises:
computing a channel average of pixels in each skin blob of the set of skin blobs;
computing a global channel average based on the channel average of pixels in each skin blob; and
generating the set of time series signals by subtracting the global channel average from the channel average of pixels, and dividing a result of the subtraction by a global channel standard deviation.
21. The system of claim 14 , wherein processing the set of time series signals comprises performing an independent component analysis (ICA) on the set of time series signals.
22. The system of claim 14 , wherein estimating the heart rate of the individual comprises:
applying a Fast Fourier Transform (FFT) on a source signal from the set of time series signals to generate an FFT source signal; and
calculating a square of the FFT source signal; and
estimating the heart rate of the individual based on the calculation of the square of the FFT source signal.
23. The system of claim 22 , wherein estimating the heart rate of the individual based on the calculation of the square of the FFT source signal comprises:
identifying a greatest magnitude of the square of the FFT source signal within a frequency band of interest.
24. The system of claim 14 , wherein providing the heart rate to the device comprises:
outputting the heart rate of the individual to a display monitor.
25. The system of claim 14 , wherein the processing further comprises:
outputting a source signal used to estimate the heart rate on a display monitor.