Processing source video for real-time enhancement of a signal of interest
View Patent ↗What is disclosed is a system and method for real-time enhancement of an identified time-series signal of interest in a video that has a similar spatial and temporal structure to a given reference signal, as determined by a measure of closeness. A closeness measure is computed for pixels of each image frame of each channel of a multi-channel video to identify a time-series signal of interest. The intensity of pixels associated with that time-series signal is modified based on a product of the closeness measure and the reference signal scaled by an amplification factor. The modified pixel intensity values are provided back into the source video to generate a reconstructed video such that, upon playback of the reconstructed video, viewers thereof can visually examine the amplified time-series signal, see how it is distributed and how it propagates. The methods disclosed find their uses in remote sensing applications such as telemedicine.
1. A method for real-time enhancement of a signal of interest in a video, the method comprising:
receiving a source video acquired using a multi-channel video acquisition system with C channels where channel k=1 . . . C, said source video comprising a plurality of image frames, each frame comprising an array of pixels, each pixel having an intensity value corresponding to detected reflected energy projected by an illumination source over a wavelength range of interest;
for each pixel location (i, j) of each image frame of each channel k of said source video:
determining a measure of closeness with respect to a reference signal R(t) in both a spatial and time direction;
identifying, using said measure of closeness, a time-series signal I(t) of interest in said source video, where signal I ijk (t) appears at pixel location (i, j, k);
in response to having identified said time-series signal of interest at pixel location (i, j, k):
modifying an intensity of said pixel at said location to generate modified video data J ijk (t); and
retrofitting said modified video data into said source video to generate a reconstructed source video; and
communicating said reconstructed source video to a display device.
2. The method of claim 1 , wherein said multi-channel video acquisition system comprises any combination of: a monochrome video camera, a color video camera, a multi-spectral camera, a hyper-spectral camera, and a hybrid device.
3. The method of claim 1 , wherein said measure of closeness at pixel location (i, j, k) comprises:
C ijk ( t+ 1)=β C ijk ( t )+( I ijk ( t )− Ī ijk ( t ))( R ( t )− R ( t )),
Ī ijk ( t+ 1)=α Ī ijk ( t )+(1−α) I ijk ( t ),
R ( t+ 1)=α R ( t )+(1−α) R ( t ),
where 0<β<1, and 0<α<1.
4. The method of claim 3 , wherein said modification at pixel location (i, j, k) comprises:
J ijk ( t )= I ijk ( t )+δ C ijk ( t ) R ( t ).
5. The method of claim 1 , further comprising shifting said reference signal by a pre-specified shift τ l , where l=1, 2, . . . , m, and m is a user-specified number of shifts.
6. The method of claim 5 , wherein said measure of closeness at pixel location (i, j, k) comprises:
C ijk τ l ( t+ 1)=β C ijk τ l ( t )+( I ijk ( t )− Ī ijk ( t ))( R ( t−τ l )− R ( t−τ l )),
Ī ijk ( t+ 1)=α Ī ijk ( t )+(1−α) I ijk ( t ),
R ( t+ 1)=α R ( t )+(1−α) R ( t ),
where 0<β<1, and 0<α<1.
7. The method of claim 6 , wherein said modification at pixel location (i, j, k) comprises:
J
ijk
(
t
)
=
I
ijk
(
t
)
+
δ
∑
l
=
1
m
C
ijk
τ
l
(
t
)
R
(
t
-
τ
l
)
.
8. The method of claim 1 , wherein said reference signal is extracted from said source video.
9. The method of claim 1 , wherein said time-series signal is a signal of a subject's cardiac pulse, and wherein said reference signal is a signal which has a frequency range that approximates a frequency range of said subject's cardiac pulse.
10. The method of claim 1 , further comprising using said reconstructed source video to facilitate a determination of any of: a cardiac signal, respiratory rate and function, CO 2 concentration in exhaled breath, perspiration, brainwave pattern and signals, degree of infection, blood flow and circulation, and a bodily motion.
11. The method of claim 1 , wherein said pixel location (i, j) of each image frame of each channel k of said source video is along a vascular pattern.
12. A system for real-time enhancement of a signal of interest in a video, the system comprising:
a multi-channel video acquisition system for acquiring source video with C channels, where channel k=1 . . . C; and
a processor in communication with a memory and said multi-channel video acquisition system, said processor executing machine readable instructions for performing:
receiving a source video acquired using said multi-channel video acquisition system, said source video comprising a plurality of image frames, each frame comprising an array of pixels, each pixel having an intensity value corresponding to detected reflected energy projected by an illumination source over a wavelength range of interest;
for each pixel location (i, j) of each image frame of each channel k of said source video:
determining a measure of closeness with respect to a reference signal R(t) in both a spatial and time direction;
identifying, using said measure of closeness, a time-series signal I(t) of interest in said source video, where signal I ijk (t) appears at pixel location (i, j, k); and
in response to having identified said time-series signal of interest at pixel location (i, j, k):
modifying an intensity of said pixel at said location to generate modified video data J ijk (t); and
retrofitting said modified video data into said source video to generate a reconstructed source video; and
communicating said reconstructed source video to a display device.
13. The system of claim 12 , wherein said multi-channel video acquisition system comprises any combination of: a monochrome video camera, a color video camera, a multi-spectral camera, a hyper-spectral camera, and a hybrid device.
14. The system of claim 12 , wherein said measure of closeness at pixel location (i, j, k) comprises:
C ijk ( t+ 1)=β C ijk ( t )+( I ijk ( t )− Ī ijk ( t ))( R ( t )− R ( t )),
Ī ijk ( t+ 1)=α Ī ijk ( t )+(1−α) I ijk ( t ),
R ( t+ 1)=α R ( t )+(1−α) R ( t ),
where 0<β<1, and 0<α<1.
15. The system of claim 14 , wherein said modification at pixel location (i, j, k) comprises:
J ijk ( t )= I ijk ( t )+δ C ijk ( t ) R ( t ).
16. The system of claim 12 , further comprising shifting said reference signal by a pre-specified shift τ l , where l=1, 2, . . . , m, and m is a user-specified number of shifts.
17. The system of claim 16 , wherein said measure of closeness at pixel location (i, j, k) comprises:
C ijk τ l ( t+ 1)=β C ijk τ l ( t )+( I ijk ( t )− Ī ijk ( t ))( R ( t−τ l )− R ( t−τ l )),
Ī ijk ( t+ 1)=α Ī ijk ( t )+(1−α) I ijk ( t ),
R ( t+ 1)=α R ( t )+(1−α) R ( t ),
where 0<β<1, and 0<α<1.
18. The system of claim 17 , wherein said modification at pixel location (i, j, k) comprises:
J
ijk
(
t
)
=
I
ijk
(
t
)
+
δ
∑
l
=
1
m
C
ijk
τ
l
(
t
)
R
(
t
-
τ
l
)
.
19. The system of claim 12 , wherein said reference signal is extracted from said source video.
20. The system of claim 12 , wherein said time-series signal is a signal of a subject's cardiac pulse, and wherein said reference signal is a signal which has a frequency range that approximates a frequency range of said subject's cardiac pulse.
21. The system of claim 12 , further comprising using said reconstructed source video to facilitate a determination of any of: a cardiac signal, respiratory rate and function, CO 2 concentration in exhaled breath, perspiration, brainwave pattern and signals, degree of infection, blood flow and circulation, and a bodily motion.
22. The system of claim 12 , wherein said pixel location (i, j) of each image frame of each channel k of said source video is along a vascular pattern.