Real time end-to-end learning system for a high frame rate video compressive sensing network
A real time end-to-end learning system for a high frame rate video compressive sensing network is described. The slow reconstruction speed of conventional compressive sensing approaches is overcome by directly modeling an inverse mapping from compressed domain to original domain in a single forward propagation. Through processing massive unlabeled video data such a mapping is learned by a neural network using data-driven methods. Systems and methods according to this disclosure incorporate a multi-rate convolutional neural network (CNN) and a synthesizing recurrent neural network (RNN) to achieve real time compression and reconstruction of video data.
1. A method for reconstructing compressive sensing (CS) video data, comprising:
receiving CS reference video data based on a reference frame;
receiving CS remaining video data based on a set of remaining frames;
reconstructing the reference frame from the CS reference video data using a reference convolutional neural network (CNN);
training a synthesizing recurrent neural network (RNN) using output data from the reference CNN; and
reconstructing the set of remaining frames from the CS remaining video data using a remaining CNN and the synthesizing RNN.
2. The method of claim 1 , further comprising generating a reconstructed group of pictures from the reconstructed reference frame and the reconstructed set of remaining frames.
3. The method of claim 2 , wherein reconstructing the reference frame comprises extracting spatial visual features for the reconstructed group of pictures from the CS reference video data.
4. The method of claim 1 , wherein the synthesizing RNN comprises a long short-term memory (LSTM) network.
5. The method of claim 4 , wherein reconstructing the set of remaining frames comprises:
processing the CS remaining video data with the remaining CNN to produce decompressed remaining frames; and
enhancing the decompressed remaining frames by the synthesizing RNN.
6. The method of claim 4 , wherein:
training the synthesizing RNN comprises extracting motion features from the reconstructed reference frame; and
reconstructing the set of remaining frames comprises:
extracting spatial features from the CS remaining video data with the remaining CNN; and
reconstructing the set of remaining frames using the extracted spatial features from the CS remaining video data and the extracted motion features from the reconstructed reference frame.
7. The method of claim 1 , wherein the reference frame of the CS reference video data comprises more measurements than each of the set of remaining frames of the CS remaining video data.
8. The method of claim 1 , further comprising simultaneously sampling video data at a sub-Nyquist sampling rate and compressing the video data to generate the CS reference video data and the CS remaining video data.
9. The method of claim 8 wherein simultaneously sampling the video data at the sub-Nyquist sampling rate and compressing the video data comprises:
randomly encoding the reference frame at a first compression ratio (CR);
randomly encoding the set of remaining frames at a second CR which is between 5 and 100 times the first CR.
10. A video reconstruction network, comprising:
a reference convolutional neural network (CNN) adapted to reconstruct a reference frame from compressive sensing (CS) reference video data;
a remaining CNN adapted to partially reconstruct a set of remaining frames from CS remaining video data; and
a synthesizing recurrent neural network (RNN) adapted to enhance the set of remaining frames partially reconstructed by the remaining CNN.
11. The video reconstruction network of claim 10 , wherein the reference CNN comprises an eight-layer CNN.
12. The video reconstruction network of claim 11 , wherein the remaining CNN comprises a plurality of three-layer CNNs.
13. The video reconstruction network of claim 12 , wherein the reference CNN is pre-trained using CS training data from a video database.
14. The video reconstruction network of claim 13 , wherein the video reconstruction network is trained using ground truth video frames and at least one of stochastic gradient descent (SGD), Adagrad, or adaptive moment estimation (Adam).
15. The video reconstruction network of claim 10 , wherein the synthesizing RNN comprises a long short-term memory (LSTM) network adapted to extract motion features from an output of the reference CNN.
16. A video processing system, comprising:
a sensing (CS) decoder, comprising:
a reference convolutional neural network (CNN) adapted to reconstruct a reference frame from CS reference video data;
a recurrent neural network (RNN) trained by output data of the reference CNN; and
a remaining CNN adapted to reconstruct a set of remaining frames from CS remaining video data using the RNN.
17. The video processing system of claim 16 , wherein the RNN is adapted to:
enhance the reconstructed set of remaining frames; and
generate a reconstructed group of pictures from the reconstructed reference frame and the enhanced and reconstructed set of remaining frames.
18. The video processing system of claim 16 , further comprising a CS encoder configured to randomly sample the set of remaining frames at a sub-Nyquist sampling rate.
19. The video processing system of claim 18 , wherein the CS encoder randomly samples and compresses the reference frame and the set of remaining frames with at least one of a random Bernoulli matrix or a random Gaussian matrix.
20. The video processing system of claim 18 , further comprising:
a low compression ratio (CR) random encoder adapted to produce the CS reference video data; and
a high-CR random encoder adapted to produce the CS remaining video data.