Efficient video processing via temporal progressive learning
Systems and methods for performing temporal progressive learning for video processing are provided herein. Some examples include receiving a video that includes a plurality of frames, extracting a first subset of frames from the plurality of frames, and inputting the first subset of frames into a model that includes an encoder and a decoder. The examples further include comparing a first output of the model to the first subset of frames and updating the encoder, thereby training the encoder, and extracting a second subset of frames from the plurality of frames. The second subset of frames includes a number of frames that is larger than a number of frames in the first subset of frames. The examples further include inputting the second subset of frames into the model, comparing a second output of the model to the second subset of frames and updating the encoder, thereby further training the encoder.
1. A method of performing temporal progressive learning for video processing, the method comprising:
receiving a video stream comprising a plurality of frames;
extracting a first subset of frames from the plurality of frames;
inputting the first subset of frames into a model, wherein the model includes an encoder and a decoder;
comparing a first output of the model to the first subset of frames and updating the encoder based on the comparison, thereby training the encoder;
extracting a second subset of frames from the plurality of frames, the second subset of frames comprising a number of frames that is larger than a number of frames in the first subset of frames;
inputting the second subset of frames into the model;
comparing a second output of the model to the second subset of frames and updating the encoder based on the comparison, thereby further training the encoder; and
providing the encoder.
2. The method of claim 1 , wherein the model is a masked auto encoder (MAE) model.
3. The method of claim 2 , wherein each frame in the first and second subsets of frames are randomly masked, prior to being input into the MAE model.
4. The method of claim 1 , wherein the first and second subsets of frames are randomly selected from the plurality of frames.
5. The method of claim 1 , wherein the second subset of frames comprises twice as many frames as the first subset of frames.
6. The method of claim 1 , further comprising, prior to providing the model:
extracting a third subset of frames from the plurality of frames, the third subset of frames being randomly selected from the plurality of frames and comprising a number of frames that is larger than the number of frames in the second subset of frames;
inputting the third subset of frames into the model; and
comparing a third output of the model to the third subset of frames and updating the encoder based on the comparison, thereby further training the encoder.
7. The method of claim 6 , wherein the third subset of frames comprises twice as many frames as the second subset of frames.
8. The method of claim 1 , wherein each sequence of the extracting, the inputting, and the comparing define a respective stage, and wherein the number of frames extracted in the subset of frames for each stage are determined based on a total number of stages and a computational budget.
9. A system for performing temporal progressive learning for video processing, the system comprising:
a processor;
memory storing instructions that, when executed by the processor, cause the system to perform a set of operations, the set of operations comprising:
receiving a video stream comprising a plurality of frames;
extracting a first subset of frames from the plurality of frames;
inputting the first subset of frames into a model, wherein the model includes an encoder and a decoder;
comparing a first output of the model to the first subset of frames and updating the encoder based on the comparison, thereby training the encoder;
extracting a second subset of frames from the plurality of frames, the second subset of frames comprising a number of frames that is larger than a number of frames in the first subset of frames;
inputting the second subset of frames into the model;
comparing a second output of the model to the second subset of frames and updating the encoder based on the comparison, thereby further training the encoder; and
providing the encoder.
10. The system of claim 9 , wherein the model is a masked auto encoder (MAE) model.
11. The system of claim 10 , wherein each frame in the first and second subsets of frames are randomly masked, prior to being input into the MAE model.
12. The system of claim 9 , wherein the first and second subsets of frames are randomly selected from the plurality of frames.
13. The system of claim 9 , wherein the second subset of frames comprises twice as many frames as the first subset of frames.
14. The system of claim 9 , further comprising, prior to providing the model:
extracting a third subset of frames from the plurality of frames, the third subset of frames being randomly selected from the plurality of frames and comprising a number of frames that is larger than the number of frames in the second subset of frames;
inputting the third subset of frames into the model; and
comparing a third output of the model to the third subset of frames and updating the encoder based on the comparison, thereby further training the encoder.
15. The system of claim 14 , wherein the third subset of frames comprises twice as many frames as the second subset of frames.
16. The system of claim 9 , wherein each sequence of the extracting, the inputting, and the comparing define a respective stage, and wherein the number of frames extracted in the subset of frames for each stage are determined based on a total number of stages and a computational budget.
17. One or more computer readable non-transitory storage media embodying software that is operable when executed, by at least one processor of a device, to:
receive a video stream comprising a plurality of frames;
extract a first subset of frames from the plurality of frames;
input the first subset of frames into a model, wherein the model includes an encoder and a decoder;
compare a first output of the model to the first subset of frames and updating the encoder based on the comparison, thereby training the encoder;
extract a second subset of frames from the plurality of frames, the second subset of frames comprising a number of frames that is larger than a number of frames in the first subset of frames;
input the second subset of frames into the model;
compare a second output of the model to the second subset of frames and updating the encoder based on the comparison, thereby further training the encoder; and
provide the encoder.
18. The one or more computer readable non-transitory storage media of claim 17 , wherein the model is a masked auto encoder (MAE) model.
19. The one or more computer readable non-transitory storage media of claim 18 , wherein each frame in the first and second subsets of frames are randomly masked, prior to being input into the MAE model.
20. The one or more computer readable non-transitory storage media of claim 17 , wherein the first and second subsets of frames are randomly selected from the plurality of frames.