Temporal augmentation for training video reasoning system
A method for augmenting video sequences in a video reasoning system is presented. The method includes randomly subsampling a sequence of video frames captured from one or more video cameras, randomly reversing the subsampled sequence of video frames to define a plurality of sub-sequences of randomly reversed video frames, training, in a training mode, a video reasoning model with temporally augmented input, including the plurality of sub-sequences of randomly reversed video frames, to make predictions over temporally augmented target classes, updating parameters of the video reasoning model by a machine leaning algorithm, and deploying, in an inference mode, the video reasoning model in the video reasoning system to make a final prediction related to a human action in the sequence of video frames.
1. A method for augmenting video sequences in a video reasoning system, the method comprising:
randomly subsampling a sequence of video frames captured from one or more video cameras;
randomly reversing the subsampled sequence of video frames to define a plurality of sub-sequences of randomly reversed video frames;
training, in a training mode, a video reasoning model with temporally augmented input, including the plurality of sub-sequences of randomly reversed video frames, to make predictions over temporally augmented target classes;
updating parameters of the video reasoning model by a machine learning algorithm by discarding a final prediction to retain the temporally augmented target classes with corresponding original classes; and
deploying, in an inference mode, the video reasoning model, with overfitting suppression through a learned temporal order of video frames to determine the original classes, in the video reasoning system to make the final prediction related to classify a human action in the sequence of video frames.
2. The method of claim 1 , wherein a target class is offset by a total number of original classes when the subsampled sequence of video frames is randomly reversed.
3. The method of claim 1 , wherein the reversing of the subsampled sequence of video frames is implemented to classify a doubled number of class categories.
4. The method of claim 1 , wherein the video reasoning model learns from temporal ordering for classification.
5. The method of claim 1 , wherein taking a softmax of logits of original classes only is employed to arrive at the final prediction.
6. The method of claim 1 , wherein combining logits of original classes and corresponding augmented classes for a softmax value is employed to arrive at the final prediction.
7. The method of claim 1 , wherein the final prediction is discarded if a top class does not belong to any original classes.
8. A non-transitory computer-readable storage medium comprising a computer-readable program for explaining sensor time series data in natural language, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:
randomly subsampling a sequence of video frames captured from one or more video cameras;
randomly reversing the subsampled sequence of video frames to define a plurality of sub-sequences of randomly reversed video frames;
training, in a training mode, a video reasoning model with temporally augmented input, including the plurality of sub-sequences of randomly reversed video frames, to make predictions over temporally augmented target classes;
updating parameters of the video reasoning model by a machine learning algorithm by discarding a final prediction to retain the temporally augmented target classes with corresponding original classes; and
deploying, in an inference mode, the video reasoning model, with overfitting suppression through a learned temporal order of video frames to determine the original classes, in a video reasoning system to make the final prediction related to classify a human action in the sequence of video frames.
9. The non-transitory computer-readable storage medium of claim 8 , wherein a target class is offset by a total number of original classes when the subsampled sequence of video frames is randomly reversed.
10. The non-transitory computer-readable storage medium of claim 8 , wherein the reversing of the subsampled sequence of video frames is implemented to classify a doubled number of class categories.
11. The non-transitory computer-readable storage medium of claim 8 , wherein the video reasoning model learns from temporal ordering for classification.
12. The non-transitory computer-readable storage medium of claim 8 , wherein taking a softmax of logits of original classes only is employed to arrive at the final prediction.
13. The non-transitory computer-readable storage medium of claim 8 , wherein combining logits of original classes and corresponding augmented classes for a softmax value is employed to arrive at the final prediction.
14. The non-transitory computer-readable storage medium of claim 8 , wherein the final prediction is discarded if a top class does not belong to any original classes.
15. A system for explaining sensor time series data in natural language, the system comprising:
a memory; and
one or more processors in communication with the memory configured to:
randomly subsample a sequence of video frames captured from one or more video cameras;
randomly reverse the subsampled sequence of video frames to define a plurality of sub-sequences of randomly reversed video frames;
train, in a training mode, a video reasoning model with temporally augmented input, including the plurality of sub-sequences of randomly reversed video frames, to make predictions over temporally augmented target classes;
update parameters of the video reasoning model by a machine learning algorithm by discarding a final prediction to retain the temporally augmented target classes with corresponding original classes; and
deploy, in an inference mode, the video reasoning model, with overfitting suppression through a learned temporal order of video frames to determine the original classes, in the video reasoning system to make the final prediction related to classify a human action in the sequence of video frames.
16. The system of claim 15 , wherein a target class is offset by a total number of original classes when the subsampled sequence of video frames is randomly reversed.
17. The system of claim 15 , wherein the reversing of the subsampled sequence of video frames is implemented to classify a doubled number of class categories.
18. The system of claim 15 , wherein the video reasoning model learns from temporal ordering for classification.
19. The system of claim 15 , wherein taking a softmax of logits of original classes only is employed to arrive at the final prediction.
20. The system of claim 15 , wherein combining logits of original classes and corresponding augmented classes for a softmax value is employed to arrive at the final prediction.