Deep reinforcement learning framework for characterizing video content
Methods and systems for performing sequence level prediction of a video scene are described. Video information in a video scene is represented as a sequence of features depicted each frame. An environment state for each time step t corresponding to each frame is represented by the video information for time step t and predicted affective information from a previous time step t−1. An action A(t) as taken with an agent controlled by a machine learning algorithm for the frame at step t, wherein an output of the action A(t) represents affective label prediction for the frame at the time step t. A pool of predicted actions is transformed to a predicted affective history at a next time step t+1. The predictive affective history is included as part of the environment state for the next time step t+1. A reward R is generated on predicted actions up to the current time step t, by comparing them against corresponding annotated movie scene affective labels.
1. A method for performing sequence level prediction of a video scene, comprising:
representing video information in the video scene as a sequence of features depicted each frame;
representing an environment state for each time step t corresponding to each frame by the video information for time step t and predicted affective information from a previous time step t−1
taking an action A(t) with an agent controlled by a machine learning algorithm for the frame at step t, wherein an output of the action A(t) represents affective label prediction for the frame at the time step t;
transforming a pool of predicted actions to a predicted affective history at a next time step t+1;
including the predictive affective history as part of the environment state for the next time step t+1; and
generating a reward R on predicted actions up to the current time step t, by comparing them against corresponding annotated movie scene affective labels.
2. The method of claim 1 , wherein the affective label prediction at the time step t represents whether the video scene is funny or not funny.
3. The method of claim 1 , wherein a value of the reward R is based on a majority vote of frame-level affective label prediction results up to the current frame at the time step t.
4. The method of claim 1 , wherein a value of the reward R is based on the minimum percentage vote of the frame-level results.
5. The method of claim 1 , wherein a value of the reward R is zero until a frame when human labels are available.
6. The method of claim 1 , wherein the sequence of features is a sequence of faces appearing in the video scene.
7. The method of claim 6 , wherein the sequence of faces is a sequence of faces of an individual character in the video scene.
8. The method of claim 6 , wherein the sequence of faces is a sequence of faces of an individual character in the video scene and wherein when multiple faces are shown in the video scene the sequence of faces is a sequence of faces of located in the center of the scene.
9. A system for performing sequence level prediction of a video scene, comprising:
a processor;
a memory coupled to the processor; and
processor-executable instructions embodied in the memory, the instructions being configured to cause the processor to perform the following when executed:
represent video information in a video scene as a sequence of features depicted each frame;
represent an environment state for each time step t corresponding to each frame by the video information for time step t and predicted affective information from a previous time step t−1
take an action A(t) with an agent controlled by a machine learning algorithm for the frame at step t, wherein an output of the action A(t) represents affective label prediction for the frame at the time step t;
transform a pool of predicted actions to a predicted affective history at a next time step t+1;
include the predictive affective history as part of the environment state for the next time step t+1; and
generate a reward R on predicted actions up to the current time step t, by comparing them against corresponding annotated movie scene affective labels.
10. The system of claim 9 , wherein the affective label prediction at the time step t represents whether the video scene is funny or not funny.
11. The system of claim 9 , wherein a value of the reward R is based on a majority vote of frame-level affective label prediction results up to the current frame at the time step t.
12. The system of claim 9 , wherein a value of the reward R is based on the minimum percentage vote of the frame-level results.
13. The system of claim 9 , wherein a value of the reward R is zero until a frame when human labels are available.
14. The system of claim 9 , wherein the sequence of features is a sequence of faces appearing in the video scene.
15. The system of claim 14 , wherein the sequence of faces is a sequence of faces of an individual character in the video scene.
16. The system of claim 14 , wherein the sequence of faces is a sequence of faces of an individual character in the video scene and wherein when multiple faces are shown in the video scene the sequence of faces is a sequence of faces of located in the center of the scene.
17. A non-transitory computer-readable medium having executable instructions embodied therein, the instructions being configured cause a computer to perform the following when executed:
represent video information in a video scene as a sequence of features depicted each frame;
represent an environment state for each time step t corresponding to each frame by the video information for time step t and predicted affective information from a previous time step t−1
take an action A(t) with an agent controlled by a machine learning algorithm for the frame at step t, wherein an output of the action A(t) represents affective label prediction for the frame at the time step t;
transform a pool of predicted actions to a predicted affective history at a next time step t+1;
include the predictive affective history as part of the environment state for the next time step t+1; and
generate a reward R on predicted actions up to the current time step t, by comparing them against corresponding annotated movie scene affective labels.
18. The non-transitory computer-readable medium of claim 17 , wherein the affective label prediction at the time step t represents whether the video scene is funny or not funny.
19. The non-transitory computer-readable medium of claim 17 , wherein a value of the reward R is based on a majority vote of frame-level affective label prediction results up to the current frame at the time step t.
20. The non-transitory computer-readable medium of claim 17 , wherein a value of the reward R is based on the minimum percentage vote of the frame-level results.
21. The non-transitory computer-readable medium of claim 17 , wherein a value of the reward R is zero until a frame when human labels are available.
22. The non-transitory computer-readable medium of claim 17 , wherein the sequence of features is a sequence of faces appearing in the video scene.
23. The non-transitory computer-readable medium of claim 22 , wherein the sequence of faces is a sequence of faces of an individual character in the video scene.
24. The non-transitory computer-readable medium of claim 22 , wherein the sequence of faces is a sequence of faces of an individual character in the video scene and wherein when multiple faces are shown in the video scene the sequence of faces is a sequence of faces of located in the center of the scene.