Anticipating future video based on present video

1. A method comprising:

accessing a first set of images of a plurality of images of a scene, wherein the first set of images show the scene during a time period;

generating, by processing the first set of images using a first machine-learning model, one or more attributes representing one or more of observed actions performed in the scene during the time period or visual attributes appearing in the scene during the time period, wherein the one or more attributes further includes probabilities indicating one or more of likelihoods that the observed actions are accurately represented or likelihoods that the visual attributes are accurately represented, wherein the first machine-learning model receives the first set of images as input and wherein the one or more attributes represent at least one of observed characteristics of the scene during the time period, a description of the scene during the time period, a description of an action performed in the scene, or a description of entities in the scene, and

predicting, with a second machine-learning model, one or more actions that would happen in the scene after the time period based on the one or more of the observed actions or the visual attributes, and the probabilities generated by the first machine-learning model,

wherein the first machine-learning model and the second machine-learning model are different types of learning models.

2. The method of claim 1 , further comprising:

predicting, by processing the first set of images using a third machine-learning model, one or more actions that would happen in the scene after the time period; and

determining one or more composite predicted actions based on the actions predicted by processing the generated one or more attributes using the second machine-learning model and the actions predicted by processing the first set of images using the third machine-learning model.

3. The method of claim 2 , wherein determining the one or more composite predicted actions based on the actions predicted by processing the generated one or more attributes using the second machine-learning model and the actions predicted by processing the first set of images using the third machine-learning model further comprises:

assigning a score for each action predicted by processing the generated one or more attributes using the second machine-learning model;

assigning a score for each action predicted by processing the first set of images using the third machine-learning model; and

determining a weighted average of the assigned scores for each action for the scene.

4. The method of claim 2 , wherein the third machine-learning model was trained using loss minimization on a set of training images and corresponding potential predicted actions.

5. The method of claim 1 , wherein the predicting, with the second machine-learning model, one or more actions that would happen in the scene after the time period comprises:

accessing a set of potential actions for the scene, wherein each action of the set of potential actions is associated with a set of attributes considered predictive of the action occurring; and

selecting one or more actions from the set of potential actions for the scene after the time period based on the determined probabilities and the actions considered predictive of the action occurring.

6. The method of claim 5 , wherein the set of potential actions for the scene have been pre-generated by a machine-learning model.

7. The method of claim 1 , further comprising:

accessing a second set of images of the plurality of the images of the scene, wherein the second set of images show the scene before the time period;

generating, by processing the second set of images using the first machine-learning model, one or more attributes representing observed actions performed in the scene before the time period; and

wherein the predicting one or more actions that would happen in the scene after the time period is performed by processing the generated one or more attributes representing observed actions performed in the scene before the time period and the generated one or more attributes representing observed actions performed in the scene during the time period.

8. The method of claim 1 , wherein the one or more actions that would happen in the scene after the time period is further predicted by the second machine-learning model based on content of the first set of images.

9. The method of claim 1 , wherein processing the generated one or more attributes using the second machine-learning model comprises correlating the generated one or more attributes with potential predicted actions for the scene after the time period.

10. The method of claim 1 , wherein the plurality of images are frames of a video recording of the scene during the time period.

11. The method of claim 1 , further comprising:

predicting a label for the scene after the time period based on the generated one or more attributes representing one or more actions performed in the scene during the time period.

12. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

access a first set of images of a plurality of images of a scene, wherein the first set of images show the scene during a time period;

generate, by processing the first set of images using a first machine-learning model, one or more attributes representing one or more of observed actions performed in the scene during the time period or visual attributes appearing in the scene during the time period, wherein the one or more attributes further includes probabilities indicating one or more of likelihoods that the observed actions are accurately represented or likelihoods that the visual attributes are accurately represented, wherein the first machine-learning model receives the first set of images as input and wherein the one or more attributes represent at least one of observed characteristics of the scene during the time period, a description of the scene during the time period, a description of an action performed in the scene, or a description of entities in the scene; and

predict, with a second machine-learning model, one or more actions that would happen in the scene after the time period based on the one or more of the observed actions or the visual attributes, and the probabilities generated by the first machine-learning model,

wherein the first machine-learning model and the second machine-learning model are different types of learning models.

13. The computer-readable non-transitory storage media of claim 12 , wherein the software is further operable when executed to:

predict, by processing the first set of images using a third machine-learning model, one or more actions that would happen in the scene after the time period; and

determine one or more composite predicted actions based on the actions predicted by processing the generated one or more attributes using the second machine-learning model and the actions predicted by processing the first set of images using the third machine-learning model.

14. The computer-readable non-transitory storage media of claim 13 , wherein the software operable to determine the one or more composite predicted actions based on the actions predicted by processing the generated one or more attributes using the second machine-learning model and the actions predicted by processing the first set of images using the third machine-learning model is further operable to when executed to:

assign a score for each action predicted by processing the generated one or more attributes using the second machine-learning model;

assign a score for each action predicted by processing the first set of images using the third machine-learning model; and

determine a weighted average of the assigned scores for each action for the scene.

15. The computer-readable non-transitory storage media of claim 13 , wherein the third machine-learning model was trained using loss minimization on a set of training images and corresponding potential predicted actions.

16. A system comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:

access a first set of images of a plurality of images of a scene, wherein the first set of images show the scene during a time period;

generate, by processing the first set of images using a first machine-learning model, one or more attributes representing one or more of observed actions performed in the scene during the time period or visual attributes appearing in the scene during the time period, wherein the one or more attributes further includes probabilities indicating one or more of likelihoods that the observed actions are accurately represented or likelihoods that the visual attributes are accurately represented, wherein the first machine-learning model receives the first set of images as input and wherein the one or more attributes represent at least one of observed characteristics of the scene during the time period, a description of the scene during the time period, a description of an action performed in the scene, or a description of entities in the scene; and

predict, with a second machine-learning model, one or more actions that would happen in the scene after the time period based on the one or more of the observed actions or the visual attributes, and the probabilities generated by the first machine-learning model,

wherein the first machine-learning model and the second machine-learning model are different types of learning models.

17. The system of claim 16 , wherein the processors are further operable when executing the instructions to:

predict, by processing the first set of images using a third machine-learning model, one or more actions that would happen in the scene after the time period; and

determine one or more composite predicted actions based on the actions predicted by processing the generated one or more attributes using the second machine-learning model and the actions predicted by processing the first set of images using the third machine-learning model.

18. The system of claim 17 , wherein the instructions operable when executed by one or more of the processors to cause the system to determine the one or more composite predicted actions based on the actions predicted by processing the generated one or more attributes using the second machine-learning model and the actions predicted by processing the first set of images using the third machine-learning model are further operable to cause the system to:

assign a score for each action predicted by processing the generated one or more attributes using the second machine-learning model;

assign a score for each action predicted by processing the first set of images using the third machine-learning model; and

determine a weighted average of the assigned scores for each action for the scene.

19. The system of claim 17 , wherein the third machine-learning model was trained using loss minimization on a set of training images and corresponding potential predicted actions.