Systems and methods for video models with procedure understanding
Embodiments described herein provide systems and methods for training video models to perform a task from an input instructional video. A procedure knowledge graph (PKG) may be generated with nodes representing procedure steps, and edges representing relationships between the steps. The PKG may be generated based on text and/or video training data which includes procedures (e.g., instructional videos). Using the PKG, a video model may be trained using the PKG to provide supervisory training signals for a number of tasks. Once the model is trained, it may be fine-tuned for a specific task which benefits from the model being trained in a way that makes the model embed procedural information when encoding videos.
1. A method of training a neural network based video model to perform a task from an input instructional video, the method comprising:
receiving, via a data interface, at least one instructional video relating to completing a task having a series of steps and at least one text-based procedure description describing the series of steps;
generating, based on the instructional video and the text-based procedure description, a procedure knowledge graph having a plurality of nodes representing the series of steps and a plurality of edges representing sequential relationships between the series of steps;
generating, by the neural network based video model, a first training output according to a first training objective from an input of a video segment from the at least one instructional video;
generating a plurality of pseudo-labels corresponding to the first training objective based on the procedure knowledge graph;
computing a first loss corresponding to the first training objective based on the first training output and the plurality of pseudo-labels; and
updating parameters of the neural network based video model based on the computed first loss via backpropagation.
2. The method of claim 1 , wherein generating, by the neural network based video model, the first training output comprises:
generating, by a pre-trained video foundation encoder and a light-weight encoder, a latent representation of the input of the video segment;
generating, by a decoder corresponding to the first training objective, the first training output; and
updating parameters in the light-weight encoder based on the first loss while keeping the pre-trained video foundation encoder frozen.
3. The method of claim 1 , wherein the generating the plurality of pseudo-labels includes performing one or more queries on the procedure knowledge graph according to the first training objective.
4. The method of claim 3 , wherein the first training output includes a plurality of outputs, each of the plurality of outputs corresponding to a respective pseudo-label of the plurality of pseudo-labels.
5. The method of claim 4 , wherein the plurality of outputs are associated with different queries performed on the procedure knowledge graph.
6. The method of claim 1 , wherein generating, by the neural network based video model, the first training output includes generating a latent representation of the input of the video segment, further comprising:
training a task-specific neural network model which takes the latent representation as an input, and outputs a prediction associated with the input of the video segment.
7. The method of claim 1 , further comprising:
generating a second training output according to a second training objective from the input of the video segment;
generating a second plurality of pseudo-labels corresponding to the second training objective based on the procedure knowledge graph;
computing a second loss corresponding to the second training objective based on the second training output and the second plurality of pseudo-labels; and
updating parameters of the neural network based video model based on the computed second loss via backpropagation.
8. The method of claim 7 , wherein the first training objective or the second training objective is one of video-node matching, video-task matching, task context learning or node relation learning.
9. A system for training a neural network based video model to perform a task from an input instructional video, the system comprising:
a memory that stores the video model and a plurality of processor executable instructions;
a communication interface that receives at least one instructional video relating to completing a task having a series of steps and at least one text-based procedure description describing the series of steps; and
one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising:
generating, based on the instructional video and the text-based procedure description, a procedure knowledge graph having a plurality of nodes representing the series of steps and a plurality of edges representing sequential relationships between the series of steps;
generating, by the neural network based video model, a first training output according to a first training objective from an input of a video segment from the at least one instructional video;
generating a plurality of pseudo-labels corresponding to the first training objective based on the procedure knowledge graph;
computing a first loss corresponding to the first training objective based on the first training output and the plurality of pseudo-labels; and
updating parameters of the neural network based video model based on the computed first loss via backpropagation.
10. The system of claim 9 , wherein generating, by the neural network based video model, the first training output comprises:
generating, by a pre-trained video foundation encoder and a light-weight encoder, a latent representation of the input of the video segment;
generating, by a decoder corresponding to the first training objective, the first training output; and
updating parameters in the light-weight encoder based on the first loss while keeping the pre-trained video foundation encoder frozen.
11. The system of claim 9 , wherein the generating the plurality of pseudo-labels includes performing one or more queries on the procedure knowledge graph according to the first training objective.
12. The system of claim 11 , wherein the first training output includes a plurality of outputs, each of the plurality of outputs corresponding to a respective pseudo-label of the plurality of pseudo-labels.
13. The system of claim 12 , wherein the plurality of outputs are associated with different queries performed on the procedure knowledge graph.
14. The system of claim 9 , wherein generating, by the neural network based video model, the first training output includes generating a latent representation of the input of the video segment, further comprising:
training a task-specific neural network model which takes the latent representation as an input, and outputs a prediction associated with the input of the video segment.
15. The system of claim 9 , the operations further comprising:
generating a second training output according to a second training objective from the input of the video segment;
generating a second plurality of pseudo-labels corresponding to the second training objective based on the procedure knowledge graph;
computing a second loss corresponding to the second training objective based on the second training output and the second plurality of pseudo-labels; and
updating parameters of the neural network based video model based on the computed second loss via backpropagation.
16. The system of claim 15 , wherein the first training objective or the second training objective is one of video-node matching, video-task matching, task context learning or node relation learning.
17. A non-transitory machine-readable medium comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:
receiving, via a data interface, at least one instructional video relating to completing a task having a series of steps and at least one text-based procedure description describing the series of steps;
generating, based on the instructional video and the text-based procedure description, a procedure knowledge graph having a plurality of nodes representing the series of steps and a plurality of edges representing sequential relationships between the series of steps;
generating, by a neural network based video model, a first training output according to a first training objective from an input of a video segment from the at least one instructional video;
generating a plurality of pseudo-labels corresponding to the first training objective based on the procedure knowledge graph;
computing a first loss corresponding to the first training objective based on the first training output and the plurality of pseudo-labels; and
updating parameters of the neural network based video model based on the computed first loss via backpropagation.
18. The non-transitory machine-readable medium of claim 17 , wherein the generating the plurality of pseudo-labels includes performing one or more queries on the procedure knowledge graph according to the first training objective.
19. The non-transitory machine-readable medium of claim 18 , wherein the first training output includes a plurality of outputs, each of the plurality of outputs corresponding to a respective pseudo-label of the plurality of pseudo-labels.
20. The non-transitory machine-readable medium of claim 17 , further comprising:
generating a second training output according to a second training objective from the input of the video segment;
generating a second plurality of pseudo-labels corresponding to the second training objective based on the procedure knowledge graph;
computing a second loss corresponding to the second training objective based on the second training output and the second plurality of pseudo-labels; and
updating parameters of the neural network based video model based on the computed second loss via backpropagation.