Generating corrected simulations using video generation models
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating corrected simulations using video generation models. One of the methods includes obtaining an input video including a sequence of frames depicting a state transition of an environment that does not meet a state transition criterion; generating, based on a frame from a sequence of frames that depicts an incorrect end state of the state transition, a synthetic ending frame depicting a corrected end state of the state transition; and processing an input including one or more key frames from sequence of frames of the input video and the synthetic ending frame depicting the corrected end state of the state transition using a video generation model to generate an output video depicting a synthetic state transition of the environment that meets the state transition criterion.
1 . A method, comprising:
obtaining an input video comprising a sequence of frames depicting a state transition of an environment that does not meet a state transition criterion;
generating, based on a frame from a sequence of frames that depicts an incorrect end state of the state transition, a synthetic ending frame depicting a corrected end state of the state transition; and
processing an input comprising one or more key frames from sequence of frames of the input video and the synthetic ending frame depicting the corrected end state of the state transition using a video generation model to generate an output video depicting a synthetic state transition of the environment that meets the state transition criterion.
2 . The method of claim 1 , comprising:
processing the sequence of the frames of the input video using a visual language model to generate a respective state annotation for each frame in the sequence of the frames of the input video; and
determining the one or more key frames from the sequence of the frames of the input video based on the respective state annotations for the sequence of the frames of the input video.
3 . The method of claim 1 , wherein the sequence of the frames of the input video comprises a transition frame, wherein the state transition of the environment depicted in the transition frame and frames in the input video before the transition frame meets the state transition criterion, and the state transition of the environment depicted in frames in the input video after the transition frame does not meet the state transition criterion.
4 . The method of claim 3 , wherein the input to the video generation model comprises the transition frame.
5 . The method of claim 3 , wherein the input to the video generation model comprises one or more frames that precede the transition frame.
6 . The method of claim 1 , comprising:
processing an input comprising at least the frame that depicts the incorrect end state of the state transition using an image editing model to generate the synthetic ending frame depicting the corrected end state of the state transition.
7 . The method of claim 1 , wherein the one or more key frames comprise a starting frame depicting the environment before the state transition happens, and the method comprises:
obtaining a set of points on the starting frame;
obtaining a target trajectory for the set of points associated with a target condition of the environment; and
processing the input comprising the starting frame, the synthetic ending frame, and the target trajectory for the set of points using the video generation model to generate the output video that meets the state transition criterion and is conditioned on the starting frame, the synthetic ending frame, and the target trajectory for the set of points, wherein a first frame of the output video is the starting frame, a last frame of the output video is the synthetic ending frame, and locations for the set of points in at least some frames of the output video approximately follow the target trajectory.
8 . The method of claim 1 , further comprising:
generating control data for controlling one or more objects in the environment that causes the one or more objects to follow respective trajectories for each of the one or more objects depicted in the output video.
9 . The method of claim 1 , wherein the state transition of the environment comprises a landing or a takeoff of an aircraft.
10 . The method of claim 1 , further comprising:
obtaining a set of points on an object in the environment on a starting frame of the output video;
processing the output video using a point tracking model to generate trajectories for the set of points in the output video; and
generating an evaluation result for the output video based on the trajectories for the set of points in the output video.
11 . The method of claim 10 , wherein generating the evaluation result for the output video comprises:
determining that at least one trajectory of the trajectories for the set of points in the output video is discontinuous; and
in response to determining that at least one trajectory of the trajectories for the set of points in the output video is discontinuous, generating the evaluation result for the output video indicating that the output video has an error.
12 . The method of claim 10 , wherein generating the evaluation result for the output video comprises:
determining a difference value between the trajectories for the set of points in the output video and reference trajectories for the set of points generated by a simulation engine that is based on one or more laws of physics; and
determining whether the trajectories for the set of points in the output video meet the one or more laws of physics based on whether the difference value is less than a threshold.
13 . A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
obtaining an input video comprising a sequence of frames depicting a state transition of an environment that does not meet a state transition criterion;
generating, based on a frame from a sequence of frames that depicts an incorrect end state of the state transition, a synthetic ending frame depicting a corrected end state of the state transition; and
processing an input comprising one or more key frames from sequence of frames of the input video and the synthetic ending frame depicting the corrected end state of the state transition using a video generation model to generate an output video depicting a synthetic state transition of the environment that meets the state transition criterion.
14 . The system of claim 13 , wherein the operations comprise:
processing the sequence of the frames of the input video using a visual language model to generate a respective state annotation for each frame in the sequence of the frames of the input video; and
determining the one or more key frames from the sequence of the frames of the input video based on the respective state annotations for the sequence of the frames of the input video.
15 . The system of claim 13 , wherein the sequence of the frames of the input video comprises a transition frame, wherein the state transition of the environment depicted in the transition frame and frames in the input video before the transition frame meets the state transition criterion, and the state transition of the environment depicted in frames in the input video after the transition frame does not meet the state transition criterion.
16 . The system of claim 15 , wherein the input to the video generation model comprises the transition frame.
17 . The system of claim 15 , wherein the input to the video generation model comprises one or more frames that precede the transition frame.
18 . The system of claim 13 , wherein the operations comprise:
processing an input comprising at least the frame that depicts the incorrect end state of the state transition using an image editing model to generate the synthetic ending frame depicting the corrected end state of the state transition.
19 . The system of claim 13 , wherein the one or more key frames comprise a starting frame depicting the environment before the state transition happens, and the operations comprise:
obtaining a set of points on the starting frame;
obtaining a target trajectory for the set of points associated with a target condition of the environment; and
processing the input comprising the starting frame, the synthetic ending frame, and the target trajectory for the set of points using the video generation model to generate the output video that meets the state transition criterion and is conditioned on the starting frame, the synthetic ending frame, and the target trajectory for the set of points, wherein a first frame of the output video is the starting frame, a last frame of the output video is the synthetic ending frame, and locations for the set of points in at least some frames of the output video approximately follow the target trajectory.
20 . One or more non-transitory storage media encoded with instructions that when executed by a computing device cause the computing device to perform operations comprising:
obtaining an input video comprising a sequence of frames depicting a state transition of an environment that does not meet a state transition criterion;
generating, based on a frame from a sequence of frames that depicts an incorrect end state of the state transition, a synthetic ending frame depicting a corrected end state of the state transition; and
processing an input comprising one or more key frames from sequence of frames of the input video and the synthetic ending frame depicting the corrected end state of the state transition using a video generation model to generate an output video depicting a synthetic state transition of the environment that meets the state transition criterion.