Object-based prediction of scene transitions using neural networks
Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for predicting scene transitions. A computer system receives an input sequence of images of a scene with each image corresponding to a different time point in an observation time sequence. For each time point, the system processes the corresponding image using a decomposition neural network to generate one or more feature representations. The system processes the feature representations for the time points using an alignment neural network to generate a set of aligned sequences of feature representations. The system further processes the set of aligned sequences of feature representations using a transition neural network to predict, for each of the aligned sequences of feature representations, one or more feature representations that represent predicted features of the object represented by the aligned sequence at one or more successive time points.
1 . A method performed by one or more computers, the method comprising:
receiving an input sequence of images of a scene, each image corresponding to a different time point in an observation time sequence;
for each time point, processing the corresponding image using a decomposition neural network to generate one or more feature representations, each feature representation representing one of a set of objects that appears in the corresponding image;
processing the feature representations for the time points using an alignment neural network to generate a set of aligned sequences of feature representations, wherein each aligned sequence of feature representations represents a same object across the different time points in the observation time sequence; and
processing the set of aligned sequences of feature representations using a transition neural network to predict, for each of the aligned sequences of feature representations, one or more feature representations that represent predicted features of the object represented by the aligned sequence at one or more successive time points.
2 . The method of claim 1 , wherein the decomposition neural network, the alignment neural network, and the transition neural network have been jointly trained to minimize a joint loss.
3 . The method of claim 2 , wherein the joint loss includes an unsupervised decomposition loss measuring a difference between an input image and a reconstructed image based on object masks reconstructed by a decoder of the decomposition neural network.
4 . The method of claim 2 , wherein the joint loss includes a transition loss measuring a difference between an output of the transition neural network for a current time point based on aligned historical feature representations and aligned current feature representations outputted by the alignment neural network.
5 . The method of claim 1 , wherein:
the set of aligned sequences of feature representations are respectively stored in a set of slots in a slot-based memory.
6 . The method of any one of claim 5 , wherein the alignment network predicts an alignment output that defines an assignment for assigning current feature representations at a current time point to each of the set of slots in the slot-based memory.
7 . The method of claim 6 , wherein the alignment output includes an adjacency matrix for performing alignment of the current feature representations with the set of slots in the slot-based memory.
8 . The method of claim 7 , further comprising: performing a hard alignment by generating a permutation matrix based on the current feature representations and the adjacency matrix.
9 . The method of claim 8 , wherein the hard alignment is performed using a Hungarian algorithm.
10 . The method of claim 8 , wherein:
the set of aligned sequences of feature representations are generated by performing the hard alignment on the feature representations generated from the input sequence of images.
11 . The method of claim 8 , wherein the decomposition neural network, the alignment neural network, and the transition neural network have been jointly trained to minimize a joint loss, and the joint loss includes an unsupervised alignment loss, the unsupervised alignment loss including a reconstruction loss that measures a difference between an output of the transition neural network for the current time point based on aligned historical feature representations and aligned current feature representations generated by applying the adjacency matrix to the current feature representations.
12 . The method of claim 11 , wherein the aligned current feature representations are generated by performing a soft alignment that multiplies the adjacency matrix with the current feature representations.
13 . The method of claim 11 , wherein the unsupervised alignment loss further includes a measurement of an entropy of the alignment output.
14 . The method of claim 11 , wherein the unsupervised alignment loss further includes a measurement that penalizes columns of the adjacency matrix that sum to more than one.
15 . The method of claim 1 , wherein an environment corresponding to the scene is being interacted with by an agent.
16 . The method of claim 15 , wherein the input sequence of images of the scene are obtained by the agent.
17 . The method of claim 15 , wherein processing the set of aligned sequences of feature representations using the transition neural network includes:
generating aligned feature representations at a next time point by applying the transition neural network on (i) the set of aligned sequences of feature representations generated from the input sequence of images and (ii) one or more actions taken by the agent after the last time point in the observation time sequence.
18 . The method of claim 17 , wherein:
the transition neural network includes a recurrent neural network (RNN); and
processing the set of aligned sequences of feature representations using the transition neural network includes:
for each aligned sequence of feature representations, processing, using the RNN, an RNN input derived based on the aligned sequence of feature representations, the one or more actions taken by the agent, and a hidden state, to generate an update; and
for the next time point after the current time point, generating a predicted feature representation for the next time point by applying the update to the feature representation for the current time point.
19 . The method of claim 18 , wherein the set of aligned sequences of feature representations are respectively stored in a set of slots in a slot-based memory, and the RNN is a long short-term memory (LSTM) network applied independently to each slot and sharing weights between slots.
20 . The method of claim 18 , wherein:
the transition neural network further includes a transformer network, wherein the transformer network is configured to process the set of aligned sequences to generate a set of encoded aligned sequences of feature representations that encode interaction among the set of objects; and
the RNN input includes the set of encoded aligned feature representations.
21 . The method of claim 15 , further comprising:
controlling the agent according to at least an output of the transition neural network.
22 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
receiving an input sequence of images of a scene, each image corresponding to a different time point in an observation time sequence;
for each time point, processing the corresponding image using a decomposition neural network to generate one or more feature representations, each feature representation representing one of a set of objects that appears in the corresponding image;
processing the feature representations for the time points using an alignment neural network to generate a set of aligned sequences of feature representations, wherein each aligned sequence of feature representations represents a same object across the different time points in the observation time sequence; and
processing the set of aligned sequences of feature representations using a transition neural network to predict, for each of the aligned sequences of feature representations, one or more feature representations that represent predicted features of the object represented by the aligned sequence at one or more successive time points.
23 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving an input sequence of images of a scene, each image corresponding to a different time point in an observation time sequence;
for each time point, processing the corresponding image using a decomposition neural network to generate one or more feature representations, each feature representation representing one of a set of objects that appears in the corresponding image;
processing the feature representations for the time points using an alignment neural network to generate a set of aligned sequences of feature representations, wherein each aligned sequence of feature representations represents a same object across the different time points in the observation time sequence; and
processing the set of aligned sequences of feature representations using a transition neural network to predict, for each of the aligned sequences of feature representations, one or more feature representations that represent predicted features of the object represented by the aligned sequence at one or more successive time points.