Processing of sequential data via machine learning models featuring temporal residual connections
Systems and methods can include or leverage a machine-learned model (e.g., a convolutional neural network) that includes one or more temporal residual connections. In particular, each temporal residual connection can respectively supply one or more sets of intermediate feature data generated by a current instantiation of the model from a current sequential input to one or more other instantiations of the machine-learned model applied to process one or more other sequential inputs. For example, the other instantiations of the machine-learned model can include subsequent instantiations of the machine-learned model applied to process one or more subsequent sequential inputs that follow the current sequential input in a sequence and/or preceding instantiations of the machine-learned model applied to process one or more preceding sequential inputs that precede the current sequential input in a sequence.
1 . A computing system for improved temporal processing of sequential data, the computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store:
a machine-learned convolutional neural network that comprises one or more temporal residual connections that respectively supply one or more sets of intermediate feature data generated from a current sequential input to a plurality of other instantiations of the machine-learned convolutional neural network applied to process a plurality of other sequential inputs; and
instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
for each of a plurality of sequential inputs included in a sequence:
processing the current sequential input with at least a portion of a current instantiation of the machine-learned convolutional neural network to generate a current set of intermediate feature data;
storing the current set of intermediate feature data for provision to a plurality of subsequent instantiations of the machine-learned convolutional neural network applied to process a plurality of subsequent sequential inputs that are subsequent to the current sequential input in the sequence;
accessing a plurality of sets of preceding intermediate feature data generated by a plurality of preceding instantiations of the machine-learned convolutional neural network applied to process a plurality of preceding sequential inputs that precede the current sequential input in the sequence, wherein the plurality of sets of preceding intermediate feature data comprise feature maps from a plurality of different previous timestamps; and
generating a model output from the current instantiation of the machine-learned convolutional neural network based at least in part on the current set of intermediate feature data and the plurality of sets of preceding intermediate feature data.
2 . The computing system of claim 1 , wherein the one or more temporal residual connections consist only of forward-propagating temporal residual connections that supply the one or more sets of intermediate feature data generated from the current sequential input to the subsequent instantiations of the machine-learned convolutional neural network.
3 . The computing system of claim 1 , wherein the machine-learned convolutional neural network comprises both:
forward-propagating temporal residual connections that supply the one or more sets of intermediate feature data generated from the current sequential input to the subsequent instantiations of the machine-learned convolutional neural network; and
backward-propagating temporal residual connections that supply the one or more sets of intermediate feature data generated from the current sequential input to the preceding instantiations of the machine-learned convolutional neural network.
4 . The computing system of claim 1 , wherein at least one of the one or more temporal residual connections is configured to supply the one or more sets of intermediate feature data to a same layer of the plurality of instantiations of the machine-learned convolutional neural network.
5 . The computing system of claim 1 , wherein at least one of the one or more temporal residual connections is configured to supply the one or more sets of intermediate feature data to a different layer of the plurality of other instantiations of the machine-learned convolutional neural network.
6 . The computing system of claim 1 , wherein the one or more temporal residual connections comprise a plurality of temporal residual connections present at different respective depths within the machine-learned convolutional neural network.
7 . The computing system of claim 1 , wherein generating the model output from the current instantiation of the machine-learned convolutional neural network based at least in part on the current set of intermediate feature data and the plurality of sets of preceding intermediate feature data comprises:
combining at least one of the sets of preceding intermediate feature data with at least one existing set of feature data to form a combined set of feature data; and
generating the model output from the current instantiation of the machine-learned convolutional neural network based at least in part on the combined set of feature data.
8 . The computing system of claim 7 , wherein combining the at least one of the sets of preceding intermediate feature data with the at least one existing set of feature data to form the combined set of feature data comprises summing the at least one of the sets of preceding intermediate feature data with the at least one existing set of feature data.
9 . The computing system of claim 7 , wherein combining the at least one of the sets of preceding intermediate feature data with the at least one existing set of feature data to form the combined set of feature data comprises:
concatenating the at least one of the sets of preceding intermediate feature data with the at least one existing set of feature data; and
applying one or more convolutions to the concatenated data.
10 . The computing system of claim 7 , wherein combining the at least one of the sets of preceding intermediate feature data with the at least one existing set of feature data to form the combined set of feature data comprises:
concatenating the at least one of the sets of preceding intermediate feature data with the at least one existing set of feature data;
applying multiple convolutional filters to the concatenated data in parallel; and
combining the outputs of the multiple convolutional filters.
11 . The computing system of claim 10 , wherein the multiple convolutional filters have different respective filter sizes.
12 . The computing system of claim 10 , wherein the multiple convolutional filters have different respective dilation rates.
13 . The computing system of claim 7 , wherein the at least one existing set of feature data comprises the current set of intermediate feature data.
14 . The computing system of claim 1 , wherein the plurality of subsequent instantiations of the machine-learned convolutional neural network applied to process the plurality of subsequent sequential inputs comprise a next sequential instantiation of the machine-learned convolutional neural network applied to process a next sequential input in the sequence.
15 . The computing system of claim 1 , wherein the plurality of subsequent instantiations of the machine-learned convolutional neural network applied to process the plurality of subsequent sequential inputs comprise a greater-than-next sequential instantiation of the machine-learned convolutional neural network applied to process a greater-than-next sequential input in the sequence.
16 . The computing system of claim 1 , wherein the current set of intermediate feature data comprises an activation map for a convolutional layer of the machine-learned convolutional neural network.
17 . The computing system of claim 1 , wherein the machine-learned convolutional neural network comprises one or more convolutional layers followed by a long short term memory layer, wherein the one or more temporal residual connections are present at the one or more convolutional layers.
18 . The computing system of claim 1 , wherein the plurality of sequential inputs included in the sequence comprise a plurality of image frames included in a video.
19 . The computing system of claim 1 , wherein the plurality of sequential inputs included in the sequence comprise a plurality of sets of Light Detection and Ranging (LiDAR) data included in a LiDAR data sequence.
20 . The computing system of claim 1 , wherein the machine-learned convolutional neural network is configured to perform a task, wherein the task comprises action recognition or object detection.
21 . A computer-implemented method, the method comprising:
for each of a plurality of sequential inputs included in a sequence:
processing a current sequential input with at least a portion of a current instantiation of a machine-learned convolutional neural network to generate a current set of intermediate feature data;
storing the current set of intermediate feature data for provision to a plurality of subsequent instantiations of the machine-learned convolutional neural network applied to process a plurality of subsequent sequential inputs that are subsequent to the current sequential input in the sequence, wherein the current instantiation of the machine-learned convolutional neural network is connected to the plurality of subsequent instantiations of the machine-learned convolutional neural network via one or more temporal residual connections;
accessing a plurality of sets of preceding intermediate feature data generated by a plurality of preceding instantiations of the machine-learned convolutional neural network applied to process a plurality of preceding sequential inputs that precede the current sequential input in the sequence, wherein the current instantiation of the machine-learned convolutional neural network is connected to the a plurality of preceding instantiations of the machine-learned convolutional neural network via the one or more temporal residual connections, wherein the plurality of sets of preceding intermediate feature data comprise feature maps from a plurality of different previous timestamps; and
generating a model output from the current instantiation of the machine-learned convolutional neural network based at least in part on the current set of intermediate feature data and the plurality of sets of preceding intermediate feature data.
22 . One or more non-transitory computer-readable media that collectively store:
a machine-learned convolutional neural network that comprises one or more temporal residual connections that respectively supply one or more sets of intermediate feature data generated from a current sequential input to a plurality of other instantiations of the machine-learned convolutional neural network applied to process a plurality of other sequential inputs; and
computer-readable instructions for executing the machine-learned convolutional neural network
instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising:
for each of a plurality of sequential inputs included in a sequence:
processing the current sequential input with at least a portion of a current instantiation of the machine-learned convolutional neural network to generate a current set of intermediate feature data;
storing the current set of intermediate feature data for provision to a plurality of subsequent instantiations of the machine-learned convolutional neural network applied to process a plurality of subsequent sequential inputs that are subsequent to the current sequential input in the sequence;
accessing a plurality of sets of preceding intermediate feature data generated by a plurality of preceding instantiations of the machine-learned convolutional neural network applied to process a plurality of preceding sequential inputs that precede the current sequential input in the sequence, wherein the plurality of sets of preceding intermediate feature data comprise feature maps from a plurality of different previous timestamps; and
generating a model output from the current instantiation of the machine-learned convolutional neural network based at least in part on the current set of intermediate feature data and the plurality of sets of preceding intermediate feature data.