Spatial action localization in the future (SALF)

1 . A system for spatial action localization in the future (SALF), comprising:

a memory storing one or more instructions; and

a processor executing one or more of the instructions stored on the memory to perform:

feeding a frame from a time step of a video clip through an encoder to generate a latent feature;

feeding the latent feature and one or more latent features from one or more previous time steps of the video clip through a future feature predictor to generate a cumulative information for the time step by extrapolating the latent feature based on an ordinary differential equation (ODE) function modeled using self-attention and a causal mask or multi-head attention and the causal mask and wherein the ODE function considers function values from multiple previous steps; and

feeding the cumulative information through a decoder to generate a predicted action area and a predicted action classification associated with the predicted action area.

2 . The system for SALF of claim 1 , wherein the feeding the frame from the time step of the video clip through the encoder to generate the latent feature includes:

feeding the frame from the time step of the video clip through a 2-dimensional (2D) neural network to generate a first output;

feeding the frame from the time step of the video clip through a 3-dimensional (3D) neural network to generate a second output; and

concatenating the first output and the second output to generate the latent feature.

3 . The system for SALF of claim 2 , wherein the 2D neural network is a 2D convolutional neural network (CNN).

4 . The system for SALF of claim 2 , wherein the 3D neural network is a 3D convolutional neural network (CNN).

5 . The system for SALF of claim 1 , wherein the future feature predictor generates the cumulative information for the time step by extrapolating the latent feature from the time step and one or more of the latent features from one or more of the previous time steps of the video clip to a future time horizon.

6 . The system for SALF of claim 1 , wherein the feeding the latent feature and one or more of the latent features from one or more of the previous time steps of the video clip through the future feature predictor to generate the cumulative information for the time step includes feeding all of the latent features from all of the previous time steps of the video clip through the future feature predictor.

7 . The system for SALF of claim 1 , wherein the frame from the time step of the video clip is a predicted future frame.

8 . The system for SALF of claim 1 , wherein the processor implements an action based on the predicted action area and the predicted action classification.

9 . A computer-implemented method for spatial action localization in the future (SALF), comprising:

feeding a frame from a time step of a video clip through an encoder to generate a latent feature;

feeding the latent feature and one or more latent features from one or more previous time steps of the video clip through a future feature predictor to generate a cumulative information for the time step by extrapolating the latent feature based on an ordinary differential equation (ODE) function modeled using self-attention and a causal mask or multi-head attention and the causal mask and wherein the ODE function considers function values from multiple previous steps; and

feeding the cumulative information through a decoder to generate a predicted action area and a predicted action classification associated with the predicted action area.

10 . The computer-implemented method for SALF of claim 9 , wherein the feeding the frame from the time step of the video clip through the encoder to generate the latent feature includes:

feeding the frame from the time step of the video clip through a 2-dimensional (2D) neural network to generate a first output;

feeding the frame from the time step of the video clip through a 3-dimensional (3D) neural network to generate a second output; and

concatenating the first output and the second output to generate the latent feature.

11 . The computer-implemented method for SALF of claim 10 , wherein the 2D neural network is a 2D convolutional neural network (CNN).

12 . The computer-implemented method for SALF of claim 10 , wherein the 3D neural network is a 3D convolutional neural network (CNN).

13 . The computer-implemented method for SALF of claim 9 , wherein the future feature predictor generates the cumulative information for the time step by extrapolating the latent feature from the time step and one or more of the latent features from one or more of the previous time steps of the video clip to a future time horizon.

14 . A system for spatial action localization in the future (SALF), comprising:

a memory storing one or more instructions;

a processor executing one or more of the instructions stored on the memory to perform:

feeding a frame from a time step of a video clip through an encoder to generate a latent feature;

feeding the latent feature and one or more latent features from one or more previous time steps of the video clip through a future feature predictor to generate a cumulative information for the time step by extrapolating the latent feature based on an ordinary differential equation (ODE) function modeled using self-attention and a causal mask or multi-head attention and the causal mask and wherein the ODE function considers function values from multiple previous steps; and

feeding the cumulative information through a decoder to generate a predicted action area and a predicted action classification associated with the predicted action area; and

a controller controlling an actuator to implement an action based on the predicted action area and the predicted action classification.

15 . The system for SALF of claim 14 , wherein the feeding the frame from the time step of the video clip through the encoder to generate the latent feature includes:

feeding the frame from the time step of the video clip through a 2-dimensional (2D) neural network to generate a first output;

feeding the frame from the time step of the video clip through a 3-dimensional (3D) neural network to generate a second output; and

concatenating the first output and the second output to generate the latent feature.

16 . The system for SALF of claim 15 , wherein the 2D neural network is a 2D convolutional neural network (CNN).

17 . The system for SALF of claim 15 , wherein the 3D neural network is a 3D convolutional neural network (CNN).