Spatio-temporal features for video analysis

1. A method of determining a spatio-temporal feature value for frames of a sequence of video using a neural network that has a motion layer for determining the spatio-temporal feature value as an intermediate layer, the method comprising:

receiving a first frame and second frame from the sequence of video;

determining spatial feature values in each of the first and second frames using a layer upstream of the motion layer of the neural network;

determining at least one emergence response value indicating emergence of the spatial feature values between the first and second frames and at least one vanishing response value indicating vanishing of the spatial feature values between the first and second frames using the motion layer;

combining the emergence response value and the vanishing response value using the motion layer;

determining a change in the spatial feature values between the first and second frames based on the combined emergence response value and vanishing response value using the motion layer;

determining the spatio-temporal feature value by combining the determined change in the spatial feature values using the motion layer; and

performing classification regarding an object included in the sequence of video, based on the spatio-temporal feature value, using a layer downstream of the motion layer of the neural network.

2. The method according to claim 1 , wherein the change in the spatial feature values is determined based on background motion between the first and second frames.

3. The method according to claim 1 , further comprising determining at least one emergence response value for determining the change in the spatial feature values.

4. The method according to claim 3 , further comprising smoothing the emergence response value.

5. The method according to claim 1 , further comprising determining at least one vanishing response value for determining the change in the spatial feature values.

6. The method according to claim 5 , further comprising smoothing the vanishing response value.

7. The method according to claim 1 , wherein the emergence response value and the vanishing response value are combined using a geometric mean.

8. The method according to claim 1 , wherein a spatio-temporal feature value is determined for each of a plurality of directions.

9. The method according to claim 1 , wherein the spatio-temporal feature value is determined by combining motion response values.

10. The method according to claim 1 , wherein the vanishing response corresponds to a decrease of the spatial feature values over the time period between the first and second frames and the emergence response corresponds to an increase of the spatial feature values over the time period between the two first and second frames.

11. An apparatus for determining a spatio-temporal feature value for frames of a sequence of video using a neural network that has a motion layer for determining the spatio-temporal feature value as an intermediate layer, the apparatus comprising:

at least one processor; and

a memory that is in communication with the at least one processor and stores one or more computer-readable instructions, wherein the computer-readable instructions cause, when executed by the at least one processor, the at least one processor to operate to:

receive a first frame and second frame from the sequence of video;

determine spatial feature values in each of the first and second frames using a layer upstream of the motion layer of the neural network;

determine at least one emergence response value indicating emergence of the spatial feature values between the first and second frames and at least one vanishing response value indicating vanishing of the spatial feature values between the first and second frames using the motion layer;

combine the emergence response value and the vanishing response value using the motion layer;

determine a change in the spatial feature values between the first and second frames based on the combined emergence response value and vanishing response value using the motion layer;

determine the spatio-temporal feature value by combining the determined change in spatial feature values for each of the spatial feature functions; and

perform classification regarding an object included in the sequence of video, based on the spatio-temporal feature value, using a layer downstream of the motion layer of the neural network.

12. A system for determining a spatio-temporal feature value for frames of a sequence of video using a neural network that has a motion layer for determining the spatio-temporal feature value as an intermediate layer, the system comprising:

a memory for storing data and a computer program;

a processor coupled to the memory for executing the computer program, the program having instructions for:

receiving a first frame and second frame from the sequence of video;

determining spatial feature values in each of the first and second frames using a layer upstream of the motion layer of the neural network;

determining at least one emergence response value indicating emergence of the spatial feature values between the first and second frames and at least one vanishing response value indicating vanishing of the spatial feature values between the first and second frames using the motion layer;

combining the emergence response value and the vanishing response value using the motion layer;

determining a change in the spatial feature values between the first and second frames based on the combined emergence response value and vanishing response value using the motion layer;

determining the spatio-temporal feature value by combining the determined change in spatial feature values using the motion layer; and

performing classification regarding an object included in the sequence of video, based on the spatio-temporal feature value, using a layer downstream of the motion layer of the neural network.

13. A non-transitory computer readable medium having a computer program for determining a spatio-temporal feature value for frames of a sequence of video using a neural network that has a motion layer for determining the spatio-temporal feature value as an intermediate layer, the program comprising:

code for receiving a first frame and second frame from the sequence of video;

code for determining spatial feature values in each of the first and second frames using a layer upstream of the motion layer of the neural network;

code for determining at least one emergence response value indicating emergence of the spatial feature values between the first and second frames and at least one vanishing response value indicating vanishing of the spatial feature values between the first and second frames using the motion layer;

code for combining the emergence response value and the vanishing response value using the motion layer;

code for determining a change in the spatial feature values between the first and second frames based on the combined emergence response value and vanishing response value using the motion layer;

code for determining the spatio-temporal feature value by combining the determined change in spatial feature values for each of the spatial feature functions; and

code for performing classification regarding an object included in the sequence of video, based on the spatio-temporal feature value, using a layer downstream of the motion layer of the neural network.