Methods and systems for object tracking
The present disclosure relates to methods and systems for object tracking, for example for object detection and grid segmentation using recurrent neural networks. A computer implemented method for object tracking comprises the following steps carried out by computer hardware components: providing random values as a hidden state of a trained neural network for an initial time step, wherein the hidden state represents an encoding of sensor data acquired over consecutive time steps in a grid structure, wherein the hidden state further represents an offset indicating a movement of the object between the consecutive time steps; iteratively determining an updated hidden state by processing a present hidden state and present sensor data using the trained neural network; and determining object tracking information based on the updated hidden state.
1 . A computer implemented method for tracking an object, the method comprising:
providing random values as a hidden state of a trained neural network for an initial time step, the hidden state representing:
an encoding of sensor data acquired over consecutive time steps in a grid structure, and
an offset indicating a movement of the object between the consecutive time steps;
iteratively determining an updated hidden state by processing a present hidden state and present sensor data using the trained neural network;
determining object tracking information based on the updated hidden state;
acquiring the present hidden state for a first pre-determined time step and an offset for the first pre-determined time step;
determining a transformed hidden state based on the present hidden state for the first pre-determined time step and the offset for the first pre-determined time step;
acquiring sensor data for a second pre-determined time step;
evaluating the trained neural network based on the transformed hidden state and the sensor data for the second pre-determined time step to obtain intermediate data for the second pre-determined time step;
determining an offset for the second pre-determined time step based on the intermediate data; and
determining the updated hidden state for the second pre-determined time step based on the intermediate data and the offset for the second pre-determined time step.
2 . The method of claim 1 , wherein the offset for the first pre-determined time step is determined based on the present hidden state for the first pre-determined time step.
3 . The method of claim 1 , wherein the updated hidden state is determined further based on at least one of:
concatenating the intermediate data and the offset for the second pre-determined time step;
the offset for the first pre-determined time step;
a mean of the offset for the first pre-determined time step and the offset for the second pre-determined time step; or
an attention method based on the offset for the first pre-determined time step and the offset for the second pre-determined time step.
4 . The method of claim 1 , further comprising:
determining second intermediate data based on the transformed hidden state using a matching method; and
evaluating the trained neural network based on the second intermediate data.
5 . The method of claim 4 , wherein using the matching method further comprises:
determining a similarity between an embedding vector of the first pre-determined time step and an embedding vector of the second pre-determined time step.
6 . The method of claim 5 , wherein the embedding vector of the first pre-determined time step comprises an embedding based on the transformed hidden state for the first pre-determined time step.
7 . The method of claim 5 , wherein the embedding vector of the second pre-determined time step comprises an embedding based on the sensor data for the second pre-determined time step.
8 . The method of claim 5 , wherein the similarity is determined based on a dot product between the embedding vector of the first pre-determined time step and the embedding vector of the second pre-determined time step.
9 . The method of claim 1 , wherein the updated hidden state for the second pre-determined time step is determined based on speed matching.
10 . The method of claim 1 , further comprising:
performing object detection based on the updated hidden state for the second pre-determined time step.
11 . The method of claim 1 , further comprising:
performing grid segmentation based on the updated hidden state for the second pre-determined time step.
12 . The method of claim 1 , wherein the offset for the second
pre-determined time step is determined based on sampling.
13 . The method of claim 12 , wherein the sampling is based on a Gaussian distribution.
14 . The method of claim 1 , wherein the trained neural network comprises a recurrent neural network.
15 . The method of claim 14 , wherein the recurrent neural network comprises at least one of long short-term memory (LSTM) or gated recurrent units (GRUs).
16 . The method of claim 1 , wherein the object comprises an ego vehicle.
17 . The method of claim 1 , wherein the object comprises an object with movements.
18 . A computer system comprising:
a plurality of computer hardware components including a processor; and
a non-transitory computer readable medium comprising instructions, which when executed by the processor, cause the processor to:
provide random values as a hidden state of a trained neural network for an initial time step, the hidden state representing:
an encoding of sensor data acquired over consecutive time steps in a grid structure, and
an offset indicating a movement of an object between the consecutive time steps;
iteratively determine an updated hidden state by processing a present hidden state and present sensor data using the trained neural network;
determine object tracking information for the object based on the updated hidden state;
acquire the present hidden state for a first pre-determined time step and an offset for the first pre-determined time step;
determine a transformed hidden state based on the present hidden state for the first pre-determined time step and the offset for the first pre-determined time step;
acquire sensor data for a second pre-determined time step;
evaluate the trained neural network based on the transformed hidden state and the sensor data for the second pre-determined time step to obtain intermediate data for the second pre-determined time step;
determine an offset for the second pre-determined time step based on the intermediate data; and
determine the updated hidden state for the second pre-determined time step based on the intermediate data and the offset for the second pre-determined time step.
19 . A vehicle comprising:
a sensor;
a processor;
a non-transitory computer-readable medium comprising instructions, which when executed by the processor cause the processor to:
provide random values as a hidden state of a trained neural network for an initial time step, the hidden state representing:
an encoding of sensor data from the sensor acquired over consecutive time steps in a grid structure, and
an offset indicating a movement of an object between the consecutive time steps;
iteratively determine an updated hidden state by processing a present hidden state and present sensor data using the trained neural network;
determine object tracking information for the object based on the updated hidden state;
acquire the present hidden state for a first pre-determined time step and an offset for the first pre-determined time step;
determine a transformed hidden state based on the present hidden state for the first pre-determined time step and the offset for the first pre-determined time step;
acquire sensor data for a second pre-determined time step;
evaluate the trained neural network based on the transformed hidden state and the sensor data for the second pre-determined time step to obtain intermediate data for the second pre-determined time step;
determine an offset for the second pre-determined time step based on the intermediate data; and
determine the updated hidden state for the second pre-determined time step based on the intermediate data and the offset for the second pre-determined time step.