Ego motion estimation based on consecutive time frames input to machine learning model
Disclosed is an ego motion estimation method and apparatus. The ego motion estimation apparatus may generate input data based on radar sensing data collected by one or more radar sensors for each of a plurality of time frames, and estimate ego motion information based on the input data using a motion recognition model.
1. A processor-implemented ego motion estimation method of an electronic apparatus, the method comprising:
generating input data based on radar sensing data, collected by one or more radar sensors, for each of a plurality of time frames;
for each of the plurality of time frames, estimating ego motion information of the time frame by executing, using the input data of the time frame and the input data of a previous frame of the plurality of time frames as inputs to a machine learning motion recognition model, the machine learning motion recognition model generating extracted feature data of the time frame, and generating estimated respective ego motion information of the time frame dependent on the extracted feature data of the time frame; and
recognizing a motion of an object, exterior of the electronic apparatus, based on the estimated respective ego motion information of the plurality of time frames,
wherein, for each of the plurality of time frames, the estimated respective ego motion information of the time frame includes information of a position and/or pose of the apparatus relative to the object.
2. The method of claim 1 , wherein, for each of the plurality of time frames, the estimating of the ego motion information comprises:
performing the extracting of the feature data using a first model of the machine learning motion recognition model; and
performing the generating of the estimated respective ego motion information using a second model of the machine learning motion recognition model.
3. The method of claim 1 , wherein the estimating of the ego motion information comprises executing the machine learning motion recognition model using radar sensing data corresponding to at least two time frames input to a layer of the machine learning motion recognition model,
wherein the at least two time frames comprises the time frame and the previous frame, and
wherein the radar sensing data comprises the extracted feature data of the time frame.
4. The method of claim 1 , wherein, for each of the plurality of time frames, the estimating of the ego motion information comprises:
performing the generating of the extracted feature data by extracting the feature data of the time frame from the input data of the time frame and the input data of the previous frame of the time frames, using a first model of the machine learning motion recognition model; and
performing the generating of the estimated respective ego motion information of the time frame based on the extracted feature data of the time frame, using a second model of the machine learning motion recognition model.
5. The method of claim 4 , wherein the estimating of the ego motion information comprises:
extracting subsequent feature data from input data of a subsequent frame of the time frames and the time frame, using the first model; and
determining subsequent ego motion information based on the subsequent feature data, using the second model.
6. The method of claim 5 , wherein the extracting of the subsequent feature data comprises excluding the input data of the previous frame from the extracting of the subsequent feature data.
7. The method of claim 4 , wherein the first model comprises a convolutional neural network and the second model comprises a recurrent neural network.
8. The method of claim 1 ,
wherein the machine learning motion recognition model comprises:
a first model including layers, wherein each of the layers corresponds to a respective one of the plurality of time frames; and
a second model connected to the layers of the first model, and
wherein, for each of the plurality of time frames, the estimating of the ego motion information comprises:
performing the generating of the extracted feature information by extracting, using a layer of the layers in the first model corresponding to a time frame of the plurality of time frames, the feature data of the time frame from the input data of the time frame; and
performing the generating of the estimated respective ego motion information of the time frame based on the extracted feature data of the time frame using the second model.
9. The method of claim 1 , wherein, for each of the plurality of time frames, the estimating of the ego motion information comprises:
performing the generating of the extracted feature information by extracting the feature data of the time frame from the input data of the time frame using a first model of the machine learning motion recognition model;
loading previous feature data corresponding to the previous frame from a memory; and
performing the generating of the estimated respective ego motion information based on the previous feature data and the feature data of the time frame using a second model of the machine learning motion recognition model.
10. The method of claim 1 , further comprising:
for each of the plurality of time frames, storing, in a memory, the extracted feature data of the time frame that was generated, for the time frame, by a first model included in the machine learning motion recognition model.
11. The method of claim 1 , wherein the generating of the input data comprises:
detecting a radar signal using one or more radar sensors arranged along an outer face of the electronic apparatus or another apparatus that comprises the electronic apparatus;
generating the radar sensing data by preprocessing the detected radar signal; and
generating the input data based on the preprocessed radar signal.
12. The method of claim 1 , wherein the generating of the input data comprises:
selecting two or more items of the radar sensing data corresponding to time frames of the time frames differing from each other by a preset time interval; and
generating the input data based on the selected items of radar sensor data.
13. The method of claim 1 , wherein the generating of the input data comprises:
excluding radar sensing data corresponding to a first frame of the plurality of time frames stacked in the input data, in response to radar sensing data corresponding to a subsequent frame being received.
14. The method of claim 1 , wherein the generating of the input data comprises:
generating radar sensing data indicating an angle and a distance from a point detected by the one or more radar sensors for each quantized velocity from a radar signal.
15. The method of claim 1 , wherein the generating of the input data comprises:
generating input data indicating a horizontal angle and a distance from a point detected by the one or more radar sensors for each quantized elevation angle from the radar sensing data.
16. The method of claim 1 , wherein the generating of the input data comprises:
classifying the radar sensing data into static data of a static point detected by the one or more radar sensors and dynamic data of a dynamic point detected by the one or more radar sensors;
generating static input data indicating a horizontal angle and a distance from the static point for each quantized elevation angle based on the static data; and
generating dynamic input data indicating a horizontal angle and a distance from the dynamic point for each quantized elevation angle based on the dynamic data.
17. The method of claim 1 , wherein the machine learning motion recognition model includes a convolutional neural network and a recurrent neural network (RNN).
18. The method of claim 17 , wherein the RNN is a bi-directional neural network.
19. The method of claim 1 , wherein the estimating of the ego motion information comprises:
in response to a plurality of items of radar sensing data corresponding to a plurality of time frames being stacked in the input data, performing the generating of the estimated respective ego motion information for each of the plurality of time frames.
20. The method of claim 1 , further comprising:
generating reference input data for a plurality of training time frames based on reference radar sensing data and based on reference output data corresponding to the reference input data; and
generating the machine learning motion recognition model by training parameters of a model to output the reference output data based on the reference input data through a back-propagation training process.
21. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 1 .
22. An electronic apparatus comprising:
one or more radar sensors for collecting radar sensing data; and
one or more processors configured to execute instructions; and
one or more memories storing the instructions, which when executed by the one or more processors configures the one or more processors to:
generate input data based on the radar sensing data for each of a plurality of time frames of the radar sensing data;
for each of the plurality of time frames estimate ego motion information of the time frame through execution, using the input data of the time frame and the input data of a previous frame of the plurality of time frames as inputs to a machine learning motion recognition model, of the machine learning motion recognition model to generate extracted feature data of the time frame, and generate estimated respective ego motion information of the time frame dependent on the extracted feature data of the time frame; and
recognize a motion of an object, exterior of the electronic apparatus, based on the estimated respective ego motion information of the plurality of time frames,
wherein, for each of the plurality of time frames, the estimated respective ego motion information of the time frame includes information of a position and/or pose of the apparatus relative to the object.
23. The method of claim 1 , further comprising performing autonomous driving or driver assistance operations of a vehicle, as the electronic apparatus that further includes the one or more radar sensors, relative to the object based on the estimated respective ego motion information of the plurality of time frames.
24. The method of claim 1 , further comprising generating a moving path using the information of the position and/or pose of the apparatus of each of the estimated respective ego motion information of the plurality of time frames.
25. The method of claim 1 , further comprising generating an image with the object using the information of the position and/or pose of the apparatus of each of the estimated respective ego motion information of the plurality of time frames.