Method and system for detecting and alerting collision by autonomous robot based on multi-sensor LSTM

1. A method of detecting a collision by an autonomous robot, the method comprising:

performing preprocess such as distortion correction by a camera lens, illumination effect correction, and contrast equalization on the input image received from an autonomous robot;

performing preprocess to extract from 360° point cloud information based on a position of an installed sensor received from light detection and ranging (LiDAR), distance information associated with a point cloud in front of an autonomous robot on a movement path in operation;

removing three-axis gravitational acceleration and a sensor bias from the three-axis acceleration information received from acceleration sensor, performing preprocess with a high-pass filter for filtering in operation, and output acceleration information;

generating an input of a long short-term memory (LSTM) model by combining the preprocessed input image, the distance information, and the acceleration information;

learning a collision alert situation by inputting the input to the LSTM model; and

determining a collision situation using an output of the LSTM model and a fully connected neural network (FNN) model.

2. The method of claim 1 , wherein the LSTM model is configured to:

learn, when an obstacle is detected using the distance information, but the obstacle is not detected using the preprocessed input image, as a normal situation

learn, when an obstacle is detected using the distance information, and the obstacle is detected using the preprocessed input image, as a collision alert situation; and

learn, when the distance information and a feature point map of the input image decrease, but the acceleration information increases, as the collision situation.

3. The method of claim 1 , wherein the determining of the collision situation comprises, from among three outputs (normal driving, collision alert, collision detection) generated by the FNN model, selecting an output in which a probability distribution value is the greatest based on a normal distribution of the three outputs, and determining the collision situation based on the selected output.

4. The method of claim 1 , further comprising:

determining a cost function for optimizing a training model by comparing the collision situation determined using the FNN model and an actual situation; and

optimizing a training model of the LSTM model and a training model of the FNN model based on the cost function.

5. The method of claim 1 , further comprising:

generating, from the preprocessed input image, a feature point map of the input image,

wherein the generating of the input of the LSTM model comprises combining the feature point map of the input image, the LiDAR distance information, and the acceleration information.

6. The method of claim 5 , wherein the generating of the feature point map comprises:

extracting feature points from the input image using a convolution filter of a preset size and reducing a size of the input image;

linearizing the extracted feature points; and

dividing the linearized feature points by a preset size and generating the feature point map using representative values of the divided feature points.

7. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .

8. A system for detecting a collision by an autonomous robot, the system comprising:

preprocessor to:

perform preprocessing such as distortion correction by a camera lens, illumination effect correction, and contrast equalization on the input image received from an autonomous robot;

perform preprocessing to extract, from 360° point cloud information based on a position of an installed sensor received from light detection and ranging (LiDAR), distance information associated with a point cloud in front of an autonomous robot on a movement path in operation; and

remove three-axis gravitational acceleration and a sensor bias from the three-axis acceleration information received from acceleration sensor, perform preprocessing with a high-pass filter for filtering in operation, and output acceleration information;

a long short-term memory (LSTM) model part configured to:

generate an input of an LSTM model by combining the preprocessed input image, the distance information, and the acceleration information; and

learn a collision alert situation by inputting the input to the LSTM model; and

a fully connected neural network (FNN) model part configured to determine a collision situation using an output of the LSTM model and an FNN model.

9. The system of claim 8 , wherein the LSTM model is configured to:

learn, when an obstacle is detected using the distance information, but the obstacle is not detected using the preprocessed input image, as a normal situation

learn, when an obstacle is detected using the distance information, and the obstacle is detected using the preprocessed input image, as a collision alert situation; and

learn, when the distance information and a feature point map of the input image decrease, but the acceleration information increases, as the collision situation.

10. The system of claim 8 , wherein the FNN model part is configured to, from among three outputs (normal driving, collision alert, collision detection) generated by the FNN model, select an output in which a probability distribution value is the greatest based on a normal distribution of the three outputs and determine the collision situation based on the selected output.

11. The system of claim 8 , wherein the FNN model part is configured to determine a cost function for optimizing a training model by comparing the collision situation determined using the FNN model and an actual situation, and optimize a training model of the FNN model based on the cost function,

wherein the LSTM model is configured to optimize a training model of the LSTM model based on the cost function.

12. The system of claim 8 , further comprising:

a feature point map generator configured to generate, from the preprocessed input image, a feature point map of the input image,

wherein the LSTM model part is configured to combine the feature point map of the input image, the LiDAR distance information, and the acceleration information.

13. The system of claim 12 , wherein the feature point map generator is configured to extract feature points from the input image using a convolution filter of a preset size, reduce a size of the input image, linearize the extracted feature points, divide the linearized feature points by a preset size, and generate the feature point map using representative values of the divided feature points.