Vehicle drivable area detection method, system, and autonomous vehicle using the system
This application discloses a vehicle drivable area detection method, an autonomous driving assistance system, and an autonomous driving vehicle. The method includes: processing, by using a neural network, image data obtained by a camera apparatus, to obtain a first probability distribution of an obstacle; obtaining a second probability distribution of the obstacle based on a time of flight and an echo width of a radar echo signal; and obtaining, based on the first probability distribution of the obstacle and the second probability distribution of the obstacle, a drivable area of a vehicle represented by a probability, where the probability is a probability that the vehicle cannot drive through the area. The autonomous driving assistance system includes a camera apparatus, at least one radar, and a processor. The system is configured with the technical solutions that can implement the method. The autonomous driving vehicle includes the foregoing autonomous driving assistance system.
1. A vehicle drivable area detection method, comprising:
processing, using a neural network, image data obtained by a camera apparatus, to obtain a first probability distribution of an obstacle;
obtaining a second probability distribution of the obstacle based on a time of flight and an echo width of a radar echo signal, wherein the echo width is a difference between a second time of flight of the echo signal and a first time of flight of the echo signal, wherein the second time of flight corresponds to a longest echo distance between a radar and the obstacle, and wherein the first time of flight corresponds to a shortest echo distance between the radar and the obstacle;
obtaining, based on the first probability distribution of the obstacle and the second probability distribution of the obstacle, a drivable area of a vehicle represented by a probability, wherein the probability is a probability that the vehicle cannot drive through the area; and either:
a) wherein the vehicle is an autonomous vehicle, planning a driving route for the vehicle based on the obtained drivable area;
or
b) wherein the vehicle is a manually driven vehicle, wherein the drivable area of the vehicle is represented in a form of a probability grid map; and wherein the probability that the vehicle cannot drive through the area is represented in the probability grid map, displaying the probability grid map on a display of the vehicle.
2. The method according to claim 1 , wherein
the camera apparatus comprises at least one of the following: a fisheye camera, a wide-angle camera, or a wide-field-of-view camera; and
the radar echo signal comprises an echo signal of at least one of the following radars: an ultrasonic radar, a laser radar, and a millimeter-wave radar.
3. The method according to claim 1 , wherein
the camera apparatus is disposed on at least one of the following positions: on the vehicle, on a road, on an apparatus around a road; and
the radar is disposed on at least one of the following positions: on the vehicle, on a road, or on an apparatus around a road.
4. The method according to claim 1 , wherein
the neural network comprises an encoder and a decoder, the encoder is configured to perform dimension reduction on data by using a pooling layer, and the decoder is configured to perform dimension increasing on data by using a deconvolutional layer.
5. The method according to claim 1 , wherein the processing, by using the neural network, image data obtained by the camera apparatus, to obtain the first probability distribution of the obstacle comprises:
performing, by using the neural network, semantic segmentation processing on the image data obtained by the camera apparatus, to obtain a type of the obstacle, wherein the type of the obstacle is one of a high obstacle type and a low obstacle type, and determining the first probability distribution of the obstacle based on the type of the obstacle.
6. The method according to claim 1 , wherein
input of the neural network comprises a plurality of frames of images that are adjacent or spaced in terms of time.
7. The method according to claim 1 , wherein
in a training process of the neural network, data enhancement is implemented by using an obstacle mapping method.
8. The method according to claim 1 , wherein
a higher weight is given to a smaller obstacle in a loss function of the neural network.
9. An autonomous driving assistance system, comprising:
a camera apparatus, wherein the camera apparatus is configured to be capable of obtaining image data;
at least one radar, wherein the radar is configured to be capable of obtaining a radar echo signal; and
at least one processor; and
a memory coupled to the at least one processor and storing programming instructions for execution by the at least one processor, the programming instructions for execution by the at least one processor, the programming instructions instruct the at least one processor to perform the following operations:
processing, using a neural network, image data obtained by the camera apparatus, to obtain a first probability distribution of an obstacle;
obtaining a second probability distribution of the obstacle based on a time of flight and an echo width of a radar echo signal, wherein the echo width is a difference between a second time of flight of the echo signal and a first time of flight of the echo signal, wherein the second time of flight corresponds to a longest echo distance between a radar and the obstacle, and wherein the first time of flight corresponds to a shortest echo distance between the radar and the obstacle;
obtaining, based on the first probability distribution of the obstacle and the second probability distribution of the obstacle, a drivable area of a vehicle represented by a probability, wherein the probability is a probability that the vehicle cannot drive through the area; and either:
a) based on the autonomous driving assistance system being implemented in an autonomous vehicle, planning a driving route for the vehicle based on the obtained drivable area;
or
b) based on the autonomous driving assistance system being implemented in a manually driven vehicle, wherein the drivable area of the vehicle is represented in a form of a probability grid map; and wherein the probability that the vehicle cannot drive through the area is represented in the probability grid map, displaying the probability grid map on a display apparatus of the vehicle.
10. The system according to claim 9 , wherein
the camera apparatus comprises at least one of the following: a fisheye camera, a wide-angle camera, or a wide-field-of-view camera; and
the radar echo signal comprises an echo signal of at least one of the following radars: an ultrasonic radar, a laser radar, and a millimeter-wave radar.
11. The system according to claim 9 , wherein
the neural network comprises an encoder and a decoder, the encoder is configured to perform dimension reduction on data by using a pooling layer, and the decoder is configured to perform dimension increasing on data by using a deconvolutional layer.
12. The system according to claim 9 , wherein the programming instructions instruct the at least one processor to perform the following operation:
performing, using the neural network, semantic segmentation processing on the image data obtained by the camera apparatus, to obtain a type of the obstacle, wherein the type of the obstacle is one of a high obstacle type and a low obstacle type, and determining the first probability distribution of the obstacle based on the type of the obstacle.
13. The system according to claim 9 , wherein
input of the neural network comprises a plurality of frames of images that are adjacent or spaced in terms of time, and output of the neural network comprises a type of the obstacle and the first probability distribution of the obstacle.
14. The system according to claim 9 , wherein
in a training process of the neural network, data enhancement is implemented by using an obstacle mapping method.
15. The system according to claim 9 , wherein
a higher weight is given to a smaller obstacle in a loss function of the neural network.
16. The system according to claim 9 , wherein the programming instructions instruct the at least one processor to perform the following operation:
obtaining times of flight of the obstacle around the vehicle by using a plurality of radars, and determining coordinates of a center of the obstacle in a vehicle reference coordinate system based on the times of flight and wave velocities of the radars.
17. A computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium that, when executed by a processor, cause an apparatus to:
processing, by using a neural network, image data obtained by a camera apparatus, to obtain a first probability distribution of an obstacle;
obtaining a second probability distribution of the obstacle based on a time of flight and an echo width of a radar echo signal, wherein the echo width is a difference between a second time of flight of the echo signal and a first time of flight of the echo signal, wherein the second time of flight corresponds to a longest echo distance between a radar and the obstacle, and wherein the first time of flight corresponds to a shortest echo distance between the radar and the obstacle;
obtaining, based on the first probability distribution of the obstacle and the second probability distribution of the obstacle, a drivable area of a vehicle represented by a probability, wherein the probability is a probability that the vehicle cannot drive through the area; and either:
a) wherein the vehicle is an autonomous vehicle, planning a driving route for the vehicle based on the obtained drivable area;
or
b) wherein the vehicle is a manually driven vehicle, wherein the drivable area of the vehicle is represented in a form of a probability grid map; and wherein the probability that the vehicle cannot drive through the area is represented in the probability grid map, displaying the probability grid map on a display of the vehicle.