Systems and methods for obstacle detection using a neural network model, depth maps, and segmentation maps
View Patent ↗Systems, methods, and other embodiments described herein relate to detecting obstacles from unknown objects during automated driving by a vehicle. In one embodiment, a method includes generating, from an image that includes an unknown object, a depth map from a depth estimation component and processed data from a semantic segmentation component in parallel by using a neural network model. The method also includes detecting that the unknown object is an obstacle when the unknown object satisfies criteria using an optical model according to the depth map and a segmentation map. The method also includes determining a height of the obstacle and a distance to the obstacle according to the optical model and the criteria. The method also includes adapting a vehicle plan of the automated driving according to the height.
1. A detection system for automated driving by a vehicle, comprising:
one or more processors;
a memory communicably coupled to the one or more processors and storing:
a detection module including instructions that when executed by the one or more processors cause the one or more processors to:
generate, from an image that includes an unknown object, a depth map from a depth estimation component and processed data from a semantic segmentation component in parallel by using a neural network model;
detect that the unknown object is an obstacle when the unknown object satisfies criteria using an optical model according to the depth map and a segmentation map; and
a determination module including instructions that when executed by the one or more processors cause the one or more processors to:
perceive a height of the obstacle using a predicted distance towards the obstacle according to the optical model and the criteria and using an estimate from the optical model including a pixel gain vertically between ends of the obstacle associated with the segmentation map; and
adapt a vehicle plan of the automated driving according to the height.
2. The detection system of claim 1 , wherein the detection module further comprises instructions to identify the unknown object by using the neural network model to satisfy the criteria according to monocular red, green, and blue (RGB) values of the image.
3. The detection system of claim 2 , wherein the detection module further comprises instructions to generate a grayscale depth map associated with the depth map by using the neural network model on the monocular RGB values, wherein a pixel intensity of the grayscale depth map represents an estimated distance to the unknown object.
4. The detection system of claim 3 , wherein the determination module further comprises instructions to determine the predicted distance and a per-pixel category with color-coding of the unknown object by using the grayscale depth map and the segmentation map.
5. The detection system of claim 1 , wherein the determination module includes instructions to perceive the height of the obstacle further including instructions to perceive the height of the obstacle by using an absolute gain of the pixel gain of the unknown object in an image plane according to the optical model and the height is a real-world height.
6. The detection system of claim 1 , wherein the determination module includes instructions to perceive the height of the obstacle further including instructions to measure the height by using the optical model and the predicted distance according to the depth map.
7. The detection system of claim 1 , wherein the determination module further comprises instructions to adapt a tolerance level of the criteria according to on-road known objects and unknown items in a map database for other vehicles to avoid collisions, wherein the criteria is associated with a relationship between a height and a depth derived from the depth map about the unknown object.
8. The detection system of claim 1 , wherein the determination module further comprises instructions to update a map database on a remote server with parameters of the obstacle when the unknown object is missing, unregistered, or anomalous in the map database for other vehicles to avoid collisions with the unknown object.
9. A non-transitory computer-readable medium for detections during automated driving by a vehicle and including instructions that when executed by one or more processors cause the one or more processors to:
generate, from an image that includes an unknown object, a depth map from a depth estimation component and processed data from a semantic segmentation component in parallel by using a neural network model;
detect that the unknown object is an obstacle when the unknown object satisfies criteria using an optical model according to the depth map and a segmentation map;
perceive a height of the obstacle using a predicted distance towards the obstacle according to the optical model and the criteria and using an estimate from the optical model including a pixel gain vertically between ends of the obstacle associated with the segmentation map; and
adapt a vehicle plan of the automated driving according to the height.
10. The non-transitory computer-readable medium of claim 9 further comprising instructions that when executed by one or more processors cause the one or more processors to identify the unknown object by using the neural network model to satisfy the criteria according to monocular red, green, and blue (RGB) values of the image.
11. The non-transitory computer-readable medium of claim 10 further comprising instructions that when executed by one or more processors cause the one or more processors to generate a grayscale depth map associated with the depth map by using the neural network model on the monocular RGB values, wherein a pixel intensity of the grayscale depth map represents an estimated distance to the unknown object.
12. The non-transitory computer-readable medium of claim 11 further comprising instructions that when executed by one or more processors cause the one or more processors to determine the predicted distance and a per-pixel category with color-coding of the unknown object by using the grayscale depth map and the segmentation map.
13. A method for detecting obstacles from unknown objects during automated driving by a vehicle, the method comprising:
generating, from an image that includes an unknown object, a depth map from a depth estimation component and processed data from a semantic segmentation component in parallel by using a neural network model;
detecting that the unknown object is an obstacle when the unknown object satisfies criteria using an optical model according to the depth map and a segmentation map;
perceiving a height of the obstacle using a predicted distance towards the obstacle according to the optical model and the criteria and estimating with the optical model for the height a pixel gain vertically between ends of the obstacle associated with the segmentation map; and
adapting a vehicle plan of the automated driving according to the height.
14. The method of claim 13 , further comprising:
identifying the unknown object by using the neural network model to satisfy the criteria according to monocular red, green, and blue (RGB) values of the image.
15. The method of claim 14 , further comprising:
generating a grayscale depth map associated with the depth map by using the neural network model on the monocular RGB values, wherein a pixel intensity of the grayscale depth map represents an estimated distance to the unknown object.
16. The method of claim 15 , further comprising:
determining the predicted distance and a per-pixel category with color-coding of the unknown object by using the grayscale depth map and the segmentation map.
17. The method of claim 13 , wherein perceiving the height of the obstacle further comprises perceiving the height of the obstacle by using an absolute gain of the pixel gain of the unknown object in an image plane according to the optical model and the height is a real-world height.
18. The method of claim 13 , wherein perceiving the height of the obstacle further comprises measuring the height by using the optical model and the predicted distance according to the depth map.
19. The method of claim 13 , further comprising:
adapting a tolerance level of the criteria according to on-road known objects and unknown items in a map database for other vehicles to avoid collisions, wherein the criteria is associated with a relationship between a height and a depth derived from the depth map about the unknown object.
20. The method of claim 13 , further comprising:
updating a map database on a remote server with parameters of the obstacle when the unknown object is missing, unregistered, or anomalous in the map database for other vehicles to avoid collisions with the unknown object.