Multi-network-based path generation for vehicle parking
View Patent ↗Systems and methods of deep neural network based parking assistance is provided. A system can receive data sensed by one or more sensors mounted on a vehicle located at a parking zone. The system generates, from a first neural network, a digital map based on the data sensed by the one or more sensors. The system generates, from a second neural network, a first path based on the three-dimensional dynamic map. The system receives vehicle dynamics information from a second one or more sensors located on the vehicle. The system generates, with a third neural network, a second path to park the vehicle based on the first path, vehicle dynamics information and at least one historical path stored in vehicle memory. The system provides commands to control the vehicle to follow the second path to park the vehicle in the parking zone.
1. A system of network-based path generation for a vehicle controller, comprising:
a data processing system comprising one or more processors and memory, a mapping engine component, a path generation component, and a controller component, the data processing system to:
receive data sensed by a first one or more sensors mounted on a vehicle located at a parking zone;
generate, from a first neural network tuned to process sensor input, a digital map data structure based on the data sensed by the first one or more sensors;
generate, from a second neural network tuned to process digital map data structures, a first path based on the digital map data structure;
receive vehicle dynamics information from a second one or more sensors located on the vehicle;
generate, with a third neural network tuned to process vehicle dynamics and paths, a second path to park the vehicle based on the first path, vehicle dynamics information and at least one historical path stored in vehicle memory; and
provide commands to control the vehicle to follow the second path to park the vehicle in the parking zone.
2. The system of claim 1 , comprising the data processing system to:
detect, from the first neural network and based on the data sensed by the first one or more sensors, objects located at the parking zone;
perform, from the first neural network and based on the data sensed by the first one or more sensors, scene segmentation;
determine, from the first neural network and based on the data sensed by the first one or more sensors, depth information for the parking zone; and
identify, from the first neural network and based on the data sensed by the first one or more sensors, one or more parking lines at the parking zone; and
construct the digital map data structure based on the detected objects located at the parking zone, the scene segmentation, the depth information for the parking zone, and the one or more parking lines at the parking zone.
3. The system of claim 1 , comprising the data processing system to:
establish the parking zone using location information from a satellite based global positioning system.
4. The system of claim 1 , comprising the data processing system to:
generate, from the first neural network and based on the data sensed by the first one or more sensors, the digital map data structure comprising a three-dimensional dynamic map; and
display the three-dimensional dynamic map via a display device coupled to the vehicle.
5. The system of claim 1 , comprising the data processing system to:
generate the digital map data structure, the first path, and the second path in an absence of location information from a satellite based global position system.
6. The system of claim 1 , comprising the data processing system to:
detect a second vehicle located in the parking zone;
establish a secure communication channel with the second vehicle;
request, from the second vehicle via the secure communication channel, map information for the parking zone; and
receive, from the second vehicle responsive to the request, map information for the parking zone stored in memory of the second vehicle.
7. The system of claim 1 , comprising the data processing system to:
prior to generation of the first path, request, from a second vehicle located in the parking zone, map information for the parking zone;
receive, responsive to the request, the map information from the second vehicle; and
input the map information into the second neural network to generate the first path.
8. The system of claim 1 , comprising the data processing system to:
determine, from a fourth neural network, the commands to control the vehicle to follow the second path.
9. The system of claim 1 , comprising:
the data processing system to receive the data from the first one or more sensors comprising at least one of a radar, light detection and ranging sensor (“LIDAR”), ultrasonic transducer, camera, and infrared sensor.
10. The system of claim 1 , wherein the first neural network comprises a plurality of sub-neural networks, the system comprising the data processing system to:
tune, based on historical data, the plurality of sub-neural networks of the first neural network to generate the digital map data structures;
generate, from a first sub-neural network of the plurality of sub-neural networks, object detection information;
generate, from a second sub-neural network of the plurality of sub-neural networks, scene segmentation information;
generate, from a third sub-neural network of the plurality of sub-neural networks, depth information; and
generate, from a fourth sub-neural network of the plurality of sub-neural networks, parking line information.
11. The system of claim 10 , comprising:
construct the digital map data structure based on the object detection information, the scene segmentation information, the depth information, and the parking line information.
12. The system of claim 1 , comprising:
the data processing system to tune, based on historical data, the second neural network to generate the paths, the paths comprising paths for parking vehicles.
13. The system of claim 1 , comprising:
the data processing system to tune, based on historical data, the third neural network to generate second paths for parking vehicles based on vehicle dynamics and historical paths.
14. The system of claim 1 , comprising the data processing system to:
determine that the data from the first one or more sensors is insufficient to generate the digital map data structure for input into the second neural network to generate the first path;
generate commands to control the vehicle to traverse a preliminary path within the parking zone to collect additional data about the parking zone from the first one or more sensors; and
input the additional data to the first neural network to generate the first path.
15. The system of claim 1 , wherein at least one of the first neural network, the second neural network and the third neural network comprises a convolution neural network.
16. A method of network-based path generation for a vehicle controller, comprising:
receiving, by a data processing system comprising one or more processors and memory, data sensed by a first one or more sensors mounted on a vehicle located at a parking zone;
generating, by the data processing system from a first neural network, a digital map data structure based on the data sensed by the first one or more sensors;
generating, by the data processing system from a second neural network, a first path based on the digital map data structure;
receiving, by the data processing system, vehicle dynamics information from a second one or more sensors located on the vehicle;
generating, by the data processing system with a third neural network, a second path to park the vehicle based on the first path, vehicle dynamics information and at least one historical path stored in vehicle memory; and
providing, by the data processing system, commands to control the vehicle to follow the second path to park the vehicle in the parking zone.
17. The method of claim 16 , comprising:
detecting, from the first neural network and based on the data sensed by the first one or more sensors, objects located at the parking zone;
performing, from the first neural network and based on the data sensed by the first one or more sensors, scene segmentation;
determining, from the first neural network and based on the data sensed by the first one or more sensors, depth information for the parking zone; and
identifying, from the first neural network and based on the data sensed by the first one or more sensors, one or more parking lines at the parking zone; and
constructing the digital map data structure based on the detected objects located at the parking zone, the scene segmentation, the depth information for the parking zone, and the one or more parking lines at the parking zone.
18. The method of claim 16 , comprising:
establishing the parking zone using location information from a satellite based global positioning system.
19. The method of claim 16 , comprising:
generating, from the first neural network and based on the data sensed by the first one or more sensors, the digital map data structure comprising a three-dimensional dynamic map; and
displaying the three-dimensional dynamic map via a display device coupled to the vehicle.
20. The method of claim 16 , comprising:
generating the digital map data structure, the first path, and the second path in an absence of location information from a satellite based global position system.