Vehicle neural network localization
A plurality of temporally successive vehicle sensor images are received as input to a variational autoencoder neural network that outputs an averaged semantic birds-eye view image that includes respective pixels determined by averaging semantic class values of corresponding pixels in respective images in the plurality of temporally successive vehicle sensor images. From a plurality of topological nodes that each specify respective real-world locations, a topological node closest to the vehicle, and a three degree-of-freedom pose for the vehicle relative to the topological node closest to the vehicle, is determined based on the averaged semantic birds-eye view image. A real-world three degree-of-freedom pose for the vehicle is determined by combining the three degree-of-freedom pose for the vehicle relative to the topological node and the real-world location of the topological node closest to the vehicle.
1. A system, comprising a computer including a processor and a memory, the memory storing instructions executable by the processor to:
receive a plurality of temporally successive vehicle sensor images as input to a variational autoencoder neural network that outputs an averaged semantic birds-eye view image that includes respective pixels determined by averaging semantic class values of corresponding pixels in respective images in the plurality of temporally successive vehicle sensor images;
from a plurality of topological nodes that each specify respective real-world locations, determine a topological node closest to a vehicle, and a three degree-of-freedom pose for the vehicle relative to the topological node closest to the vehicle, based on the averaged semantic birds-eye view image; and
determine a real-world three degree-of-freedom pose for the vehicle by combining the three degree-of-freedom pose for the vehicle relative to the topological node and a real-world location of the topological node closest to the vehicle.
2. The system of claim 1 , wherein the instructions further include instructions to generate the averaged semantic birds-eye view image based on rendering a semantic point cloud image of an environment around the vehicle into a two dimensional plane.
3. The system of claim 2 , wherein the instructions further include instructions to generate the semantic point cloud image based on combining a semantic image that includes regions labeled by region type and a stereo point cloud image that includes regions labeled by region distance relative to the vehicle.
4. The system of claim 3 , wherein the instructions further include instructions to generate the stereo point cloud image based on a pair of stereo images acquired by a sensor in the vehicle.
5. The system of claim 3 , wherein the instructions further include instructions to generate the semantic image based on a single stereo image acquired by a sensor in the vehicle.
6. The system of claim 3 , wherein the region types include roadway, sidewalk, vehicle, building, and foliage.
7. The system of claim 1 , wherein the instructions further include instructions to determine the topological nodes by acquiring point cloud images with a stereo camera and determining locations of the point cloud images in real-world coordinates with visual odometry.
8. The system of claim 1 , wherein the real-world three degree-of-freedom pose for the vehicle is determined in coordinates based on orthogonal x and y axes and a yaw rotation about a z axis orthogonal to the x and y axes.
9. The system of claim 1 , wherein the instructions further include instructions to train the variational autoencoder neural network to output the averaged semantic birds-eye view image using a plurality of modified semantic birds-eye view images.
10. The system of claim 9 , wherein the instructions further include instructions to generate each of the plurality of modified sematic birds-eye view images based on at least one of translating or rotating the semantic birds-eye view image.
11. The system of claim 1 , wherein the variational autoencoder neural network determines the three degree-of-freedom pose for the vehicle relative to the topological node closest to the vehicle by outputting latent variables to fully connected layers.
12. The system of claim 1 , wherein the variational autoencoder neural network determines the topological node closest to the vehicle by inputting latent variables of the averaged semantic birds-eye view to a nearest neighbor classifier trained to determine the topological node closest to the vehicle.
13. A method, comprising:
receiving a plurality of temporally successive vehicle sensor images as input to a variational autoencoder neural network that outputs an averaged semantic birds-eye view image that includes respective pixels determined by averaging semantic class values of corresponding pixels in respective images in the plurality of temporally successive vehicle sensor images;
from a plurality of topological nodes that each specify respective real-world locations, determining a topological node closest to a vehicle, and a three degree-of-freedom pose for the vehicle relative to the topological node closest to the vehicle, based on the averaged semantic birds-eye view image; and
determining a real-world three degree-of-freedom pose for the vehicle by combining the three degree-of-freedom pose for the vehicle relative to the topological node and a real-world location of the topological node closest to the vehicle.
14. The method of claim 13 , further comprising generating the averaged semantic birds-eye view image based on rendering a semantic point cloud image of an environment around the vehicle into a two dimensional plane.
15. The method of claim 14 , further comprising generating the semantic point cloud image based on combining a semantic image that includes regions labeled by region type and a stereo point cloud image that includes regions labeled by region distance relative to the vehicle.
16. The method of claim 13 , further comprising determining the topological nodes by acquiring point cloud images with a stereo camera and determining locations of the point cloud images in real-world coordinates with visual odometry.
17. The method of claim 13 , wherein the real-world three degree-of-freedom pose for the vehicle is determined in coordinates based on orthogonal x and y axes and a yaw rotation about a z axis orthogonal to the x and y axes.
18. The method of claim 13 , further comprising training the variational autoencoder neural network to output the averaged semantic birds-eye view image using a plurality of modified semantic birds-eye view images.
19. The method of claim 13 , wherein the variational autoencoder neural network determines the three degree-of-freedom pose for the vehicle relative to the topological node closest to the vehicle by outputting latent variables to fully connected layers.
20. The method of claim 13 , wherein the variational autoencoder neural network determines the topological node closest to the vehicle by inputting latent variables of the averaged semantic birds-eye view to a nearest neighbor classifier trained to determine the topological node closest to the vehicle.