Vehicle neural network perception and localization
A computer, including a processor and a memory, the memory including instructions to be executed by the processor to determine a plurality of topological nodes wherein each topological node includes a location in real-world coordinates and a three-dimensional point cloud image of the environment at the location of the topological node and process an image acquired by a sensor included in a vehicle using a variational auto-encoder neural network trained to output a semantic point cloud image, wherein the semantic point cloud image includes regions labeled by region type and region distance relative to the vehicle. The instructions include further instructions to determine a topological node closest to the vehicle and a six degree-of-freedom pose for the vehicle relative to the topological node closest to the vehicle based on the semantic point cloud data, determine a real-world six degree-of-freedom pose for the vehicle by combining the six degree-of-freedom for the vehicle relative to the topological node and the location in real-world coordinates of the topological node closest to the vehicle and determine a location and size of a three-dimensional object in the semantic point cloud image based on three-dimensional background subtraction using the three-dimensional point cloud image included in the topological node closest to the vehicle. The instructions include further instructions to improve the three-dimensional point cloud image included in the topological node based on the semantic point cloud image and the real-world six degree-of-freedom pose for the vehicle.
1. A computer, comprising:
a processor; and
a memory, the memory including instructions executable by the processor to:
input an image acquired by a sensor in a vehicle to a variational auto-encoder neural network that outputs a semantic point cloud image that includes regions labeled by region type and region distance relative to the vehicle;
from a plurality of topological nodes, determine a topological node closest to the vehicle, and a six degree-of-freedom pose for the vehicle relative to the topological node closest to the vehicle, based on the semantic point cloud image, wherein each topological node includes a location in real-world coordinates and a three-dimensional point cloud image of an environment including the location of the topological node;
determine a real-world six degree-of-freedom pose for the vehicle by combining the six degree-of-freedom pose for the vehicle relative to the topological node closest to the vehicle and the location in real-world coordinates of the topological node closest to the vehicle; and
determine a location and size of a three-dimensional object in the semantic point cloud image based on three-dimensional background subtraction using the three-dimensional point cloud image included in the topological node closest to the vehicle.
2. The computer of claim 1 , the instructions including further instructions to determine the real-world six degree-of-freedom pose for the vehicle based on a single monocular image acquired by the sensor in the vehicle without including an inertial measurement unit in the vehicle.
3. The computer of claim 1 , the instructions including further instructions to determine the semantic point cloud image based on a single monocular image acquired by the sensor in the vehicle without including a stereo camera or lidar sensor in the vehicle.
4. The computer of claim 1 , the instructions including further instructions to determine the plurality of topological nodes by acquiring point cloud images with a stereo camera and determining locations of the point cloud images in real-world coordinates with visual odometry.
5. The computer of claim 1 , wherein the real-world six degree-of-freedom pose for the vehicle is determined in coordinates based on orthogonal x, y, and z axes and roll, pitch, and yaw rotations about the x, y, and z axes, respectively.
6. The computer of claim 1 , the instructions including further instructions to determine the location and size of the three-dimensional object in the semantic point cloud image with a convolutional neural network trained to determine a three-dimensional bounding box that includes the three-dimensional object and a real-world location relative to the vehicle for the three-dimensional object.
7. The computer of claim 6 , wherein three-dimensional background subtraction using the point cloud image is used to determine the three-dimensional bounding box by reducing background clutter caused by labeled regions that occur in both a current semantic point cloud and a semantic point cloud image included in a topological node.
8. The computer of claim 7 , wherein the convolutional neural network includes convolutional layers that output first latent variables to first fully connected neural network layers.
9. The computer of claim 1 , wherein the variational auto-encoder neural network determines the semantic point cloud image by outputting second latent variables to second fully connected neural network layers.
10. The computer of claim 1 , the instructions including further instructions to train the variational auto-encoder neural network to output the semantic point cloud image in a Siamese network configuration wherein the variational auto-encoder neural network is trained in parallel with a second variational auto-encoder network trained using a plurality of three-dimensional point cloud images acquired while traveling along a route that includes the plurality of topological nodes.
11. The computer of claim 10 , wherein region types included in the semantic point cloud image include roadway, sidewalk, vehicle, building, and foliage based on the plurality of three-dimensional point cloud images.
12. A method, comprising:
inputting an image acquired by a sensor in a vehicle to a variational auto-encoder neural network that outputs a semantic point cloud image that includes regions labeled by region type and region distance relative to the vehicle;
from a plurality of topological nodes, determining a topological node closest to the vehicle, and a six degree-of-freedom pose for the vehicle relative to the topological node closest to the vehicle, based on the semantic point cloud image, wherein each topological node includes a location in real-world coordinates and a three-dimensional point cloud image of an environment including the location of the topological node;
determining a real-world six degree-of-freedom pose for the vehicle by combining the six degree-of-freedom pose for the vehicle relative to the topological node closest to the vehicle and the location in real-world coordinates of the topological node closest to the vehicle; and
determine a location and size of a three-dimensional object in the semantic point cloud image based on three-dimensional background subtraction using the three-dimensional point cloud image included in the topological node closest to the vehicle.
13. The method of claim 12 , further comprising determining the real-world six degree-of-freedom pose for the vehicle based on a single monocular image acquired by the sensor in the vehicle without including an inertial measurement unit in the vehicle.
14. The method of claim 12 , further comprising determining the semantic point cloud image based on a single monocular image acquired by the sensor in the vehicle without including a stereo camera or lidar sensor in the vehicle.
15. The method of claim 12 , further comprising determining the plurality of topological nodes by acquiring point cloud images with a stereo camera and determining locations of the point cloud images in real-world coordinates with visual odometry.
16. The method of claim 12 , wherein the real-world six degree-of-freedom pose for the vehicle is determined in coordinates based on orthogonal x, y, and z axes and roll, pitch, and yaw rotations about the x, y, and z axes, respectively.
17. The method of claim 12 , the instructions including further instructions to determine the location and size of the three-dimensional object in the semantic point cloud image with a convolutional neural network trained to determine a three-dimensional bounding box that includes the three-dimensional object and a real-world location relative to the vehicle for the three-dimensional object.
18. The method of claim 17 , wherein three-dimensional background subtraction using the point cloud image is used to determine the three-dimensional bounding box by reducing background clutter caused by labeled regions that occur in both a current semantic point cloud and a semantic point cloud image included in a topological node.
19. The method of claim 18 , wherein the convolutional neural network includes convolutional layers that output first latent variables to first fully connected neural network layers.
20. The method of claim 12 , wherein the variational auto-encoder neural network determines the semantic point cloud image by outputting second latent variables to second fully connected neural network layers.