Vehicle environment modeling with a camera
System and techniques for vehicle environment modeling with a camera are described herein. A time-ordered sequence of images representative of a road surface may be obtained. An image from this sequence is a current image. A data set may then be provided to an artificial neural network (ANN) to produce a three-dimensional structure of a scene. Here, the data set includes a portion of the sequence of images that includes the current image, motion of the sensor from which the images were obtained, and an epipole. The road surface is then modeled using the three-dimensional structure of the scene.
1 . A non-transitory computer readable medium storing instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations comprising:
obtaining a time-ordered sequence of images of a road surface, one image of the time-ordered sequence of images being a current image;
providing the time-ordered sequence of images to a first artificial neural network (ANN) trained using geometric constraints or photogrammetric constraints to produce gamma images corresponding to the time-ordered sequence of images;
aligning the time-ordered sequence of images based on the gamma images to produce an aligned-sequence of images;
providing the aligned-sequence of images to a second ANN trained to determine that an object visible in the current image is moving; and
providing a signal that indicates the object based on the object moving.
2 . The non-transitory computer readable medium of claim 1 , wherein the second ANN is a convolutional neural network (CNN).
3 . The non-transitory computer readable medium of claim 2 , wherein motion of a sensor that captured the time-ordered sequence of images or an epipole representation are provided to the CNN at a bottleneck layer.
4 . The non-transitory computer readable medium of claim 1 , wherein a target identifier for the object is provided to the second ANN.
5 . The non-transitory computer readable medium of claim 4 , wherein the target identifier includes a gradient image indicating a distance from a center of the object as represented in the current image.
6 . The non-transitory computer readable medium of claim 4 , wherein the target identifier includes a size of the object.
7 . The non-transitory computer readable medium of claim 4 , wherein the target identifier includes a mask of pixels corresponding to the object.
8 . The non-transitory computer readable medium of claim 1 , wherein the time-ordered sequence of images is captured by a sensor on a vehicle.
9 . The non-transitory computer readable medium of claim 8 , wherein the sensor is a camera.
10 . The non-transitory computer readable medium of claim 1 , wherein the first ANN is trained using an unsupervised training technique.
11 . The non-transitory computer readable medium of claim 10 , wherein the unsupervised training technique determines error by measuring a difference between predicted gamma values and observed sensor movement.
12 . The non-transitory computer readable medium of claim 1 , wherein the object represents a vehicle, a pedestrian, or an animal.
13 . A method for identifying a moving object by a vehicle, the method comprising:
obtaining a time-ordered sequence of images of a road surface, one image of the time-ordered sequence of images being a current image;
providing the time-ordered sequence of images to a first artificial neural network (ANN) trained using geometric constraints or photogrammetric constraints to produce gamma images corresponding to the time-ordered sequence of images;
aligning the time-ordered sequence of images based on the gamma images to produce an aligned-sequence of images;
providing the aligned-sequence of images to a second ANN trained to determine that an object visible in the current image is moving; and
providing a signal that indicates the object based on the object moving.
14 . The method of claim 13 , wherein the second ANN is a convolutional neural network (CNN).
15 . The method of claim 14 , wherein motion of a sensor that captured the time-ordered sequence of images or an epipole representation are provided to the CNN at a bottleneck layer.
16 . The method of claim 13 , wherein a target identifier for the object is provided to the second ANN.
17 . The method of claim 16 , wherein the target identifier includes a gradient image indicating a distance from a center of the object as represented in the current image.
18 . The method of claim 16 , wherein the target identifier includes a size of the object.
19 . The method of claim 16 , wherein the target identifier includes a mask of pixels corresponding to the object.
20 . The method of claim 13 , wherein the time-ordered sequence of images is captured by a sensor on the vehicle.
21 . The method of claim 20 , wherein the sensor is a camera.
22 . The method of claim 13 , wherein the first ANN is trained using an unsupervised training technique.
23 . The method of claim 22 , wherein the unsupervised training technique determines error by measuring a difference between predicted gamma values and observed sensor movement.
24 . The method of claim 13 , wherein the object represents a vehicle, a pedestrian, or an animal.