Vehicle environment modeling with cameras
Various systems and methods for modeling a scene. A device for modeling a scene includes a hardware interface to obtain a time-ordered sequence of images representative of a scene, the time-ordered sequence including a plurality of images, one of the sequence of images being a current image, the scene captured by a monocular imaging system; and processing circuitry to: provide a data set to an artificial neural network (ANN) to produce a three-dimensional structure of the scene, the data set including: a portion of the sequence of images, the portion of the sequence of images including the current image; and motion of a sensor that captured the sequence of images; and model the scene using the three-dimensional structure of the scene, wherein the three-dimensional structure is determined for both moving and fixed objects in the scene.
1. A device for modeling a scene, the device comprising:
a hardware interface to obtain a time-ordered sequence of images representative of a scene, the time-ordered sequence including a plurality of images, one of the sequence of images being a current image, the scene captured by a monocular imaging system; and
processing circuitry to:
provide a data set to an artificial neural network (ANN) to produce a three-dimensional structure of the scene, the data set including:
a portion of the sequence of images, the portion of the sequence of images including the current image; and
motion of a sensor that captured the sequence of images, wherein the motion of the sensor is provided as a constant value image; and
model the scene using the three-dimensional structure of the scene, wherein the three-dimensional structure is determined for both moving and fixed objects in the scene.
2. The device of claim 1 , wherein an epipole is derived from the motion of the sensor that captured the sequence of images, and wherein the epipole is provided as a gradient image with a same dimensionality as the current image, and values of pixels in the gradient image represent a distance from the epipole of pixels in the current image.
3. The device of claim 2 , wherein the gradient image represents only horizontal distances from the epipole, and wherein a second gradient image is provided to the ANN to represent vertical distances from the epipole.
4. The device of claim 1 , wherein an epipole is derived from the motion of the sensor that captured the sequence of images, and wherein the epipole is provided as a gradient image to a layer of the ANN, and wherein a dimensionality of the epipole matches the layer of the ANN.
5. The device of claim 1 , wherein the constant value image has a same dimensionality as the current image.
6. The device of claim 5 , wherein the constant value image has a value in each pixel that is a ratio of forward motion of the sensor by a height of the sensor from a plane.
7. The device of claim 1 , wherein the constant value image is provided to a layer of the ANN, and wherein a dimensionality of the constant value image matches the layer of the ANN.
8. The device of claim 1 , wherein the processing circuitry is configured to invoke a second ANN using the three-dimensional structure to determine whether an object is moving or not moving within the scene.
9. The device of claim 1 , wherein the ANN is trained with an unsupervised training technique in which error is determined by measuring a difference between a model of a current image and the current image, the model of the current image produced via a gamma warping of an image captured at the same time as the current image.
10. The device of claim 9 , wherein the model of the current image is based on a plurality of images captured by a stereo imaging system, the stereo imaging system independent from the monocular imaging system.
11. The device of claim 10 , wherein the stereo imaging system includes a first image sensor and a second image sensor, the first and second image sensors having overlapping fields of view with each other.
12. The device of claim 11 , wherein the first image sensor and the second image sensor implement rolling shutters.
13. The device of claim 1 , wherein the monocular imaging system uses a rolling shutter camera.
14. A method for modeling a scene, the method comprising:
obtaining a time-ordered sequence of images representative of a scene, the time-ordered sequence including a plurality of images, one of the sequence of images being a current image, the scene captured by a monocular imaging system;
providing a data set to an artificial neural network (ANN) to produce a three-dimensional structure of the scene, the data set including:
a portion of the sequence of images, the portion of the sequence of images including the current image; and
motion of a sensor that captured the sequence of images, wherein the motion of the sensor is provided as a constant value image; and
modeling the scene using the three-dimensional structure of the scene, wherein the three-dimensional structure is determined for both moving and fixed objects in the scene.
15. The method of claim 14 , wherein an epipole is derived from the motion of the sensor that captured the sequence of images, and wherein the epipole is provided as a gradient image with a same dimensionality as the current image, and values of pixels in the gradient image represent a distance from the epipole of pixels in the current image.
16. The method of claim 15 , wherein the gradient image represents only horizontal distances from the epipole, and wherein a second gradient image is provided to the ANN to represent vertical distances from the epipole.
17. The method of claim 14 , wherein an epipole is derived from the motion of the sensor that captured the sequence of images, and wherein the epipole is provided as a gradient image to a layer of the ANN, and wherein a dimensionality of the epipole matches the layer of the ANN.
18. The method of claim 14 , wherein the constant value image has a same dimensionality as the current image.
19. The method of claim 18 , wherein the constant value image has a value in each pixel that is a ratio of forward motion of the sensor by a height of the sensor from a plane.
20. The method of claim 14 , wherein the constant value image is provided to a layer of the ANN, and wherein a dimensionality of the constant value image matches the layer of the ANN.
21. The method of claim 14 , comprising invoking a second ANN using the three-dimensional structure to determine whether an object is moving or not moving within the scene.
22. The method of claim 14 , wherein the ANN is trained with an unsupervised training technique in which error is determined by measuring a difference between a model of a current image and the current image, the model of the current image produced via a gamma warping of an image captured at the same time as the current image.
23. The method of claim 22 , wherein the model of the current image is based on a plurality of images captured by a stereo imaging system, the stereo imaging system independent from the monocular imaging system.
24. The method of claim 23 , wherein the stereo imaging system includes a first image sensor and a second image sensor, the first and second image sensors having overlapping fields of view with each other.
25. The method of claim 24 , wherein the first image sensor and the second image sensor implement rolling shutters.
26. The method of claim 14 , wherein the monocular imaging system uses a rolling shutter camera.