Vehicle environment modeling with a camera

1. A device for modeling an environment, the device comprising:

a hardware sensor interface to obtain a sequence of unrectified images representative of a road environment, the sequence of unrectified images including a first unrectified image, a previous unrectified image, and a previous-previous unrectified image; and

processing circuitry to:

calculate a first best-fit homography from a pinhole homography and rectification parameters for a lens used to capture the first unrectified image;

warp the previous unrectified image to the first unrectified image using the first best-fit homography, to obtain a warped previous unrectified image;

calculate a second best-fit homography from the pinhole homography and rectification parameters for the lens used to capture the first unrectified image;

warp the previous-previous unrectified image to the first unrectified image using the second best-fit homography, to obtain a warped previous-previous unrectified image;

provide the first unrectified image, the warped previous unrectified image, and the warped previous-previous unrectified image to an artificial neural network (ANN) to produce a three-dimensional structure of a scene; and

apply the first or second best-fit homography to the three-dimensional structure of the scene to create a model of the road environment.

2. The device of claim 1 , wherein the sequence of unrectified images is time-ordered.

3. The device of claim 1 , wherein the road environment includes a road surface.

4. The device of claim 1 , wherein the road environment includes a lane marker, a sidewalk, or a road control object.

5. The device of claim 1 , wherein to calculate the first best-fit homography, the processing circuitry is to:

select a first set of image coordinates in the previous unrectified image;

rectify the first set of image coordinates to obtain a first set of rectified image coordinates;

apply the pinhole homography to the first set of rectified image coordinates to identify corresponding image coordinates in the first unrectified image;

distort the corresponding image coordinates in the first unrectified image to obtain estimated positions of distorted image coordinates; and

use a modeling technique to determine the first best-fit homography between the first set of image coordinates and the estimated positions of distorted image coordinates.

6. The device of claim 5 , wherein the first set of image coordinates are pixels.

7. The device of claim 5 , wherein the modeling technique is a linear regression technique.

8. The device of claim 5 , wherein the first set of image coordinates includes 50-200 image coordinates.

9. The device of claim 5 , wherein to distort the corresponding image coordinates, the processing circuitry is to use a lens distortion model for a lens used to capture the first unrectified image, the previous unrectified image, and the previous-previous unrectified image.

10. The device of claim 1 , wherein the model of the road environment includes height above a ground plane and depth from an image capture device used to capture the sequence of unrectified images.

11. The device of claim 1 , wherein the ANN is trained with an unsupervised training technique that include operations to:

provide the first unrectified image, the warped previous unrectified image, and the warped previous-previous unrectified image to the artificial neural network (ANN) to produce a gamma image;

rectify the gamma image to produce a rectified gamma image; and

calculate a loss function using the rectified gamma image.

12. The device of claim 11 , wherein to rectify the gamma image, the processing circuitry is to use a lens distortion model.

13. The device of claim 11 , wherein to calculate the loss function, the processing circuitry is to calculate an error between a model of a future image and a future image, the model of the future image produced using the first homography applied to the first unrectified image.

14. A method for modeling an environment, comprising:

obtaining a sequence of unrectified images representative of a road environment, the sequence of unrectified images including a first unrectified image, a previous unrectified image, and a previous-previous unrectified image;

calculating a first best-fit homography from a pinhole homography and rectification parameters for a lens used to capture the first unrectified image;

warping the previous unrectified image to the first unrectified image using the first best-fit homography, to obtain a warped previous unrectified image;

calculating a second best-fit homography from the pinhole homography and rectification parameters for the lens used to capture the first unrectified image;

warping the previous-previous unrectified image to the first unrectified image using the second best-fit homography, to obtain a warped previous-previous unrectified image;

providing the first unrectified image, the warped previous unrectified image, and the warped previous-previous unrectified image to an artificial neural network (ANN) to produce a three-dimensional structure of a scene; and

applying the first or second best-fit homography to the three-dimensional structure of the scene to create a model of the road environment.

15. The method of claim 14 , wherein calculating the first best-fit homography comprises:

selecting a first set of image coordinates in the previous unrectified image;

rectifying the first set of image coordinates to obtain a first set of rectified image coordinates;

applying the pinhole homography to the first set of rectified image coordinates to identify corresponding image coordinates in the first unrectified image;

distorting the corresponding image coordinates in the first unrectified image to obtain estimated positions of distorted image coordinates; and

using a modeling technique to determine the first best-fit homography between the first set of image coordinates and the estimated positions of distorted image coordinates.

16. The method of claim 15 , wherein the modeling technique is a linear regression technique.

17. The method of claim 15 , wherein distorting the corresponding image coordinates comprises using a lens distortion model for a lens used to capture the first unrectified image, the previous unrectified image, and the previous-previous unrectified image.

18. The method of claim 14 , wherein the model of the road environment includes height above a ground plane and depth from an image capture device used to capture the sequence of unrectified images.

19. The method of claim 14 , wherein the ANN is trained with an unsupervised training technique that include operations to:

provide the first unrectified image, the warped previous unrectified image, and the warped previous-previous unrectified image to the artificial neural network (ANN) to produce a gamma image;

rectify the gamma image to produce a rectified gamma image; and

calculate a loss function using the rectified gamma image.

20. The method of claim 19 , wherein to rectify the gamma image, the unsupervised training technique is to use a lens distortion model.

21. The method of claim 19 , wherein calculating the loss function comprises calculating an error between a model of a future image and a future image, the model of the future image produced using the first best-fit homography applied to the first unrectified image.

22. At least one non-transitory machine-readable medium including instructions for modeling an environment, the instructions, when executed by processing circuitry, cause the processing circuitry to perform operations comprising:

obtaining a sequence of unrectified images representative of a road environment, the sequence of unrectified images including a first unrectified image, a previous unrectified image, and a previous-previous unrectified image;

calculating a first best-fit homography from a pinhole homography and rectification parameters for a lens used to capture the first unrectified image;

warping the previous unrectified image to the first unrectified image using the first best-fit homography, to obtain a warped previous unrectified image;

calculating a second best-fit homography from the pinhole homography and rectification parameters for the lens used to capture the first unrectified image;

warping the previous-previous unrectified image to the first unrectified image using the second best-fit homography, to obtain a warped previous-previous unrectified image;

providing the first unrectified image, the warped previous unrectified image, and the warped previous-previous unrectified image to an artificial neural network (ANN) to produce a three-dimensional structure of a scene; and

applying the first or second best-fit homography to the three-dimensional structure of the scene to create a model of the road environment.

23. The machine-readable medium of claim 22 , wherein the road environment includes a road surface, a lane marker, a sidewalk, or a road control object.

24. The machine-readable medium of claim 22 , wherein the operations to calculate the first best-fit homography, comprise operations to:

select a first set of image coordinates in the previous unrectified image;

rectify the first set of image coordinates to obtain a first set of rectified image coordinates;

apply the pinhole homography to the first set of rectified image coordinates to identify corresponding image coordinates in the first unrectified image;

distort the corresponding image coordinates in the first unrectified image to obtain estimated positions of distorted image coordinates; and

use a modeling technique to determine the first best-fit homography between the first set of image coordinates and the estimated positions of distorted image coordinates.

25. The machine-readable medium of claim 24 , wherein the first set of image coordinates are pixels.

26. The machine-readable medium of claim 24 , wherein the modeling technique is a linear regression technique.

27. The machine-readable medium of claim 24 , wherein the first set of image coordinates includes 50-200 image coordinates.

28. The machine-readable medium of claim 24 , wherein the operations to distort the corresponding image coordinates comprise operations to use a lens distortion model for a lens used to capture the first unrectified image, the previous unrectified image, and the previous-previous unrectified image.

29. The machine-readable medium of claim 22 , wherein the model of the road environment includes height above a ground plane and depth from an image capture device used to capture the sequence of unrectified images.

30. The machine-readable medium of claim 22 , wherein the ANN is trained with an unsupervised training technique that include operations to:

provide the first unrectified image, the warped previous unrectified image, and the warped previous-previous unrectified image to the artificial neural network (ANN) to produce a gamma image;

rectify the gamma image to produce a rectified gamma image; and

calculate a loss function using the rectified gamma image.