Deep learning-based camera calibration

1. A computer-implemented method, comprising:

receiving, using at least one processor, a first image captured by a camera and a second image captured by the camera;

processing, using the at least one processor, the first image using a first neural network to determine a depth of at least a portion of the first image;

processing, using the at least one processor, the first image and the second image using a second neural network to determine a transformation between a pose of the camera for the first image and a pose of the camera for the second image;

generating, using the at least one processor, a projection image based on the depth of at least the portion of the first image, the transformation between the pose of the camera for the first image and the pose of the camera for the second image, and intrinsic parameters of the camera;

comparing, using the at least one processor, the second image and the projection image to determine a reprojection error; and

adjusting, using the at least one processor, at least one of the intrinsic parameters of the camera based on the reprojection error.

2. The method of claim 1 , wherein receiving the first image captured by the camera and the second image captured by the camera comprises:

receiving the first image while the camera is positioned at a first location; and

receiving the second image while the camera is positioned at a second location different from the first location,

wherein a distance between the first location and the second location satisfies a threshold associated with calibration of the camera.

3. The method of claim 1 , wherein generating the projection image comprises:

applying the transformation to the depth of at least the portion of the first image to predict a depth of at least a portion of the second image; and

processing the predicted depth of at least the portion of the second image with the intrinsic parameters of the camera to generate the projection image.

4. The method of claim 1 , wherein comparing the second image and the projection image to determine the reprojection error comprises:

determining a distance between each point of the second image and a corresponding point of the projection image; and

combining the distance between each point of the second image and the corresponding point of the projection image to determine the reprojection error.

5. The method of claim 1 , wherein adjusting at least one of the intrinsic parameters of the camera comprises applying an optimization algorithm to a cost function that relates the intrinsic parameters to the reprojection error,

wherein the optimization algorithm is configured to minimize the cost function by adjusting at least one of the intrinsic parameters.

6. The method of claim 1 , further comprising adjusting at least one parameter of the first neural network or the second neural network based on the reprojection error.

7. The method of claim 6 , wherein adjusting at least one parameter of the first neural network or the second neural network comprises applying an optimization algorithm to a cost function that relates the at least one parameter of the first neural network or the second neural network to the reprojection error,

wherein the optimization algorithm is configured to minimize the cost function by adjusting the at least one parameter of the first neural network or the second neural network.

8. The method of claim 1 , wherein the first neural network or the second neural network include a convolutional neural network.

9. The method of claim 1 , wherein the intrinsic parameters of the camera include at least one of a focal length, a pixel size, a principal point, a skew coefficient, or a distortion coefficient.

10. The method of claim 1 , further comprising:

determining that the camera is calibrated when the reprojection error satisfies a predetermined threshold.

11. A vehicle, comprising:

a camera;

at least one processor; and

at least one non-transitory computer-readable storage media storing instructions which, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

receiving a first image captured by a camera and a second image captured by the camera;

processing the first image using a first neural network to determine a depth of at least a portion of the first image;

processing the first image and the second image using a second neural network to determine a transformation between a pose of the camera for the first image and a pose of the camera for the second image;

generating a projection image based on the depth of at least the portion of the first image, the transformation between the pose of the camera for the first image and the pose of the camera for the second image, and intrinsic parameters of the camera;

comparing the second image and the projection image to determine a reprojection error; and

adjusting at least one of the intrinsic parameters of the camera based on the reprojection error.

12. The vehicle of claim 11 , wherein receiving the first image captured by the camera and the second image captured by the camera comprises:

receiving the first image while the camera is positioned at a first location; and

receiving the second image while the camera is positioned at a second location different from the first location,

wherein a distance between the first location and the second location satisfies a threshold associated with calibration of the camera.

13. The vehicle of claim 11 , wherein generating the projection image comprises:

applying the transformation to the depth of at least the portion of the first image to predict a depth of at least a portion of the second image; and

processing the predicted depth of at least the portion of the second image with the intrinsic parameters of the camera to generate the projection image.

14. The vehicle of claim 11 , wherein comparing the second image and the projection image to determine the reprojection error comprises:

determining a distance between each point of the second image and a corresponding point of the projection image; and

combining the distance between each point of the second image and the corresponding point of the projection image to determine the reprojection error.

15. The vehicle of claim 11 , wherein adjusting at least one of the intrinsic parameters of the camera comprises applying an optimization algorithm to a cost function that relates the intrinsic parameters to the reprojection error,

wherein the optimization algorithm is configured to minimize the cost function by adjusting at least one of the intrinsic parameters.

16. The vehicle of claim 11 , wherein the instructions further cause the at least one processor to perform operations comprising adjusting at least one parameter of the first neural network or the second neural network based on the reprojection error.

17. The vehicle of claim 16 , wherein adjusting at least one parameter of the first neural network or the second neural network comprises applying an optimization algorithm to a cost function that relates the at least one parameter of the first neural network or the second neural network to the reprojection error,

wherein the optimization algorithm is configured to minimize the cost function by adjusting the at least one parameter of the first neural network or the second neural network.

18. The vehicle of claim 11 , wherein the first neural network or the second neural network include a convolutional neural network.

19. The vehicle of claim 11 , wherein the intrinsic parameters of the camera include at least one of a focal length, a pixel size, a principal point, a skew coefficient, or a distortion coefficient.

20. The vehicle of claim 11 , wherein the instructions further cause the at least one processor to perform operations comprising determining that the camera is calibrated when the reprojection error satisfies a predetermined threshold.

21. At least one non-transitory computer-readable storage media comprising instructions which, when executed by at least one processor, cause the at least one processor to perform operations comprising:

receiving a first image captured by a camera and a second image captured by the camera;

processing the first image using a first neural network to determine a depth of at least a portion of the first image;

processing the first image and the second image using a second neural network to determine a transformation between a pose of the camera for the first image and a pose of the camera for the second image;

generating a projection image based on the depth of at least the portion of the first image, the transformation between the pose of the camera for the first image and the pose of the camera for the second image, and intrinsic parameters of the camera;

comparing the second image and the projection image to determine a reprojection error; and

adjusting at least one of the intrinsic parameters of the camera based on the reprojection error.

22. The at least one non-transitory computer-readable storage medium of claim 21 , wherein receiving the first image captured by the camera and the second image captured by the camera comprises:

receiving the first image while the camera is positioned at a first location; and

receiving the second image while the camera is positioned at a second location different from the first location,

wherein a distance between the first location and the second location satisfies a threshold associated with calibration of the camera.

23. The at least one non-transitory computer-readable storage medium of claim 21 , wherein generating the projection image comprises:

applying the transformation to the depth of at least the portion of the first image to predict a depth of at least a portion of the second image; and

processing the predicted depth of at least the portion of the second image with the intrinsic parameters of the camera to generate the projection image.

24. The at least one non-transitory computer-readable storage medium of claim 21 , wherein comparing the second image and the projection image to determine the reprojection error comprises:

determining a distance between each point of the second image and a corresponding point of the projection image; and

combining the distance between each point of the second image and the corresponding point of the projection image to determine the reprojection error.

25. The at least one non-transitory computer-readable storage medium of claim 21 , wherein adjusting at least one of the intrinsic parameters of the camera comprises applying an optimization algorithm to a cost function that relates the intrinsic parameters to the reprojection error,

wherein the optimization algorithm is configured to minimize the cost function by adjusting at least one of the intrinsic parameters.

26. The at least one non-transitory computer-readable storage medium of claim 21 , wherein the instructions further cause the at least one processor to perform operations comprising adjusting at least one parameter of the first neural network or the second neural network based on the reprojection error.

27. The at least one non-transitory computer-readable storage medium of claim 26 , wherein adjusting at least one parameter of the first neural network or the second neural network comprises applying an optimization algorithm to a cost function that relates the at least one parameter of the first neural network or the second neural network to the reprojection error,

wherein the optimization algorithm is configured to minimize the cost function by adjusting the at least one parameter of the first neural network or the second neural network.

28. The at least one non-transitory computer-readable storage medium of claim 21 , wherein the parameters of the camera include at least one of a focal length, a pixel size, a principal point, a skew coefficient, or a distortion coefficient.

29. The at least one non-transitory computer-readable storage medium of claim 21 , comprising instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising determining that the camera is calibrated when the reprojection error satisfies a predetermined threshold.

30. A camera, comprising:

at least one processor; and

at least one non-transitory computer-readable storage media storing instructions which, when executed by the at least one process, cause the at least one processor to perform operations comprising:

receiving a first image captured by the camera and a second image captured by a camera;

processing the first image using a first neural network to determine a depth of at least a portion of the first image;

processing the first image and the second image using a second neural network to determine a transformation between a pose of the camera for the first image and a pose of the camera for the second image;

generating a projection image based on the depth of at least the portion of the first image, the transformation between the pose of the camera for the first image and the pose of the camera for the second image, and intrinsic parameters of the camera;

comparing the second image and the projection image to determine a reprojection error; and

adjusting at least one of the intrinsic parameters of the camera based on the reprojection error.