Systems and methods for multi-camera modeling with neural camera networks
Systems and methods for self-supervised depth estimation using image frames captured from a plurality of cameras mounted on a vehicle, may include: receiving a first image from a camera mounted at a first location on the vehicle, the source image comprising pixels representing a scene of the environment of the vehicle; receiving a reference image from a camera mounted at a second location on the vehicle, the reference image comprising pixels representing a scene of the environment; predicting a depth map for the first image, the depth map comprising predicted depth values for pixels of the first image; warping the first image to a perspective of the camera mounted at the second location on the vehicle to arrive at a warped first image; projecting the warped first image onto the source image; determining a loss based on the projection; and updating the predicted depth values for the first image.
1. A method of self-supervised depth estimation using image frames captured from a plurality of cameras mounted on a vehicle in an environment, comprising:
receiving a first image from a first camera mounted at a first location on the vehicle, the first image comprising pixels representing a scene of the environment of the vehicle;
receiving a second image from a second camera mounted at a second location on the vehicle, the second image comprising pixels representing a scene of the environment of the vehicle;
predicting a depth map for the first image, the depth map comprising predicted depth values for pixels of the first image;
warping the first image to a perspective of the second camera mounted at the second location on the vehicle to arrive at a warped first image;
projecting the warped first image onto the second image;
determining a loss based on the projection; and
updating the predicted depth values for the first image.
2. The method of claim 1 , further comprising reiterating the operations of warping the first image, projecting the warped first image and determining the loss using updated predicted depth values for the first image.
3. The method of claim 1 , wherein projecting is performed using a neural camera model to model intrinsic parameters of the first camera.
4. The method of claim 1 , further comprising predicting a transformation from the first camera to the second camera based on loss calculations between the warped first image and the second image.
5. The method of claim 1 , wherein the second image is an image captured at a time, t+/−1, different from a time, t, at which the first image is captured.
6. The method of claim 5 , wherein a transformation from the first camera location to the second camera location includes movement of the vehicle between times t and t+/−1.
7. The method of claim 1 , wherein projecting the warped first image onto the second image comprises lifting 2D points of the warped first image to 3D points, determining a transformation between the first and second cameras and using the transformation to project the 3D points onto the second image in 2D.
8. The method of claim 7 , wherein the transformation comprises a distance in three dimensions between image sensors of the first and second cameras.
9. A system for self-supervised learning depth estimation using image frames captured from a plurality of cameras mounted on a vehicle in an environment, the system comprising:
a non-transitory memory configured to store instructions;
a processor configured to execute the instructions to perform the operations of:
receiving a first image from a first camera mounted at a first location on the vehicle, the first image comprising pixels representing a scene of the environment of the vehicle;
receiving a second image from a second camera mounted at a second location on the vehicle, the second image comprising pixels representing a scene of the environment of the vehicle;
predicting a depth map for the first image, the depth map comprising predicted depth values for pixels of the first image;
warping the first image to a perspective of the second camera mounted at the second location on the vehicle to arrive at a warped first image;
projecting the warped first image onto the reference image;
determining a loss based on the projection; and
updating the predicted depth values for the first image.
10. The system of claim 9 , wherein the operations further comprise reiterating the operations of warping the first image, projecting the warped first image and determining the loss using updated predicted depth values for the first image.
11. The system of claim 9 , wherein projecting is performed using a neural camera-model to model intrinsic parameters of the first camera.
12. The system of claim 9 , wherein the operations further comprise predicting a transformation from the first camera to the second camera based on loss calculations between the warped first image and the second image.
13. The system of claim 9 , wherein the second image is an image captured at a time, t+/−1, different from a time, t, at which the first image is captured.
14. The system of claim 13 , wherein a transformation from the first camera location to the second camera location includes movement of the vehicle between times t and t+/−1.
15. The system of claim 9 , wherein projecting the warped first image onto the second image comprises lifting 2D points of the warped first image to 3D points, determining a transformation between the first and second cameras and using the transformation to project the 3D points onto the second image in 2D.
16. The system of claim 15 , wherein the transformation comprises a distance in three dimensions between image sensors of the first and second cameras.
17. A system for self-supervised learning depth estimation, the system comprising:
a plurality of cameras mounted on a vehicle and configured to capture images of an environment surrounding the vehicle; wherein a first camera is mounted at a first location on the vehicle to capture a first image comprising pixels representing a scene of the environment of the vehicle; and a second camera is mounted at a second location on the vehicle to capture a second image comprising pixels representing a scene of the environment of the vehicle;
a depth encoder configured to receive the first image and to predict a depth map for the first image, the depth map comprising predicted depth values for pixels of the first image;
a warping module to warp the first image to a perspective of the second camera mounted at the second location on the vehicle to arrive at a warped first image;
a projection module to project the warped first image onto the second image; and
a loss module to determine a loss based on the projection.
18. The system of claim 17 , further comprising a neural camera model configured to model intrinsic parameters of the first camera.
19. The system of claim 17 , wherein the second image is an image captured at a time, t+/−1, different from a time, t, at which the first image is captured.
20. The system of claim 19 , wherein a transformation from the first camera location to the second camera location includes movement of the vehicle between times t and t+/−1.