Systems and methods for depth synthesis with transformer architectures
Systems and methods for enhanced computer vision capabilities, particularly including depth synthesis, which may be applicable to autonomous vehicle operation are described. A vehicle may be equipped with a geometric scene representation (GSR) architecture for synthesizing depth views at arbitrary viewpoints. The GSR architecture synthesizes depth views enable advanced functions, including depth interpolation and depth extrapolation. The GSR architecture implements functions (i.e., depth interpolation, depth extrapolation) that are useful for various computer vision applications for autonomous vehicles, such as predicting depth maps from unseen locations. For example, a vehicle includes a processor device synthesizing depth views at multiple viewpoints, where the multiple viewpoints are from image data of a surrounding environment for the vehicle. Further, the vehicle can have a controller device that receives depth views from the processor device and performs autonomous operations in response to analysis of the depth views.
1. A vehicle, comprising:
one or more processors; and
memory storing machine-executable instructions in non-transitory memory, which when executed by the one or more processors, cause the vehicle to:
synthesize depth views at multiple viewpoints, wherein synthesizing the depth views at the multiple viewpoints comprises:
encoding images of an environment surrounding the vehicle into image embeddings, wherein the images are obtained from the multiple viewpoints,
encoding intrinsic parameters and relatives poses of one or more cameras into camera embeddings, wherein the one or more cameras captured the images,
projecting the image embeddings and the camera embeddings onto a latent representation for the environment using cross-attention layers of a neural network,
conditioning the latent representation using self-attention layers of the neural network,
generating second camera encodings for arbitrary cameras at arbitrary relative poses, and
querying the conditioned latent representation using the second camera embeddings to synthesize the depth views; and
perform one or more autonomous operations in response to analysis of the synthesized depth views.
2. The vehicle of claim 1 , wherein the one or more processors comprise a geometric scene representation (GSR) component.
3. The vehicle of claim 2 , wherein synthesizing depth views comprises depth estimations, depth interpolations, and depth extrapolations.
4. The vehicle of claim 3 , wherein the depth extrapolations comprise completed unseen portions of a scene including the surrounding environment for the vehicle.
5. The vehicle of claim 4 , wherein the depth extrapolations comprise generated dense depth maps from the multiple viewpoints.
6. The vehicle of claim 5 , wherein performing the one or more autonomous operations is in response to the dense depth maps.
7. The vehicle of claim 1 , wherein the one or more processors comprise a computer vision component performing one or more computer visual capabilities for the one or more autonomous operations.
8. The vehicle of claim 7 , wherein the one or more computer visual capabilities comprise object detection.
9. The vehicle of claim 1 , wherein the vehicle comprises an autonomous vehicle.