Foundation model for zero-shot stereo matching
Systems and methods are disclosed that use a Foundational Stereo Model to generate an output disparity map. The Foundational Stereo Model includes side-tuning adapters (STA) that utilize a vision transformer (ViT) and a convolutional neural network (CNN) to generate feature maps. Specifically, the CNN may be used to adapt the ViT-based monocular depth estimation network for the stereo setup, which synergizes the strengths of both CNN and ViT architectures. In addition, the Foundational Stereo Model includes an attentive hybrid cost filtering (AHCF) that uses two branches that also utilizes the advantages of both a transformer architecture and the CNN architecture. Furthermore, the Foundational Stereo Model may perform iterative refinement of an initial disparity map to obtain the output disparity map based on performing a convolutional gated recurrent unit (GRU) operation.
1 . A computer-implemented method for using a Foundational Stereo Model to generate an output disparity map, comprising:
processing a first stereo image using a first side-tuning adapter (STA) to generate a first feature map associated with the first stereo image;
processing a second stereo image using a second STA to generate a second feature map associated with the second stereo image, wherein the second STA is different from the first STA;
generating an initial hybrid cost volume based on the first feature map and the second feature map;
processing the initial hybrid cost volume using attentive hybrid cost filtering (AHCF) to generate a filtered hybrid cost volume; and
generating the output disparity map based on the filtered hybrid cost volume.
2 . The computer-implemented method of claim 1 , wherein the first STA comprises a first monocular depth estimation model and a first side-tuning convolutional neural network (CNN), wherein generating the first feature map comprises:
processing the first stereo image using the first monocular depth estimation model to generate monocular depth features;
processing the first stereo image using the first side-tuning CNN to generate CNN pyramid features; and
concatenating the monocular depth features and the CNN pyramid features to generate the first feature map.
3 . The computer-implemented method of claim 2 , wherein the second STA comprises a second monocular depth estimation model and a second side-tuning CNN, wherein generating the second feature map comprises:
processing the second stereo image using the second monocular depth estimation model to generate second monocular depth features;
processing the second stereo image using the second side-tuning CNN to generate second CNN pyramid features; and
concatenating the second monocular depth features and the second CNN pyramid features to generate the second feature map.
4 . The computer-implemented method of claim 1 , wherein generating the initial hybrid cost volume comprises:
generating a group correlation cost volume based on the first feature map and the second feature map;
generating a concatenation cost volume based on the first feature map and the second feature map; and
concatenating the group correlation cost volume and the concatenation cost volume to obtain the initial hybrid cost volume.
5 . The computer-implemented method of claim 4 , wherein generating the group correlation cost volume comprises:
performing a dot product of the first feature map and the second feature map to obtain the group correlation cost volume.
6 . The computer-implemented method of claim 5 , wherein generating the concatenation cost volume comprises:
concatenating the first feature map and the second feature map to obtain the concatenation cost volume.
7 . The computer-implemented method of claim 1 , wherein the AHCF comprises an axial-planar convolution (APC) filtering block and a disparity transformer (DT), and wherein processing the initial hybrid cost volume using the AHCF to generate the filtered hybrid cost volume comprises:
filtering the initial hybrid cost volume using the APC filtering block to generate an APC output;
processing disparity tokens associated with the initial hybrid cost volume using the DT to obtain a DT output; and
generating the filtered hybrid cost volume based on the APC output and the DT output.
8 . The computer-implemented method of claim 7 , wherein the APC filtering block comprises an hourglass neural network, and wherein filtering the initial hybrid cost volume using the APC filtering block to generate the APC output comprises performing convolution operations using the hourglass neural network to obtain the APC output from the initial hybrid cost volume.
9 . The computer-implemented method of claim 7 , wherein processing the disparity tokens to obtain the DT output comprises:
downsampling the initial hybrid cost volume to obtain a downsampled initial hybrid cost volume;
generating the disparity tokens based on reshaping the downsampled initial hybrid cost volume; and
processing the disparity tokens using the DT to generate the DT output, and
wherein generating the filtered hybrid cost volume based on the APC output and the DT output comprises:
upsampling the DT output to obtain an upsampled DT output; and
generating the filtered hybrid cost volume based on the APC output and the upsampled DT output.
10 . The computer-implemented method of claim 9 , wherein processing the disparity tokens using the DT to generate the DT output further comprises:
encoding positional information into the disparity tokens to obtain encoded disparity tokens; and
processing the encoded disparity tokens to generate the DT output.
11 . The computer-implemented method of claim 7 , wherein generating the filtered hybrid cost volume based on the APC output and the DT output comprises:
performing an element-wise add to the APC output and the DT output to obtain the filtered hybrid cost volume.
12 . The computer-implemented method of claim 1 , wherein generating the output disparity map based on the filtered hybrid cost volume comprises:
generating an initial disparity map based on the filtered hybrid cost volume;
performing one or more convolution gated recurrent unit (ConvGRU) operations to generate one or more intermediate disparity maps based on the initial disparity map; and
generating the output disparity map based on the one or more intermediate disparity maps.
13 . The computer-implemented method of claim 12 , wherein generating the one or more intermediate disparity maps comprises:
performing a first ConvGRU operation, from the one or more ConvGRU operations, to generate a first intermediate disparity map from the one or more intermediate disparity maps;
generating a first feature vector based on the filtered hybrid cost volume and the first intermediate disparity map;
generating a second feature vector based on the first intermediate disparity map and a correlation cost volume, wherein the correlation cost volume is generated based on performing a dot-product of the first feature map and the second feature map; and
performing a second ConvGRU operation, from the one or more ConvGRU operations, to generate a second intermediate disparity map, from the one or more intermediate disparity maps, based on the first intermediate disparity map, the first feature vector, and the second feature vector, and wherein generating the output disparity map is based on the second intermediate disparity map.
14 . The computer-implemented method of claim 12 , further comprising:
processing the second stereo image using a convolutional neural network to generate context features; and
concatenating the context features and monocular depth features from a monocular depth estimation model to obtain a context feature map, and wherein performing the one or more ConvGRU operations to generate the one or more intermediate disparity maps is further based on the context feature map.
15 . The computer-implemented method of claim 1 , wherein at least one of the steps of processing and generating are performed on a server or in a data center to generate the output disparity map, and the output disparity map is streamed to a user device.
16 . The computer-implemented method of claim 1 , wherein at least one of the steps of processing and generating are performed within a cloud computing environment.
17 . The computer-implemented method of claim 1 , wherein at least one of the steps of processing and generating are performed for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle.
18 . The computer-implemented method of claim 1 , wherein at least one of the steps of processing and generating are performed on a virtual machine comprising a portion of a graphics processing unit.
19 . A computer-implemented method for using a Foundational Stereo Model to generate an output disparity map, comprising:
processing a first stereo image using a first side-tuning adapter (STA) to generate a first feature map associated with the first stereo image, wherein the first STA comprises a first monocular depth estimation model and a first side-tuning convolutional neural network (CNN);
processing a second stereo image using a second STA to generate a second feature map associated with the second stereo image, wherein the second STA comprises a second monocular depth estimation model and a second side-tuning CNN, wherein the second STA is different from the first STA;
generating an initial hybrid cost volume based on the first feature map and the second feature map; and
generating the output disparity map based on the initial hybrid cost volume.
20 . The computer-implemented method of claim 19 , wherein generating the first feature map comprises:
processing the first stereo image using the first monocular depth estimation model to generate monocular depth features;
processing the first stereo image using the first side-tuning CNN to generate CNN pyramid features; and
concatenating the monocular depth features and the CNN pyramid features to generate the first feature map.
21 . The computer-implemented method of claim 20 , wherein generating the second feature map comprises:
processing the second stereo image using the second monocular depth estimation model to generate second monocular depth features;
processing the second stereo image using the second side-tuning CNN to generate second CNN pyramid features; and
concatenating the second monocular depth features and the second CNN pyramid features to generate the second feature map.
22 . The computer-implemented method of claim 19 , wherein generating the initial hybrid cost volume comprises:
generating a group correlation cost volume based on the first feature map and the second feature map;
generating a concatenation cost volume based on the first feature map and the second feature map; and
concatenating the group correlation cost volume and the concatenation cost volume to obtain the initial hybrid cost volume.
23 . The computer-implemented method of claim 22 , wherein generating the group correlation cost volume comprises performing a dot product of the first feature map and the second feature map to obtain the group correlation cost volume and wherein generating the concatenation cost volume comprises concatenating the first feature map and the second feature map to obtain the concatenation cost volume.
24 . A computer-implemented method for using a Foundational Stereo Model to generate an output disparity map, comprising:
generating an initial hybrid cost volume based on a first feature map associated with a first stereo image and a second feature map associated with a second stereo image;
filtering the initial hybrid cost volume by an axial-planar convolution (APC) filtering block to generate an APC output, wherein an attentive hybrid cost filtering (AHCF) comprises the APC filtering block and a disparity transformer (DT);
processing disparity tokens associated with the initial hybrid cost volume using the DT to obtain a DT output;
generating the filtered hybrid cost volume based on the APC output and the DT output; and
generating the output disparity map based on the filtered hybrid cost volume.
25 . The computer-implemented method of claim 24 , wherein the APC filtering block comprises an hourglass neural network, and wherein filtering the initial hybrid cost volume using the APC filtering block to generate the APC output comprises performing convolution operations using the hourglass neural network to obtain the APC output from the initial hybrid cost volume.
26 . The computer-implemented method of claim 24 , wherein processing the disparity tokens to obtain the DT output comprises:
downsampling the initial hybrid cost volume to obtain a downsampled initial hybrid cost volume;
generating the disparity tokens based on reshaping the downsampled initial hybrid cost volume; and
processing the disparity tokens using the DT to generate the DT output.
27 . The computer-implemented method of claim 26 , wherein generating the filtered hybrid cost volume based on the APC output and the DT output comprises:
upsampling the DT output to obtain an upsampled DT output; and
generating the filtered hybrid cost volume based on the APC output and the upsampled DT output.
28 . The computer-implemented method of claim 26 , wherein processing the disparity tokens using the DT to generate the DT output further comprises:
encoding positional information into the disparity tokens to obtain encoded disparity tokens; and
processing the encoded disparity tokens to generate the DT output.
29 . The computer-implemented method of claim 24 , wherein generating the filtered hybrid cost volume based on the APC output and the DT output comprises:
performing an element-wise add to the APC output and the DT output to obtain the filtered hybrid cost volume.