Granted Patent B1

US 12,470,683 · App. 18/817,540 · Granted Nov 11, 2025

Inventors: Felix Heide (New York City, NY); Anush Kumar (Austin, TX); Shile Li (Munich, DE); Omid Hosseini Jafari (Munich, DE); Fahim Mannan (Toronto, CA)

Assignee: Torc Robotics, Inc.

H04N13/246G06V10/44G06V10/761G06V10/771H04N13/239

Abstract

Claims (72)

1 . An autonomous vehicle comprising:

a stereo camera pair disposed on the autonomous vehicle, the stereo camera pair comprising a first camera and a second camera separated by a baseline distance, the first camera and the second camera configured to capture a first image and a second image, respectively;

at least one memory device storing computer executable instructions; and

at least one processor coupled to the at least one memory device and the stereo camera pair, the least one processor, upon execution of the computer executable instructions, configured to:

receive the first image and the second image captured using respective cameras in the stereo camera pair;

predict, using a neural network model, a rotation matrix between the first image and the second image by:

extracting a first feature map and a second feature map based on the first image and the second image;

applying positional feature enhancement on the first feature map and the second feature map to derive a first enhanced feature map and a second enhanced feature map;

computing a correlation volume across the first enhanced feature map and the second enhanced feature map;

determining a set of likely matches between the first enhanced feature map and the second enhanced feature map based on the correlation volume;

computing a predicted relative pose based on the set of likely matches; and

computing the rotation matrix based on the predicted relative pose; and

calibrate the stereo camera pair by:

employing differentiable rectification to rectify the first image and the second image based on the rotation matrix.

2 . The autonomous vehicle of claim 1 wherein the first camera and the second camera are positioned laterally at least 60 cm apart.

3 . The autonomous vehicle of claim 1 , wherein the at least one processor is further configured to:

train the neural network model by a self-supervised learning.

4 . The autonomous vehicle of claim 1 , wherein the at least one processor is further configured to:

train the neural network model by optimizing a loss function including a vertical optical flow of the first image and the second image.

5 . The autonomous vehicle of claim 4 , wherein the at least one processor is further configured to train the neural network model by:

employing the differentiable rectification to the first image and the second image; and

adjusting the neural network model to minimize the vertical optical flow of the first image and the second image.

6 . The autonomous vehicle of claim 1 , wherein the at least one processor is further configured to:

calibrate the stereo camera pair by calibrating the camera pair while an autonomous vehicle equipped with the camera pair is operating.

7 . The autonomous vehicle of claim 1 , wherein the at least one processor is further configured to:

extract the first feature map and the second feature map further by:

employing the differentiable rectification to rectify the first feature map and the second feature map; and

apply the positional feature enhancement by:

applying the positional feature enhancement to the first rectified feature map and the second rectified feature map.

8 . The autonomous vehicle of claim 1 , wherein the at least one processor is further configured to:

compute the correlation volume by:

flattening the first feature map along height by width of the first feature map to produce a two-dimensional first feature map; and

flattening the second feature map along height by width of the second feature map to produce a two-dimensional second feature map.

9 . A computer-implemented method of calibrating a stereo camera pair, the method comprising:

capturing a first image and a second image using respective cameras in a stereo camera pair;

predicting a rotation matrix between the first image and the second image by:

extracting a first feature map and a second feature map based on the first image and the second image;

applying positional feature enhancement on the first feature map and the second feature map to derive a first enhanced feature map and a second enhanced feature map;

computing a correlation volume across the first enhanced feature map and the second enhanced feature map;

determining a set of likely matches between the first enhanced feature map and the second enhanced feature map based on the correlation volume;

computing a predicted relative pose based on the set of likely matches; and

computing the rotation matrix based on the predicted relative pose; and

calibrating the stereo camera pair by:

employing differentiable rectification to rectify the first image and the second image based on the rotation matrix.

10 . The method of claim 9 further comprising:

training the neural network model by a self-supervised learning.

11 . The method of claim 9 , further comprising:

training the neural network model by:

optimizing a loss function including a vertical optical flow of the first image and the second image.

12 . The method of claim 11 , wherein the training the neural network model further comprises:

employing the differentiable rectification to the first image and the second image; and

adjusting the neural network model to minimize the vertical optical flow of the first image and the second image.

13 . The method of claim 9 , wherein the calibrating the stereo camera pair further comprises:

calibrating the camera while a machine equipped with the camera pair is operating.

14 . The method of claim 9 , wherein:

the extracting the first feature map and the second feature map further comprises:

employing the differentiable rectification to rectify the first feature map and the second feature map; and

the applying the positional feature enhancement further comprises:

applying the positional feature enhancement to the first rectified feature map and the second rectified feature map.

15 . The method of claim 9 , wherein the computing the correlation volume further comprises:

flattening the first feature map along height by width of the first feature map to produce a two-dimensional first feature map; and

flattening the second feature map along height by width of the second feature map to produce a two-dimensional second feature map.

16 . The method of claim 15 , wherein the computing the correlation volume further comprises:

applying a soft-max along the last two dimensions of the correlation volume to convert the correlation volume into a likelihood of the set of likely matches.

17 . The method of claim 9 , wherein the computing the predicted relative pose further comprises:

flattening the set of likely matches to a one-dimensional list.

18 . The method of claim 9 , wherein the computing the predicted relative pose further comprises:

orthogonalizing the predicted relative pose using Gram-Schmidt orthogonalization.

19 . The method of claim 9 , wherein the determining the set of likely matches comprises:

determining a set of likely matches using a decoder layer of the neural network model.

20 . The method of claim 9 , further comprising:

training the neural network model according to a loss function: L=λ 1 L rot +λ 2 L flow , where λ 1 , λ 2 are scalar weights, L rot is a pose loss supervised on ground truth calibration data, and L flow is a self-supervised vertical-flow loss.

Assignments (1)

Continuity (1)

References Cited (37)