IP Library › Granted Patent US 12664690

Granted Patent B2

US 12664690 · App. 18/233,806 · Granted Jun 23, 2026

Apparatus and methods to calibrate a stereo camera pair

Inventors: Anurag Ganguli (Saratoga, CA); Timothy P. Daly, Jr. (San Jose, CA); Mayank Gupta (Mountain View, CA); Wenbin Wang (Cupertino, CA); Huan Yang Chang (San Jose, CA)

Assignee: PlusAI, Inc.

G06T7/85 G06T7/593 H04N13/239 B60W60/00 B60W2420/403 G06T2207/30252 H04N2013/0081

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12664690
App. No.: 18/233,806
Filed: Aug 14, 2023
Granted: Jun 23, 2026
Kind: B2
Examiner: HODGES, SUSAN E
Art Unit: 2425
USPC: 348/47

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

In some embodiments, a method includes receiving a first image and a second image from a stereo camera pair. The method includes selecting a first row of pixels from the rectified image and a set of rows of pixels from the second image and comparing the first row of pixels with each row of pixels from the set of rows of pixels to determine disparity values. The method includes determining a pair of rows of pixels having the first row of pixels and a second row of pixels from the set of rows of pixels. The pair of rows of pixels has an offset no greater than an offset between the first row of pixels and each row of pixels from remaining rows of pixels. The method includes adjusting, based on the offset, the relative rotational position between the first stereo camera and the second stereo camera.

Claims (66)

1 . A computer-implemented method comprising:

transforming, by a computing system, a first image of an image pair to a rectified first image based on at least one of: (1) a translational position between a first camera that captured the first image and a second camera, (2) a rotational position between the first camera and the second camera, and (3) lens distortion;

selecting, by the computing system, a first row in a region in the rectified first image that crosses a center of an object in the rectified first image, wherein the region is selected to maximize sensitivity to calibration errors affecting long range depth estimation and comprises at least one of a horizon, a vanishing point of a lane, or a bounding box of the object;

determining, by the computing system, a second row in a second image captured by the second camera that corresponds with the first row in the rectified first image, wherein the second image is not transformed as the first image is transformed;

calibrating, by the computing system, at least one of: the first camera and the second camera based on the first row in the rectified first image and the second row in the second image.

2 . The computer-implemented method of claim 1 , wherein transforming the first image of the image pair comprises:

applying, by the computing system, an image rectification process to the first image; and

projecting, by the computing system, the rectified first image and the second image to a common image plane.

3 . The computer-implemented method of claim 1 , wherein selecting the first row in the rectified first image comprises:

generating, by the computing system, a first grid for the rectified first image; and

detecting, by the computing system, the object in the rectified first image, wherein the first row in the rectified first image is determined based on the center of the object in the first grid.

4 . The computer-implemented method of claim 1 , wherein determining the second row in the second image comprises:

generating, by the computing system, a second grid for the second image, wherein the second grid has a set of rows corresponding with a first grid generated for the rectified first image; and

determining, by the computing system, a set of disparity values for each row of the set of rows of the second image based on a comparison of the row with the first row in the rectified first image, wherein the second row in the second image is determined based on the set of disparity values for each row of the set of rows of the second image.

5 . The computer-implemented method of claim 1 , wherein determining the second row in the second image comprises:

selecting, by the computing system, a first cell in the first row of the rectified first image;

determining, by the computing system, a second cell in the second image based on a search of the second image for a match of the first cell in the first row of the rectified first image, wherein the second row in the second image is determined based on the second cell.

6 . The computer-implemented method of claim 1 , wherein the first row in the rectified first image that crosses the center of the object in the rectified first image is determined based on motion of the first camera and the second camera.

7 . The computer-implemented method of claim 1 , wherein calibrating at least one of: the first camera and the second camera comprises:

determining, by the computing system, a position difference between the first camera and the second camera; and

adjusting, by the computing system, an estimated position of the at least one of: the first camera and the second camera based on the position difference.

8 . The computer-implemented method of claim 1 , wherein calibrating at least one of: the first camera and the second camera comprises:

determining, by the computing system, a position difference between the first camera and the second camera; and

causing, by the computing system, the at least one of: the first camera and the second camera to be moved based on the position difference.

9 . The computer-implemented method of claim 1 , wherein the first image and the second image are captured at a first timestamp, the computer-implemented method further comprising:

determining, by the computing system, a first set of disparity values based on the rectified first image and the second image; and

determining, by the computing system, a second set of disparity values based on a third image captured by the first camera at a second timestamp and a fourth image captured by the second camera at the second timestamp, wherein calibrating the at least one of: the first camera and the second camera is based on the first set of disparity values and the second set of disparity values.

10 . The computer-implemented method of claim 1 , wherein calibrating the at least one of: the first camera and the second camera comprises:

determining, by the computing system, an offset between first pixels of the first row in the rectified first image and second pixels of the second row in the second image;

determining, by the computing system, a position difference between the first camera and the second camera based on the offset, wherein the calibrating the at least one of: the first camera and the second camera is based on the position difference.

11 . A system comprising:

at least one processor; and

a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising:

transforming a first image of an image pair to a rectified first image based on at least one of: (1) a translational position between a first camera that captured the first image and a second camera, (2) a rotational position between the first camera and the second camera, and (3) lens distortion;

selecting a first row in a region in the rectified first image that crosses a center of an object in the rectified first image, wherein the region is selected to maximize sensitivity to calibration errors affecting long range depth estimation and comprises at least one of a horizon, a vanishing point of a lane, or a bounding box of the object;

determining a second row in a second image captured by the second camera that corresponds with the first row in the rectified first image, wherein the second image is not transformed as the first image is transformed;

calibrating at least one of: the first camera and the second camera based on the first row in the rectified first image and the second row in the second image.

12 . The system of claim 11 , wherein transforming the first image of the image pair comprises:

applying an image rectification process to the first image; and

projecting the rectified first image and the second image to a common image plane.

13 . The system of claim 11 , wherein selecting the first row in the rectified first image comprises:

generating a first grid for the rectified first image; and

detecting the object in the rectified first image, wherein the first row in the rectified first image is determined based on the center of the object in the first grid.

14 . The system of claim 11 , wherein determining the second row in the second image comprises:

generating a second grid for the second image, wherein the second grid has a set of rows corresponding with a first grid generated for the rectified first image; and

determining a set of disparity values for each row of the set of rows of the second image based on a comparison of the row with the first row in the rectified first image, wherein the second row in the second image is determined based on the set of disparity values for each row of the set of rows of the second image.

15 . The system of claim 11 , wherein determining the second row in the second image comprises:

selecting a first cell in the first row of the rectified first image;

determining a second cell in the second image based on a search of the second image for a match of the first cell in the first row of the first image, wherein the second row in the second image is determined based on the second cell.

16 . A non-transitory computer-readable storage medium including instructions that, when executed by at least on processor of a computing system, cause the computing system to perform operations comprising:

calibrating at least one of: the first camera and the second camera based on the first row in the rectified first image and the second row in the second image.

17 . The non-transitory computer-readable storage medium of claim 16 , wherein transforming the first image of the image pair comprises:

applying an image rectification process to the first image; and

projecting the rectified first image and the second image to a common image plane.

18 . The non-transitory computer-readable storage medium of claim 16 , wherein selecting the first row in the rectified first image comprises:

generating a first grid for the rectified first image; and

detecting the object in the rectified first image, wherein the first row in the rectified first image is determined based on the center of the object in the first grid.

19 . The non-transitory computer-readable storage medium of claim 16 , wherein determining the second row in the second image comprises:

generating a second grid for the second image, wherein the second grid has a set of rows corresponding with a first grid generated for the rectified first image; and

20 . The non-transitory computer-readable storage medium of claim 16 , wherein determining the second row in the second image comprises:

selecting a first cell in the first row of the rectified first image;

determining a second cell in the second image based on a search of the second image for a match of the first cell in the first row of the rectified first image, wherein the second row in the second image is determined based on the second cell.