IP Library Granted Patent US 10,389,993
Granted Patent B2
US 10,389,993 · App. 15/077,911 · Granted Aug 20, 2019

Auto-alignment of image sensors in a multi-camera system

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,389,993
App. No.
15/077,911
Granted
Aug 20, 2019
Kind
B2
Abstract

A pair of cameras having an overlapping field of view is aligned based on images captured by image sensors of the pair of cameras. A pixel shift is identified between the images. Based on the identified pixel shift, a calibration is applied to one or both of the pair of cameras. To determine the pixel shift, the camera applies correlation methods including edge matching. Calibrating the pair of cameras may include adjusting a read window on an image sensor. The pixel shift can also be used to determine a time lag, which can be used to synchronize subsequent image captures.

Claims (47)

1. A method for synchronizing a pair of image sensors, the method comprising:

accessing a pair of images synchronously captured by the pair of image sensors, wherein the pair of image sensors include an image sensor of a first camera and an image sensor of a second camera, wherein the pair of image sensors are controlled by the first camera controlling the image sensor of the second camera;

identifying a pixel shift of a first of the pair of images relative to a second of the pair of images by matching a first edge of the first of the pair of images with a second edge of the second of the pair of images;

determining a time lag between the pair of image sensors based on the identified pixel shift, the time lag comprising an amount of time required to capture a number of pixel lines corresponding to the identified pixel shift;

generating a calibration measure based on the identified pixel shift for calibrating at least one image sensor to synchronize subsequent image capture by the pair of image sensors; and

capturing a subsequent pair of images based on the generated calibration measure.

2. The method of claim 1 , wherein identifying the pixel shift comprises:

compressing, for each image, image pixels representative of an overlapping field of view of the pair of image sensors; and

correlating the compressed pixels of each image.

3. The method of claim 1 , further comprising:

calibrating one or both of the pair of image sensors using the generated calibration measure.

4. The method of claim 3 , wherein calibrating one or both of the pair of image sensors comprises:

adjusting a read window of one or both of the pair image sensors in a rolling shutter direction of the pair of image sensors based at least in part on the generated calibration measure, each read window comprising a portion of an image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

5. The method of claim 3 , wherein each image sensor is located within a different camera system of a pair of communicatively coupled camera systems, and wherein one of the pair of communicatively coupled camera systems provides the generated calibration measure to the other of the pair of communicatively coupled camera systems.

6. The method of claim 3 , wherein each image sensor is located within a different camera system of a pair of camera systems, wherein each of the pair of camera systems is communicatively coupled to an external controller, and wherein the external controller is configured to provide the generated calibration measure to one or both of the pair of camera systems.

7. The method of claim 6 , wherein the pair of images are synchronously captured in response to an instruction provided by the external controller.

8. A multi-camera system, each camera having an image sensor, the multi-camera system configured to:

access a pair of images synchronously captured by the image sensors, wherein the image sensors include an image sensor of a first camera and an image sensor of a second camera, wherein the image sensors are controlled by the first camera controlling the image sensor of the second camera;

identify a pixel shift of a first of the pair of images relative to a second of the pair of images by matching a first edge of the first of the pair of images with a second edge of the second of the pair of images;

determine a time lag between the pair of image sensors based on the identified pixel shift, the time lag comprising an amount of time required to capture a number of pixel lines corresponding to the identified pixel shift;

generate a calibration measure based on the identified pixel shift for calibrating at least one image sensor to synchronize subsequent image capture by the image sensors; and

capture a subsequent pair of images based on the generated calibration measure.

9. The system of claim 8 , wherein identifying the pixel shift comprises:

compressing, for each image, image pixels representative of an overlapping field of view of the image sensors; and

correlating the compressed pixels of each image.

10. The system of claim 8 , further configured to calibrate one or more of the image sensors using the generated calibration measure.

11. The system of claim 10 , wherein calibrating one or more of the image sensors comprises:

adjusting a read window of one or more of the image sensors in a rolling shutter direction of the image sensors based at least in part on the generated calibration measure, each read window comprising a portion of an image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

12. The system of claim 10 , wherein one camera provides the generated calibration measure to another camera.

13. The system of claim 10 , further comprising:

a controller configured to provide the generated calibration measure to each camera.

14. The system of claim 13 , wherein the pair of images are synchronously captured in response to an instruction provided by the controller.

15. A non-transitory computer-readable medium comprising instructions for synchronizing a pair of image sensors, the instructions executable by at least one processor, the instructions comprising instructions for:

accessing a pair of images synchronously captured by the pair of image sensors, wherein the pair of image sensors include an image sensor of a first camera and an image sensor of a second camera, wherein the pair of image sensors are controlled by the first camera controlling the image sensor of the second camera;

identifying a pixel shift of a first of the pair of images relative to a second of the pair of images by matching a first edge of the first of the pair of images to a second edge of the second of the pair of images;

determining a time lag between the pair of image sensors based on the identified pixel shift, the time lag comprising an amount of time required to capture a number of pixel lines corresponding to the identified pixel shift;

generating a calibration measure based on the identified pixel shift for calibrating at least one image sensor to synchronize subsequent image capture by the pair of image sensors; and

capturing a subsequent pair of images based on the generated calibration measure.

16. The computer-readable medium of claim 15 , wherein identifying the pixel shift comprises:

compressing, for each image, image pixels representative of an overlapping field of view of the pair of image sensors; and

correlating the compressed pixels of each image.

17. The computer-readable medium of claim 15 , the instructions further comprising instructions for:

calibrating one or both of the pair of image sensors using the generated calibration measure.

18. The computer-readable medium of claim 17 , wherein calibrating one or both of the pair of image sensors comprises:

adjusting a read window of one or both of the pair of image sensors in a rolling shutter direction of the pair of image sensors based at least in part on the generated calibration measure, each read window comprising a portion of an image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

19. The method of claim 17 , wherein each image sensor is located within a different camera system of a pair of communicatively coupled camera systems, and wherein one of the pair of communicatively coupled camera systems provides the generated calibration measure to the other of the pair of communicatively coupled camera systems.

20. The method of claim 17 , wherein each image sensor is located within a different camera system of a pair of camera systems, wherein each of the pair of camera systems is communicatively coupled to an external controller, and wherein the external controller is configured to provide the generated calibration measure to one or both of the pair of camera systems.

Assignments (6)
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: FARALLON CAPITAL MANAGEMENT, L.L.C., AS AGENT
Reel/Frame 072340/0676 →
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: WELLS FARGO BANK, NATIONAL ASSOCIATION, AS AGENT
Reel/Frame 072358/0001 →
RELEASE OF PATENT SECURITY INTEREST Recorded Jan 25, 2021
From: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
To: GOPRO, INC.
Reel/Frame 055106/0434 →
SECURITY INTEREST Recorded Oct 19, 2020
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 054113/0594 →
SECURITY AGREEMENT Recorded Aug 29, 2016
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 039851/0611 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 17, 2016
From: MACMILLAN, TIMOTHY; CAMPBELL, SCOTT PATRICK; NEWMAN, DAVID A.; SUN, YAJIE
To: GOPRO, INC.
Reel/Frame 038621/0703 →
Cited By (5)
US 12,368,976 US 12,541,881 US 12,556,802 US 12,608,041 US 12,699,420