IP Library Granted Patent US 9,325,917
Granted Patent B2
US 9,325,917 · App. 14/606,019 · Granted Apr 26, 2016

Auto-alignment of image sensors in a multi-camera system

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,325,917
App. No.
14/606,019
Granted
Apr 26, 2016
Kind
B2
Abstract

A pair of cameras having an overlapping field of view is aligned based on images captured by image sensors of the pair of cameras. A pixel shift is identified between the images. Based on the identified pixel shift, a calibration is applied to one or both of the pair of cameras. To determine the pixel shift, the camera applies correlation methods including edge matching. Calibrating the pair of cameras may include adjusting a read window on an image sensor. The pixel shift can also be used to determine a time lag, which can be used to synchronize subsequent image captures.

Claims (47)

1. A computer-implemented method for synchronizing a pair of image sensors, the method comprising:

capturing an image with each image sensor at a substantially same time, the image sensors having a rolling shutter direction and an overlapping field of view;

correlating the captured image data representative of the overlapping field of view by shifting at least one image by a first number of pixels along the rolling shutter direction such that a measure of difference between pixels of the captured image data representative of the overlapping field of view is substantially minimized;

identifying, by one or more processors, a pixel shift between the captured images based on the first number of pixels; and

calibrating, based on the identified pixel shift, at least one image sensor to synchronize subsequent image capture by the image sensors.

2. The method of claim 1 , wherein correlating the captured image data representative of the overlapping field of view comprises:

compressing, for each image, pixels in the captured image data representative of the overlapping field of view; and

correlating the compressed pixels of each image.

3. The method of claim 1 , further comprising:

responsive to a determination that the captured images include less than a threshold amount of image data representative of the overlapping field of view required to correlate the captured image data, capturing an additional pair of images having an overlapping field of view with the image sensors.

4. The method of claim 1 , further comprising:

determining a time lag between the image sensors based on the identified pixel shift.

5. The method of claim 4 , wherein determining the time lag between the image sensors comprises determining an amount of time necessary for one of the image sensors to capture a number of pixel lines corresponding to the pixel shift.

6. The method of claim 4 , wherein a first image sensor of the pair of image sensors lags a second image sensor of the pair of image sensors during image capture, and wherein calibrating the at least one image sensor comprises:

initiating image capture at the first image sensor; and

after delaying for an amount of time equal to the time lag, initiating image capture at the second image sensor.

7. The method of claim 1 , wherein calibrating the at least one image sensor comprises:

adjusting a read window of the at least one image sensor in the rolling shutter direction, the read window comprising a portion of the image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

8. The method of claim 1 , wherein the pair of image sensors are controlled by a master camera controlling a slave camera.

9. The method of claim 1 , repeated to synchronize a plurality of pairs of image sensors.

10. A system comprising a pair of cameras, each camera having an image sensor, the system further comprising at least one processor and a non-transitory computer readable medium comprising instructions for synchronizing the image sensors, the instructions executable by the at least one processor, the instructions comprising instructions for:

capturing an image with each image sensor at a substantially same time, the image sensors having a rolling shutter direction and an overlapping field of view;

correlating the captured image data representative of the overlapping field of view by shifting at least one image by a first number of pixels along the rolling shutter direction such that a measure of difference between pixels of the captured image data representative of the overlapping field of view is substantially minimized;

identifying, by one or more processors, a pixel shift between the captured images based on the first number of pixels; and

calibrating, based on the identified pixel shift, at least one image sensor to synchronize subsequent image capture by the image sensors.

11. The system of claim 10 , wherein calibrating the at least one image sensor comprises:

adjusting a read window of the at least one image sensor in the rolling shutter direction, the read window comprising a portion of the image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

12. The system of claim 10 , further comprising instructions for:

determining a time lag between the image sensors based, at least in part, on the identified pixel shift and an amount of time necessary for one of the image sensors to capture a number of pixel lines corresponding to the pixel shift.

13. The system of claim 12 , wherein a first image sensor of the pair of image sensors lags a second image sensor of the pair of image sensors during image capture, and wherein calibrating the at least one image sensor comprises:

initiating image capture at the first image sensor; and

after delaying for an amount of time equal to the time lag, initiating image capture at the second image sensor.

14. The system of claim 10 , further comprising a housing encompassing the pair of cameras.

15. The system of claim 10 , further comprising a third camera having an additional image sensor, the instructions further comprising instructions to synchronize the additional image sensor with the image sensors.

16. A non-transitory computer-readable medium comprising instructions for synchronizing a pair of image sensors, the instructions executable by at least one processor, the instructions comprising instructions for:

capturing an image with each image sensor at a substantially same time, the image sensors having a rolling shutter direction and an overlapping field of view;

correlating the captured image data representative of the overlapping field of view by shifting at least one image by a first number of pixels along the rolling shutter direction such that a measure of difference between pixels of the captured image data representative of the overlapping field of view is substantially minimized;

identifying, by one or more processors, a pixel shift between the captured images based on the first number of pixels; and

calibrating, based on the identified pixel shift, at least one image sensor to synchronize subsequent image capture by the image sensors.

17. The computer-readable medium of claim 16 , wherein calibrating the at least one image sensor comprises:

adjusting a read window of the at least one image sensor in the rolling shutter direction, the read window comprising a portion of the image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

18. The computer-readable medium of claim 16 , further comprising instructions for:

determining a time lag between the image sensors based, at least in part, on the identified pixel shift and an amount of time necessary for one of the image sensors to capture a number of pixel lines corresponding to the pixel shift.

19. The computer-readable medium of claim 18 , wherein a first image sensor of the pair of image sensors lags a second image sensor of the pair of image sensors during image capture, and wherein calibrating the at least one image sensor comprises:

initiating image capture at the first image sensor; and

after delaying for an amount of time equal to the time lag, initiating image capture at the second image sensor.

20. The computer readable medium of claim 16 , the instructions further comprising instructions to synchronize an additional image sensor with the pair of image sensors.

Assignments (6)
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: FARALLON CAPITAL MANAGEMENT, L.L.C., AS AGENT
Reel/Frame 072340/0676 →
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: WELLS FARGO BANK, NATIONAL ASSOCIATION, AS AGENT
Reel/Frame 072358/0001 →
RELEASE OF PATENT SECURITY INTEREST Recorded Jan 25, 2021
From: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
To: GOPRO, INC.
Reel/Frame 055106/0434 →
SECURITY AGREEMENT Recorded Aug 29, 2016
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 039851/0611 →
SECURITY AGREEMENT Recorded Mar 28, 2016
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 038184/0779 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 30, 2015
From: MACMILLAN, TIMOTHY; CAMPBELL, SCOTT PATRICK; NEWMAN, DAVID A.; SUN, YAJIE
To: GOPRO, INC.
Reel/Frame 034852/0556 →