IP Library › Granted Patent US 10,911,677
Granted Patent B1
US 10,911,677 · App. 16/564,132 · Granted Feb 2, 2021

Multi-camera video stabilization techniques

Inventors: Jianping Zhou (Fremont, CA); Sebastien X. Beysserie (San Jose, CA); Tao Zhang (Campbell, CA)
Assignee: Apple Inc.
H04N5/23267G06T7/70H04N5/2258G06T2207/10016G06T2207/30244
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,911,677
App. No.
16/564,132
Granted
Feb 2, 2021
Kind
B1
Abstract

Various techniques are disclosed herein, which comprise obtaining a first stream of images from a first camera and a second stream of images from a second camera, wherein the first camera and second camera have different poses, and at least a portion of the image streams are captured concurrently. Next, a first stabilized trajectory and corresponding set of corrections used to stabilize the first stream of images are calculated. Based on determined differences between the poses of the first camera and second camera during the capture of the first and second streams of images, a second set of corrections to be applied to the second stream of images are derived to match the first stabilized trajectory. If desired, e.g., in response to determining that the stabilized first stream of images does not meet a stabilization criterion, the second stream of images may be stabilized using the second set of corrections.

Claims (76)

1. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:

obtain a first stream of images from a first camera of an electronic image capture device, wherein the first camera has a first pose and a first set of camera properties;

obtain a second stream of images from a second camera of the electronic image capture device, wherein the second camera has a second pose and a second set of camera properties, and wherein at least a portion of the second stream of images are captured concurrently with the first stream of images;

calculate a first stabilized trajectory for the first stream of images;

calculate a first set of corrections to be applied to the first stream of images to match the first stabilized trajectory;

stabilize the first stream of images using the first set of corrections;

determine differences between the first pose and the second pose during the capture of the first and second streams of images;

calculate a second set of corrections to be applied to the second stream of images to match the first stabilized trajectory based, at least in part, on the determined differences; and

stabilize the second stream of images using the second set of corrections, wherein the instructions to stabilize the second stream of images are executed in response to a determination that the stabilized first stream of images does not meet a first stabilization criterion.

2. The non-transitory program storage device of claim 1 ,

wherein the first pose comprises a first camera position and a first camera orientation,

wherein the second pose comprises a second camera position and a second camera orientation, and

wherein the first pose is different than the second pose.

3. The non-transitory program storage device of claim 1 ,

wherein the first set of camera properties comprises one or more of: an indication of a presence or status of an Optical Image Stabilization (OIS) system for the first camera; a field of view (FOV) of the first camera; a focal length of the first camera; or a zoom level of the first camera,

wherein the second set of camera properties comprises one or more of: an indication of a presence or status of an OIS system for the second camera; a FOV of the second camera; a focal length of the second camera; or a zoom level of the second camera, and

wherein at least one property in the first set of camera properties has a different value than a corresponding property in the second set of camera properties.

4. The non-transitory program storage device of claim 1 , wherein the instructions to determine differences between the first pose and the second pose during the capture of the first and second streams of images further comprise instructions to:

determine differences between the first and second cameras in one or more of: camera rotation; optical center; or parallax.

5. The non-transitory program storage device of claim 1 , wherein the instructions to calculate a first stabilized trajectory further comprise instructions to:

calculate a first stabilized trajectory that takes into account any corrections applied by an OIS system for the first camera during the capture of the first stream of images.

6. The non-transitory program storage device of claim 1 , wherein the instructions to stabilize the first stream of images using the first set of corrections further comprise instructions to:

perform a rolling shutter correction operation on the first stream of images.

7. The non-transitory program storage device of claim 1 , wherein the instructions to determine differences between the first pose and the second pose during the capture of the first and second streams of images further comprise instructions to:

determine one or more static camera alignment differences between the first camera and the second camera; and

determine one or more differences in lens position changes caused by respective OIS systems of the first or second cameras during the capture of the first and second image streams.

8. A video stabilization method, comprising:

obtaining a first stream of images from a first camera of an electronic image capture device, wherein the first camera has a first pose and a first set of camera properties;

obtaining a second stream of images from a second camera of the electronic image capture device, wherein the second camera has a second pose and a second set of camera properties, and wherein at least a portion of the second stream of images are captured concurrently with the first stream of images;

calculating a first stabilized trajectory for the first stream of images;

calculating a first set of corrections to be applied to the first stream of images to match the first stabilized trajectory;

stabilizing the first stream of images using the first set of corrections;

determining differences between the first pose and the second pose during the capture of the first and second streams of images;

calculating a second set of corrections to be applied to the second stream of images to match the first stabilized trajectory based, at least in part, on the determined differences; and

stabilizing the second stream of images using the second set of corrections, wherein the act of stabilizing the second stream of images is performed in response to a determination that the stabilized first stream of images does not meet a first stabilization criterion.

9. The video stabilization method of claim 8 ,

wherein the first set of camera properties comprises one or more of: an indication of a presence or status of an Optical Image Stabilization (OIS) system for the first camera; a FOV of the first camera; a focal length of the first camera; or a zoom level of the first camera,

wherein the second set of camera properties comprises one or more of: an indication of a presence or status of an OIS system for the second camera; a FOV of the second camera; a focal length of the second camera; or a zoom level of the second camera, and

wherein at least one property in the first set of camera properties has a different value than a corresponding property in the second set of camera properties.

10. The video stabilization method of claim 8 , wherein determining differences between the first pose and the second pose during the capture of the first and second streams of images further comprises:

determining differences between the first and second cameras in one or more of: camera rotation; optical center; or parallax.

11. The video stabilization method of claim 8 , wherein calculating a first stabilized trajectory further comprises:

calculating a first stabilized trajectory that takes into account any corrections applied by an OIS system for the first camera during the capture of the first stream of images.

12. The video stabilization method of claim 8 , wherein stabilizing the first stream of images using the first set of corrections further comprises:

performing a rolling shutter correction operation on the first stream of images.

13. The video stabilization method of claim 8 , wherein determining differences between the first pose and the second pose during the capture of the first and second streams of images further comprises:

determining one or more static camera alignment differences between the first camera and the second camera; and

determining one or more differences in lens position changes caused by respective OIS systems of the first or second cameras during the capture of the first and second image streams.

14. An electronic image capture device, comprising:

a first camera, wherein the first camera has a first pose;

a second camera, wherein the second camera has a second pose;

a memory;

one or more processors operatively coupled to the memory and configured to execute instructions causing the one or more processors to:

obtain a first stream of images from the first camera;

obtain a second stream of images from the second camera, wherein at least a portion of the second stream of images are captured concurrently with the first stream of images;

calculate a first stabilized trajectory for the first stream of images;

calculate a first set of corrections to be applied to the first stream of images to match the first stabilized trajectory;

stabilize the first stream of images using the first set of corrections;

determine differences between the first pose and the second pose during the capture of the first and second streams of images;

calculate a second set of corrections to be applied to the second stream of images to match the first stabilized trajectory based, at least in part, on the determined differences; and

stabilize the second stream of images using the second set of corrections, wherein the instructions to stabilize the second stream of images are executed in response to a determination that the stabilized first stream of images does not meet a first stabilization criterion.

15. The electronic image capture device of claim 14 ,

wherein the first pose comprises a first camera position and a first camera orientation,

wherein the second pose comprises a second camera position and a second camera orientation, and

wherein the first pose is different than the second pose.

16. The electronic image capture device of claim 14 , wherein the instructions to determine differences between the first pose and the second pose during the capture of the first and second streams of images further comprise instructions to:

determine differences between the first and second cameras in one or more of: camera rotation; optical center; or parallax.

17. The electronic image capture device of claim 14 , wherein the instructions to calculate a first stabilized trajectory further comprise instructions to:

calculate a first stabilized trajectory that takes into account any corrections applied by an OIS system for the first camera during the capture of the first stream of images.

18. The electronic image capture device of claim 14 , wherein the instructions to stabilize the first stream of images using the first set of corrections further comprise instructions to:

perform a rolling shutter correction operation on the first stream of images.

19. The electronic image capture device of claim 14 , wherein the instructions to determine differences between the first pose and the second pose during the capture of the first and second streams of images further comprise instructions to:

determine one or more static camera alignment differences between the first camera and the second camera; and

determine one or more differences in lens position changes caused by respective OIS systems of the first or second cameras during the capture of the first and second image streams.

20. The electronic image capture device of claim 14 , wherein the instructions further comprise instructions to:

store the stabilized first stream of images and the stabilized second stream of images in a common file structure in the memory.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 9, 2019
From: ZHOU, JIANPING; ZHANG, TAO; BEYSSERIE, SEBASTIEN X.
To: APPLE INC.
Reel/Frame 050310/0643 →
Cited By (4)
US 12,205,327 US 12,289,522 US 12,299,934 US 12,390,281