IP Library › Granted Patent US 10,148,893
Granted Patent B2
US 10,148,893 · App. 15/452,420 · Granted Dec 4, 2018

Methods, systems, and media for high dynamic range imaging

Inventors: Mohit Gupta (New York, NY); Tomoo Mitsunaga (Kanagawa, JP); Daisuke Iso (Tokyo, JP); Shree K. Nayar (New York, NY)
Assignee: Sony Corporation
H04N5/2355G06T5/007G06T5/50G06T7/20H04N5/2258H04N5/2352H04N5/2356G06T2207/10016G06T2207/10144G06T2207/20208H04N5/2353
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,148,893
App. No.
15/452,420
Filed
Mar 7, 2017
Granted
Dec 4, 2018
Kind
B2
Art Unit
2661
USPC
348/222.1
Abstract

Systems, methods, and media for high dynamic range imaging are provided, the systems comprising: an image sensor; and a hardware processor configured to: cause the image sensor to capture first image data having a first exposure time, second image data having a second exposure time, and third image data having a third exposure time that is substantially equal to the sum of the first exposure time and the second exposure time; generate combined image data using the first image data and the second image data.

Claims (51)

1. An image processing apparatus for high dynamic range imaging, the image processing apparatus comprising:

at least one image sensor; and

at least one hardware processor configured to;

receive a time budget for capturing a plurality of images including images corresponding to first image data, second image data, and third image data;

determine one or more properties of a scene to be captured;

determine a first exposure time and a second exposure time based, at least in part, on the one or more properties of the scene;

instruct the at least one image sensor to capture the first image data having the first exposure time, the second image data having the second exposure time, and the third image data having a third exposure time, wherein the third exposure time is substantially equal to a sum of the first exposure time and the second exposure time; and

generate processed image data based, at least in part, on the first image data, the second image data and the third image data.

2. The image processing apparatus of claim 1 , wherein generating the processed image data comprises generating combined image data by combining the first image data and the second image data.

3. The image processing apparatus of claim 2 , wherein the at least one hardware processor is further configured to determine first estimated motion data representative of motion between the combined image data and the third image data.

4. The image processing apparatus of claim 3 , wherein the at least one hardware processor is further configured to determine based, at least in part, on the first estimated motion data, second estimated motion data representative of motion between the second image data and the third image data.

5. The image processing apparatus of claim 4 , wherein generating the processed image data comprises generating the processed image data using at least the first image data, the second image data, the third image data, and the second estimated motion data.

6. The image processing apparatus of claim 1 , wherein the processed image data comprises a high dynamic range image.

7. The image processing apparatus of claim 1 , wherein a sum of the first exposure time and the second exposure time includes a time between capturing the first image data and capturing the second image data.

8. The image processing apparatus of claim 1 , wherein the first exposure time corresponds to a minimum exposure time of the at least one image sensor.

9. The image processing apparatus of claim 1 , wherein the at least one hardware processor is further configured to:

instruct the at least one image sensor to capture fourth image data having a fourth exposure time that is substantially equal to a sum of the second exposure time and the third exposure time;

generate combined image data using the second image data and the third image data;

determine first estimated motion data representative of motion between the combined image data and the third image data;

determine second estimated motion data representative of motion between the third image data and the fourth image data based on the first estimated motion data; and

generate the processed image data based, at least in part, on the fourth image data and the second estimated motion data.

10. The image processing apparatus of claim 1 , wherein the one or more properties of the scene include at least one of a dynamic range of the scene, an amount of scene motion, and an amount of camera motion.

11. The image processing apparatus of claim 4 , wherein determining the second estimated motion data comprises scaling the first estimated motion data based on a ratio of a sum of the second exposure time and the third exposure time to a sum of the first exposure time, the second exposure time and the third exposure time.

12. The image processing apparatus of claim 1 , wherein the at least one hardware processor is further configured to:

(a) receive an instruction to capture video;

(b) receive the one or more properties of the scene to be captured;

(c) set one or more exposure times to be used in capturing a frame of video based on the one or more properties of the scene;

(d) cause image data to be captured by the at least one image sensor using the first exposure time, the second exposure time, and the third exposure time in response to determining that the one or more exposure times includes the first exposure time, the second exposure time, and the third exposure time;

(e) cause image data to be captured by the at least one image sensor using at least a fourth exposure time in response to determining that the one or more exposure times do not include the first exposure time, the second exposure time, and the third exposure time;

(f) determine a value of at least one of the one or more properties of the scene to be captured based on the image data captured at either (d) or (e);

(g) repeat (b)-(f) until an instruction is received to stop capturing video;

(h) generate the processed image data using at least a portion of the image data captured at (d); and

(i) store the processed image data as a frame of the video.

13. An image processing apparatus for high dynamic range imaging, the image processing apparatus comprising:

at least one image sensor; and

at least one hardware processor configured to:

(a) receive an instruction to capture video;

(b) receive one or more properties of the scene to be captured;

(c) set one or more exposure times to be used in capturing a frame of video based on the one or more properties of the scene;

(d) instruct the at least one image sensor to capture image data having a first exposure time, a second exposure time, and a third exposure time in response to determining that the one or more exposure times includes the first exposure time, the second exposure time, and the third exposure time, wherein the third exposure time is substantially equal to a sum of the first exposure time and the second exposure time;

(e) instruct the at least one image sensor to capture image data using at least a fourth exposure time in response to determining that the one or more exposure times do not include the first exposure time, the second exposure time, and the third exposure time;

(f) determine a value of at least one of the one or more properties of the scene to be captured based on the image data captured at either (d) or (e);

(g) repeat (b)-(f) until an instruction is received to stop capturing video;

(h) generate processed image data using at least a portion of the image data captured at (d); and

(i) store the processed image data as a frame of the video.

14. An image processing method for high dynamic range imaging, the image processing method comprising:

receiving a time budget for capturing a plurality of images including images corresponding to first image data, second image data, and third image data;

determining one or more properties of a scene to be captured;

determining a first exposure time and a second exposure time based, at least in part, on the one or more properties of the scene;

instructing at least one image sensor to capture the first image data having the first exposure time, the second image data having the second exposure time, and the third image data having a third exposure time, wherein the third exposure time is substantially equal to a sum of the first exposure time and the second exposure time; and

generating processed image data based, at least in part, on the first image data, the second image data and the third image data.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 7, 2017
From: GUPTA, MOHIT; MITSUNAGA, TOMOO; ISO, DAISUKE; NAYAR, SHREE K.
To: SONY CORPORATION
Reel/Frame 043517/0024 →
Continuity (3)
Continuation 14652397
Provisional Application 61738193 · Dec 17, 2012
Related Publication 20170230562A1 · Aug 10, 2017
Cited By (1)
US 12,578,178