IP Library Granted Patent US 10,467,732
Granted Patent B2
US 10,467,732 · App. 15/827,120 · Granted Nov 5, 2019

Efficient storage for production of high dynamic range images

Inventor: Andrew Dewhurst (Cambridge, GB)
Assignee: Apical Limited
G06T5/007G06T5/40G06T5/50H04N5/2355G06T2207/10144G06T2207/20208
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,467,732
App. No.
15/827,120
Granted
Nov 5, 2019
Kind
B2
Abstract

Methods, apparatus and computer programs are provided for storing a set of sensor output images for multi-exposure high dynamic range processing to produce a processed image. The multi-exposure high dynamic range processing combines pixel values from one or more of the sensor output images to produce the processed image. In one example, a method comprises receiving from an image sensor a first sensor output image. The first sensor output image has a first exposure. The method also comprises receiving from the image sensor a second sensor output image. The second sensor output image has a second exposure different than the first exposure. First image pixels of the first sensor output image having pixel values which are unlikely to have a substantial contribution to the processed image are identified, and the amount of information stored in relation to the first image pixels is reduced on the basis of the identifying step.

Claims (51)

1. A method for storing a set of sensor output images for multi-exposure high dynamic range processing to produce a processed image, wherein the multi-exposure high dynamic range processing combines pixel values from one or more of the sensor output images to produce the processed image, the method comprising:

receiving, from an image sensor, a first sensor output image, wherein the first sensor output image has a first exposure;

receiving, from the image sensor, a second sensor output image, wherein the second sensor output image has a second exposure different than the first exposure;

identifying first image pixels of the first sensor output image having pixel values which are unlikely to have a substantial contribution to the processed image; and

reducing an amount of information stored in relation to the first image pixels on the basis of the identifying step.

2. The method according to claim 1 , comprising:

identifying second image pixels of the second sensor output image having pixel values which are unlikely to have a substantial contribution to the processed image; and

reducing the amount of information stored in relation to the second image pixels on the basis of the identifying step.

3. The method according to claim 1 , comprising processing the first and second images to produce a multi-exposure high dynamic range image, wherein the processing excludes pixels identified as having pixel values which are unlikely to have a substantial contribution to the processed image.

4. The method according to claim 1 , wherein the second exposure is smaller than the first exposure and wherein the first image pixels are identified on the basis of having a pixel value which is more than or equal to a predetermined threshold.

5. The method according to claim 1 , wherein the second exposure is greater than the first exposure and wherein the first image pixels are identified on the basis of having a pixel value which is less than or equal to a predetermined threshold.

6. The method according to claim 4 , comprising:

receiving from the image sensor a third image, wherein the third image has a third exposure smaller than the first exposure and greater than the second exposure; and

identifying as excluded from the third image pixels of the third image having pixel value between two predetermined thresholds.

7. The method according to claim 4 , wherein at least one of the thresholds is selected based on a noise profile of the image sensor.

8. The method according to claim 6 , wherein:

at least one of the thresholds is based on a ratio of the first exposure to the third exposure; or

at least one of the thresholds is based on a ratio of the third exposure to the second exposure; or

at least one of the thresholds is based on a ratio of the first exposure to the second exposure.

9. The method according to claim 1 , wherein reducing the amount of information stored in relation to the first image pixels comprises representing the first image pixels using a run-length encoding algorithm.

10. The method according to claim 4 , wherein the predetermined threshold comprises a maximum pixel value of the first image.

11. The method according to claim 5 , wherein the predetermined threshold comprises a minimum pixel value of the second image.

12. An apparatus for storing a set of sensor output images for multi-exposure high dynamic range processing to produce a processed image, wherein the multi-exposure high dynamic range processing combines pixel values from one or more of the sensor output images to produce the processed image, the apparatus comprising:

an image sensor;

a first encoder configured to:

receive from the image sensor a first sensor output image, wherein the first sensor output image has a first exposure; and

produce a modified version of the first sensor output image, wherein producing the modified version of the first image comprises:

identifying first image pixels of the first sensor output image having pixel values which are unlikely to have a substantial contribution to the processed image, and

reducing an amount of information stored in relation to the first image pixels on the basis of the identifying step;

a first buffer storage configured to receive and store the modified version of the first image;

a second encoder configured to:

receive from the image sensor a second sensor output image, wherein the second sensor output image has a second exposure different than the first exposure; and

produce a modified version of the second sensor output image, wherein producing the modified version of the second image comprises:

identifying second image pixels of the second sensor output image having pixel values which are unlikely to have a substantial contribution to the processed image; and

reducing an amount of information stored in relation to the second image pixels on the basis of the identifying step of the second image pixels, and

a second buffer storage configured to receive and store the modified version of the second image.

13. The apparatus according to claim 12 , comprising a high dynamic range image constructor configured to:

receive the modified version of the first sensor output image from the first buffer;

receive the modified version of the second sensor output image from the second buffer; and

process the modified version of the first sensor output image and the modified version of the second sensor output image to produce the processed image, wherein the processing excludes pixels identified as having pixel values which are unlikely to have a substantial contribution to the processed image.

14. The apparatus according to claim 12 , wherein the first buffer stores the modified version of the first sensor output image as a first compressed image and/or the second buffer stores the modified version of the second sensor output image as a second compressed image.

15. The apparatus according to claim 14 , wherein storing the first and/or second compressed images comprises using a run-length encoding algorithm to indicate pixels excluded from the modified versions of the first and/or second images.

16. The apparatus according to claim 12 , wherein the first and second buffers are configured to simultaneously transmit the modified versions of the first and second sensor output images to a high dynamic range image constructor.

17. A non-transitory, computer-readable storage medium comprising a set of computer-readable instructions stored thereon which, when executed by at least one processor, cause the at least one processor to:

receive, from an image sensor, a first sensor output image, wherein the first sensor output image has a first exposure;

identify, as excluded from the first sensor output image, pixels of the first sensor output image having pixel value equal to or greater than a first threshold, the first threshold corresponding to a first image intensity;

receive, from the image sensor, a second sensor output image, wherein the second sensor output image has a second exposure smaller than the first exposure;

identify, as excluded from the second sensor output image, pixels of the second sensor output image having pixel value less than or equal to a second threshold, the second threshold corresponding to a second image intensity lower than the first image intensity;

receive, from the image sensor, a third sensor output image, wherein the third sensor output image has a third exposure smaller than the first exposure and greater than the second exposure;

identify, as excluded from the third sensor output image, pixels of the third sensor output image having pixel value greater than or equal to a third threshold and less than or equal to a fourth threshold, wherein the third threshold corresponds to a third image intensity lower than the first image intensity and the fourth threshold corresponds to a fourth image intensity higher than the second image intensity; and

process the first, second and third sensor output images to produce a multi-exposure high dynamic range image, wherein the processing excludes pixels identified as excluded from the first sensor output image, excludes pixels identified as excluded from the second sensor output image and excludes pixels identified as excluded from the third sensor output image.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 23, 2022
From: APICAL LIMITED
To: ARM LIMITED
Reel/Frame 061300/0471 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 26, 2019
From: DEWHURST, ANDREW
To: APICAL LIMITED
Reel/Frame 049600/0231 →
Priority Claims (1)
GB 1509384.2 · Jun 1, 2015 · national
Continuity (2)
Continuation PCTEP2016062432 · Jun 1, 2016
Related Publication 20180082408A1 · Mar 22, 2018