IP Library Granted Patent US 11,238,564
Granted Patent B2
US 11,238,564 · App. 16/768,870 · Granted Feb 1, 2022

Temporal de-noising

Inventors: James A. McCombe (San Francisco, CA); Christoph Birkhold (San Francisco, CA)
Assignee: MINE ONE GMBH
G06T5/002G06T5/50G06T7/593G06T2207/10012G06T2207/10024G06T2207/20182G06T2207/20221
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,238,564
App. No.
16/768,870
Granted
Feb 1, 2022
Kind
B2
Abstract

Methods, systems, devices and computer software/program code products enable generating reduced-noise image frames based on image frames received from a digital, camera pipeline; and enable efficient stereo image search between corresponding images generated by at least two cameras.

Claims (48)

1. A method of generating reduced-noise image frames based on image frames received from a digital camera pipeline, the method comprising:

in a digital processing resource comprising at least one digital processor:

A. receiving at least one frame of image pixel data from a digital camera pipeline;

B. receiving image metadata corresponding, to the at least one received frame; and

C. executing temporal de-noising operations for at least selected pixels of the at least one received frame of image pixel data, wherein executing temporal de-noising operations comprises;

(1) evaluating a noise model to estimate the noise present in a selected pixel from the at least one received frame;

(2) blending data from a persistent image buffer with data from the at least one received frame;

(3) updating data in the persistent image buffer; and

(4) outputting the blended pixel data to a reduced-noise frame buffer, thereby to generate a reduced-noise image frame.

2. The method of claim 1 further comprising:

A. configuring at least two digital processing resources, each comprising at least one digital processor, to function as temporal de-noising digital processing resources, each operable to receive image pixel data, receive image metadata, execute temporal de-noising operations and receive image frames from a respective digital camera;

B. utilizing at least one digital memory element to receive output images from at least one temporal de-noising digital processing resource; and

C. utilizing a stereo correspondence module to compare an output image from a first temporal de-noising digital processing resource with a tentatively corresponding output image from a second temporal de-noising digital processing resource, to execute stereo image search between corresponding images generated by at least two digital cameras.

3. The method of claim 2 wherein the temporal de-noising digital processing resources comprise processing modules, and the processing modules comprise software modules executing on shared digital processing resources.

4. The method of claim 2 wherein the temporal de-noising digital processing resources comprise dedicated electronic components.

5. The method of claim 1 wherein the noise model comprises a simulation of the inverse of the behavior of any of: (1) at least selected aspects of the digital camera pipeline, (2) a variable gain amplifier, (3) automatic white-balance logic, or (4) automatic exposure control.

6. The method of claim 1 wherein the noise model comprises a simulation of the number of electrons in an electron well associated with a light sensing element of the digital camera pipeline.

7. The method of claim 1 wherein the persistent image buffer comprises a summation of data from prior frames.

8. The method of claim 1 wherein the persistent image buffer comprises an average of data from prior frames.

9. The method of claim 1 wherein at least one pixel in the persistent image buffer is invalidated in response to a selected result of the evaluation of the noise model.

10. The method of claim 1 further comprising: utilizing a persistent data store to retain information representing the number of valid samples.

11. The method of claim 1 further comprising: utilizing a persistent data store to retain information estimating the number of electrons or photons that have been integrated into corresponding data.

12. The method of claim 1 further comprising: utilizing information from the noise model to evaluate pixel data to quantify the likelihood that a measured variation in the pixel data is due to noise rather than a change in observed situation.

13. The method of claim 1 Further comprising: applying an output from the noise model to a video signal, to improve suitability of the signal as an input to a stereo disparity search operation.

14. The method of claim 1 in which at least one pixel in the persistent image buffer is invalidated in response to a selected result of a comparison with a pixel from the at least one received frame.

15. The method of claim 2 wherein at least two digital cameras participate in a shutter synchronization protocol.

16. The method of claim 1 wherein the noise model is evaluate in a manner dependent on the color represented by image pixel data.

17. A system for generating reduced-noise image frames based on image frames received from a digital camera pipeline, the system comprising:

a digital processing resource comprising at least one digital processor, the digital processing resource being operable to:

A. receive at least one frame of image pixel data from a digital camera pipeline;

B. receive image metadata corresponding to the at least one received frame; and

C. execute temporal de-noising operations for at least selected pixels of the at least one received frame of image pixel data, wherein executing temporal de-noising operations comprises:

(1) evaluating a noise model to estimate the noise present in a selected pixel from the at least one received frame;

(2) blending data from a persistent image buffer with data from the at least one received frame;

(3) updating data in the persistent image buffer; and

(4) outputting the blended pixel data to a reduced-noise frame buffer, thereby to generate a reduced-noise image frame.

18. The system of claim 17 , further comprising:

A. at least first and second digital processing resources, each operable to receive image pixel data, receive image metadata, execute temporal de-noising operations, and receive image frames from a respective digital camera, and each constituting a temporal de-noising digital processing resource;

B. at least one digital memory element operable to receive output images from at least one temporal de-noising digital processing resource; and

C. a stereo correspondence module operable to compare an output image from a first temporal de-noising digital processing resource with a tentatively corresponding output image from a second temporal dc-noising digital processing resource, to execute stereo image search between corresponding images generated by at least two digital cameras.

19. A program product for use with a digital processing system to enable the digital processing system to generate reduced-noise image frames based on image frames received from a digital camera pipeline, the digital processing system comprising a digital processing resource, the digital processing resource comprising at least one digital processor, the program product comprising digital processor-executable program instructions stored on a non-transitory digital processor-readable medium, which when executed in the digital processing resource cause the digital processing resource to:

A. receive at least one frame of image pixel data from a digital camera pipeline;

B. receive image metadata corresponding, to the at least one received frame; and

C. execute temporal de-noising operations for at least selected pixels of the at least one received frame of image pixel data, wherein executing temporal de-noising operations comprises:

(1) evaluating a noise model to estimate the noise present in a selected pixel from the at least one received frame;

(2) blending data from a persistent image buffer with data from the at least one received frame;

(3) updating data in the persistent image buffer; and

(4) outputting the blended pixel data to a reduced-noise frame buffer, thereby to generate a reduced-noise image flame.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 20, 2021
From: MCCOMBE, JAMES A.; BIRKHOLD, CHRISTOPH
To: MINE ONE GMBH
Reel/Frame 058432/0587 →
Continuity (5)
Continuation In Part PCTUS2016023433 · Mar 21, 2016
Continuation In Part 15560019
Provisional Application 62593920 · Dec 2, 2017
Provisional Application 62136494 · Mar 21, 2015
Related Publication 20210241427A1 · Aug 5, 2021
Cited By (4)
US 12,322,071 US 12,560,995 US 12,566,491 US 12,652,470