IP Library Granted Patent US 9,210,386
Granted Patent B2
US 9,210,386 · App. 13/518,687 · Granted Dec 8, 2015

Filter setup learning for binary sensor

Inventors: Tero P. Rissa (Siivikkala, FI); Tuomo Maki-Marttunen (Tampere, FI); Matti Viikinkoski (Tampere, FI)
Assignee: Nokia Technologies Oy
H04N9/045H04N5/2351H04N5/335
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,210,386
App. No.
13/518,687
Granted
Dec 8, 2015
Kind
B2
Abstract

The invention relates to forming an image using binary pixels. Binary pixels are pixels that have only two states, a white state when the pixel is exposed and a black state when the pixel is not exposed. The binary pixels have color filters on top of them, and the setup of color filters may be initially unknown. A setup making use of a statistical approach may be used to determine the color filter setup to produce correct output images. Subsequently, the color filter information may be used with the binary pixel array to produce images from the input images that the binary pixel array records.

Claims (65)

1. A method comprising:

receiving first binary pixel values, the first binary pixel values having been obtained by applying light of a first known color through an array of color filters onto an array of binary pixels, individual ones of said color filters being of unknown color;

receiving second binary pixel values, the second binary pixel values having been obtained by applying light of a second known color through said array of color filters onto said array of binary pixels;

receiving information on the colors of said light of said first and second known colors applied through said array of color filters onto said array of binary pixels;

forming an estimate of a color of a color filter associated with each of said binary pixels having binary pixel values indicating activation by said light of at least one of said known colors using said information on the colors of said light of said first and second known colors; and

storing estimates of the colors of said color filters in an image processing system for subsequent use in processing images.

2. The method according to claim 1 , further comprising:

exposing said binary pixels to said light of said first and second known colors through color filters superimposed on said binary pixels, said light having passed through an optical arrangement, and

forming said binary pixel values from the output of said binary pixels.

3. The method according to claim 1 , further comprising:

forming said estimate using likelihood estimation, and

refining said estimate iteratively.

4. The method according to claim 1 , further comprising:

determining neighborhoods of each of said binary pixels, and

using estimated values of binary pixels in said neighborhoods of each of said binary pixels as information in forming said estimate for each of said binary pixels.

5. The method according to claim 4 , further comprising:

forming said estimate by optimizing an energy function having a first component indicative of similarity of said color of said color filter and said color of said light of at least one of said known colors and a second component indicative of at least one difference in values of each of said binary pixels and a neighbor of each of said binary pixels.

6. The method according to claim 1 , wherein adjusting is performed in said image processing system to form initial values of the color filters of said binary pixels or to calibrate the values of the color filters of said binary pixels.

7. An apparatus comprising:

at least one processor, and

a memory including computer program code, the memory and the computer program code configured, with the at least one processor, to cause the apparatus to perform at least the following:

receive first binary pixel values, the first binary pixel values having been obtained by applying light of a first known color through an array of color filters onto an array of binary pixels, individual ones of said color filters being of unknown color;

receive second binary pixel values, the second binary pixel values having been obtained by applying light of a second known color through said array of color filters onto said array of binary pixels;

receive information on the colors of said light of said first and second known colors applied through said array of color filters onto said array of binary pixels;

form an estimate of a color of a color filter associated with each of said binary pixels having binary pixel values indicating activation by said light of at least one of said known color colors using said information on the color of said light; and

store estimates of the colors of said color filters in an image processing system for subsequent use in processing images.

8. The apparatus according to claim 7 , further comprising computer program code configured, with the processor, to cause the apparatus to perform at least the following:

expose said binary pixels to said light of said first and second known colors through color filters superimposed on said binary pixels, said light having passed through an optical arrangement, and

form said binary pixel values from the output of said binary pixels.

9. The apparatus according to claim 7 , further comprising computer program code configured, with the processor, to cause the apparatus to perform at least the following:

form said estimate using likelihood estimation, and

refine said estimate iteratively.

10. The apparatus according to claim 7 , further comprising computer program code configured, with the processor, to cause the apparatus to perform at least the following:

determine neighborhoods of each of said binary pixels, and

use estimated values of binary pixels in said neighborhoods of each of said binary pixels as information in forming said estimate for each of said binary pixels.

11. The apparatus according to claim 10 , further comprising computer program code configured, with the processor, to cause the apparatus to perform at least the following:

form said estimate by optimizing an energy function having a first component indicative of similarity of said color of said color filter and said color of said light of at least one of said known colors and a second component indicative of at least one difference in values of each of said binary pixels and a neighbor of each of said binary pixels.

12. The apparatus according to claim 7 , further comprising computer program code configured, with the processor, to cause the apparatus to perform at least the following:

adjusting said image processing system to form initial values of the color filters of said binary pixels or to calibrate the values of the color filters of said binary pixels.

13. The apparatus according to claim 7 , further comprising:

a color signal unit comprising at least one neural network, and a memory for storing parameters and/or weights of at least one said neural network.

14. The apparatus according to claim 7 , further comprising:

an optical arrangement for forming an image,

an array of binary pixels for detecting said image, and

groups of said binary pixels.

15. The apparatus according to claim 7 , further comprising:

at least one color filter superimposed on an array of binary pixels, said color filter being superimposed on said array of binary pixels in a manner that is at least one of the group of non-aligned, irregular, random, and unknown superimposition.

16. A system comprising:

at least one processor, and

a memory including computer program code, the memory and the computer program code configured, with the at least one processor, to cause the system to perform at least the following:

receive first binary pixel values, the first binary pixel values having been obtained by applying light of a first known color through an array of color filters onto an array of binary pixels, individual ones of said color filters being of unknown color;

receive second binary pixel values, the second binary pixel values having been obtained by applying light of a second known color through an array of color filters onto said array of binary pixels;

receive information on the colors of said light of said first and second known colors applied through said array of color filters onto said array of binary pixels;

form an estimate of a color of a color filter associated with each of said binary pixels having binary pixel values indicating activation by said light of at least one of said known color colors using said information on the colors of said light of said first and second known colors; and

store estimates of the colors of said color filters in an image processing system for subsequent use in processing images.

17. The system according to claim 16 , wherein the system further comprises:

an adjusting unit configured to receive said binary pixel values and said information on the colors of said light of said first and second known colors, and to form an estimate of a color of said color filter associated with each of said binary pixels by using said binary pixel values and said information on the colors of said light of said first and second known colors, and

an image processing unit comprising a binary pixel array with color filters and configured to form images, wherein said image processing unit is configured to receive adjustment information from said adjusting unit for adapting said image processing unit.

18. A computer program product stored on a non-transitory computer readable storage medium and executable in a data processing device, wherein the computer program product comprises:

a computer program code section for receiving first binary pixel values, the first binary pixel values having been obtained by applying light of a first known color through an array of color filters onto an array of binary pixels, individual ones of said color filters being of unknown color, and for receiving second binary pixel values, the second binary pixel values having been obtained by applying light of a second known color through said array of color filters onto said array of binary pixels;

a computer program code section for receiving information on the colors of said light of said first and second known colors applied through said array of color filters onto said array of binary pixels;

a computer program code section for forming an estimate of a color of a color filter associated with each of said binary pixels having binary pixel values indicating activation by said light of at least one of said known colors using said information on the colors of said light of said first and second known colors; and

a computer program code section for storing estimates of the colors of said color filters in an image processing system for subsequent use in processing images.

19. The computer program product according to claim 18 , wherein the computer program product further comprises:

a computer program code section for adjusting said image processing system to form initial values of the color filters of said binary pixels or to calibrate the values of the color filters of said binary pixels.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 8, 2023
From: NOKIA TECHNOLOGIES OY
To: PIECE FUTURE PTE LTD
Reel/Frame 063889/0841 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 27, 2015
From: NOKIA CORPORATION
To: NOKIA TECHNOLOGIES OY
Reel/Frame 035501/0269 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 21, 2012
From: RISSA, TERO; MAKI-MARTTUNEN, TUOMO; VIIKINKOSKI, MATTI
To: NOKIA CORPORATION
Reel/Frame 029057/0156 →
Continuity (1)
Related Publication 20130011050A1 · Jan 10, 2013