Granted Patent B2

US 12,267,604 · App. 17/831,956 · Granted Apr 1, 2025

Inventors: Woo-Shik Kim (Seongnam-si, KR); Younggeun Roh (Seoul, KR); Inhwan Ryu (Suwon-si, KR)

Assignee: SAMSUNG ELECTRONICS CO., LTD.

H04N25/13H04N9/73H04N23/45H04N23/88

Abstract

Claims (79)

1. An image acquisition apparatus comprising:

a first image sensor configured to acquire a first image based on a first wavelength band;

a second image sensor having more channels than the first image sensor and configured to acquire a second image based on a second wavelength band comprising the first wavelength band; and

a processor configured to:

register the first image and the second image, which are respectively output from the first image sensor and the second image sensor;

obtain a registration image based on the first image and the second image; and

perform color conversion on the registration image by using an illumination value estimated from the second image,

wherein the processor is further configured to:

divide the first image into one or more first regions and divide the second image into one or more second regions respectively corresponding to the one or more first regions;

estimate a respective illumination value for each of the one or more second regions;

perform the color conversion on each of the one or more first regions by using the estimated illumination values, and

when a difference between illumination values respectively estimated for adjacent second regions of the one or more second regions is greater than or equal to a first threshold value, adjust any one of the illumination values of the adjacent second regions to adjust the difference to be less than the first threshold value.

2. The image acquisition apparatus of claim 1 ,

wherein the processor is further configured to, after performing the color conversion, perform post-processing on a boundary portion between the adjacent second regions.

3. The image acquisition apparatus of claim 1 ,

wherein the processor is further configured to calculate a parameter for registering the first image and the second image based on at least one of a resolution, a field of view, and a focal length of each of the first image sensor and the second image sensor.

4. The image acquisition apparatus of claim 1 ,

wherein the processor is further configured to estimate the illumination value by using spectral information obtained from a plurality of channels output from the second image sensor.

5. The image acquisition apparatus of claim 1 ,

wherein the processor is further configured to estimate the illumination value by using a neural network trained on a plurality of second images associated with predetermined illumination values.

6. The image acquisition apparatus of claim 1 ,

wherein the processor is further configured to register the first image and the second image by respectively extracting first features from the first image and second features from the second image and matching the first extracted features with the second extracted features.

7. The image acquisition apparatus of claim 1 ,

wherein the processor is further configured to register the first image and the second image in units of pixel groups, pixels, or sub-pixels of the first image and the second image.

8. The image acquisition apparatus of claim 1 ,

wherein the first image sensor comprises a first pixel array comprising:

a first sensor layer in which a plurality of first sensing elements are arrayed; and

a color filter on the first sensor layer and comprising red, green, and blue filters which are alternately arranged,

wherein the second image sensor comprises a second pixel array comprising:

a second sensor layer in which a plurality of second sensing elements are arrayed; and

a spectral filter on the second sensor layer and in which a filter group is repeatedly arranged, the filter group comprising a plurality of unit filters each having a different transmission wavelength band from each other unit filter in the filter group.

9. The image acquisition apparatus of claim 8 ,

wherein a combined transmission wavelength band of the plurality of unit filters of the filter group includes a visible light wavelength band and is wider than the visible light wavelength band, and

wherein the plurality of unit filters comprises 16 unit filters arranged in a 4×4 array.

10. The image acquisition apparatus of claim 8 ,

wherein the first pixel array and the second pixel array are horizontally apart from each other on a circuit board.

11. The image acquisition apparatus of claim 10 ,

wherein first circuit elements configured to process a signal from the first sensor layer, and second circuit elements configured to process a signal from the second sensor layer are provided on the circuit board.

12. The image acquisition apparatus of claim 11 , further comprising a timing controller configured to synchronize operations of the first circuit elements with operations of the second circuit elements.

13. The image acquisition apparatus of claim 10 , further comprising:

a first memory storing data about the first image; and

a second memory storing data about the second image.

14. The image acquisition apparatus of claim 13 ,

wherein the first memory and the second memory are provided inside the circuit board.

15. The image acquisition apparatus of claim 1 , further comprising:

a first imaging optical system configured to form a first optical image of an object on the first image sensor, the first imaging optical system comprising at least one first lens; and

a second imaging optical system configured to form a second optical image of the object on the second image sensor, the second imaging optical system comprising at least one second lens.

16. The image acquisition apparatus of claim 15 ,

wherein the first imaging optical system and the second imaging optical system have an identical focal length and an identical field of view.

17. An electronic apparatus comprising the image acquisition apparatus of claim 1 .

18. A method of controlling an image acquisition apparatus including a plurality of image sensors, the method comprising:

acquiring a first image and a second image from a first image sensor having a first wavelength band and a second image sensor having more channels than the first image sensor and having a second wavelength band comprising the first wavelength band, respectively;

registering the acquired first and second images to obtain a registration image; and

performing color conversion on the registration image by using an illumination value estimated from the second image,

wherein the performing color conversion further comprises:

dividing the first image into one or more first regions and dividing the second image into one or more second regions respectively corresponding to the one or more first regions;

estimating a respective illumination value for each of the one or more second regions;

performing the color conversion on each of the one or more first regions by using the estimated illumination values, and

when a difference between illumination values respectively estimated for adjacent second regions of the one or more second regions is greater than or equal to a first threshold value, adjusting any one of the illumination values of the adjacent second regions to adjust the difference to be less than the first threshold value.

19. An image acquisition apparatus comprising:

a first image sensor comprising a first filter and configured to acquire a first image based on a first wavelength band;

a second image sensor comprising a second filter, having more channels than the first image sensor, and configured to acquire a second image based on a second wavelength band comprising the first wavelength band; and

a processor configured to:

receive the first image of an object from the first image sensor and the second image of the object from the second image sensor; and

based on the received first image and the received second image, generate a white balanced first image of the object by eliminating an influence of illumination reflected by the object,

wherein the processor is further configured to:

divide the first image into one or more first regions and divide the second image into one or more second regions respectively corresponding to the one or more first regions;

estimate a respective illumination value for each of the one or more second regions;

perform the color conversion on each of the one or more first regions by using the estimated illumination values, and

when a difference between illumination values respectively estimated for adjacent second regions of the one or more second regions is greater than or equal to a first threshold value, adjust any one of the illumination values of the adjacent second regions to adjust the difference to be less than the first threshold value.

20. The image acquisition apparatus of claim 19 , wherein the first image sensor comprises a plurality of first pixels,

wherein the first filter comprises a plurality of first filter groups repeatedly arranged, each first filter group comprising a plurality of first unit filters each corresponding to a respective first pixel,

wherein the second image sensor comprises a plurality of second pixels,

wherein the second filter comprises a plurality of second filter groups repeatedly arranged, each second filter group comprising a plurality of second unit filters each corresponding to a respective second pixel,

wherein each first filter group of the first filter corresponds to a respective second filter group of the second filter, and

wherein a wavelength band of each first unit filter is larger than a wavelength band of each second unit filter.

21. The image acquisition apparatus of claim 19 , wherein the processor is further configured to:

generate a conversion matrix based on the second image for eliminating the influence of illumination reflected by the object; and

generate the white balanced first image based on the conversion matrix and the first image.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (35)