IP Library › Granted Patent US 12,536,616
Granted Patent B2
US 12,536,616 · App. 17/768,853 · Granted Jan 27, 2026

Image processing device and image processing method

Inventors: Piergiorgio Sartor (Stuttgart, DE); Alexander Gatto (Stuttgart, DE); Takeshi Uemori (Stuttgart, DE); Zoltan Facius (Stuttgart, DE); Vincent Parret (Stuttgart, DE); Ralf Müller (Stuttgart, DE)
Assignee: Sony Group Corporation
G06T5/50G06T2207/10036G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,536,616
App. No.
17/768,853
Granted
Jan 27, 2026
Kind
B2
Abstract

An image processing device has circuitry, which is configured to obtain input image data being represented by a number of color channels and to input the input image data into a neural network for generating output multispectral image data, wherein the neural network is configured to generate at least first and second multispectral image data on the basis of the input image data, wherein a number of spectral channels of the second multispectral image data is larger than the number of spectral channels of the first multispectral image data.

Claims (26)

1 . An image processing device comprising circuitry configured to:

obtain input image data being represented by a number of color channels;

input the input image data into a neural network for generating output multispectral image data, wherein

the neural network is configured to generate first multispectral image data having a first number of spectral channels and second multispectral image data having a second number of spectral channels greater than the first number, both the first and second multispectral image data generated by the neural network from the input image data.

2 . The image processing device of claim 1 , wherein the circuitry is further configured to obtain the first or the second multispectral image data as the output multispectral image data.

3 . The image processing device of claim 1 , wherein the input image data include spectral image data.

4 . The image processing device of claim 3 , wherein the number of spectral channels of the output multispectral image data is larger than the number of spectral channels of the input image data.

5 . The image processing device of claim 1 , wherein a spatial resolution of the first multispectral image data is higher than a spatial resolution of the second multispectral image data.

6 . The image processing device of claim 5 , wherein the output multispectral image data is generated based on a predetermined relationship between the spatial resolution and the number of spectral channels.

7 . The image processing device of claim 1 , wherein the neural network is a convolutional neural network.

8 . The image processing device of claim 7 , wherein the convolutional neural network is trained to generate the first multispectral image data from the input image data and the second multispectral image data from the first multispectral image data.

9 . The image processing device of claim 7 , wherein the convolutional neural network is trained based on RGB image data and on multispectral image data.

10 . The image processing device of claim 1 , wherein the circuitry is further configured to perform object recognition.

11 . An image processing method comprising:

obtaining input image data being represented by a number of color channels;

inputting the input image data into a neural network for generating output multispectral image data, wherein

the neural network is configured to generate first multispectral image data having a first number of spectral channels and second multispectral image data having a second number of spectral channels greater than the first number, both the first and second multispectral image data generated by the neural network from the input image data.

12 . The image processing method of claim 11 , further comprising obtaining the first or the second multispectral image data as the output multispectral image data.

13 . The image processing method of claim 11 , wherein the input image data include spectral image data.

14 . The image processing method of claim 13 , wherein the number of spectral channels of the output multispectral image data is larger than the number of spectral channels of the input image data.

15 . The image processing method of claim 11 , wherein a spatial resolution of the first multispectral image data is higher than a spatial resolution of the second multispectral image data.

16 . The image processing method of claim 15 , wherein the output multispectral image data is generated based on a predetermined relationship between the spatial resolution and the number of spectral channels.

17 . The image processing method of claim 11 , wherein the neural network is a convolutional neural network.

18 . The image processing device of claim 17 , wherein the convolutional neural network is trained to generate the first multispectral image data from the input image data and the second multispectral image data from the first multispectral image data.

19 . The image processing method of claim 17 , wherein the convolutional neural network is trained based on RGB image data and on multispectral image data.

20 . The image processing method of claim 11 , further comprising performing object recognition.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 14, 2023
From: SARTOR, PIERGIORGIO; GATTO, ALEXANDER; UEMORI, TAKESHI; FACIUS, ZOLTAN; PARRET, VINCENT; MUELLER, RALF
To: SONY GROUP CORPORATION
Reel/Frame 064575/0670 →
Priority Claims (1)
EP 19204783 · Oct 23, 2019 · regional
Continuity (1)
Related Publication 20240303773A1 · Sep 12, 2024
References Cited (15)
US 11354804B1 · Behrooz · 2022 [cited by examiner]
US 20190096049A1 · Kim et al. · 2019 [cited by applicant]
US 20200302249A1 · Liu · 2020 [cited by examiner]
US 20210250526A1 · Germain · 2021 [cited by examiner]
US 20210350590A1 · Kim · 2021 [cited by examiner]
CN 106485688A · 2017 [cited by applicant]
CN 108830796A · 2018 [cited by applicant]
Yang et al., “Hyperspectral and Multispectral Image Fusion via Deep Two-Branches Convolutional Neural Network,” May 21, 2018 (Year: 2018). [cited by examiner]
International Search Report and Written Opinion mailed on Dec. 14, 2020, received for PCT Application PCT/EP2020/079090, Filed on Oct. 15, 2020, 10 pages. [cited by applicant]
Han et al: “Reconstruction From Multispectral to Hyperspectral Image Using Spectral Library-Based Dictionary Learning”, IEEE Transactions on Geoscience and Remote Sensing, vol. 57, No. 3, Mar. 2019, pp. 1325-1335. [cited by applicant]
Miao et al: “lambda-Net: Reconstruct Hyperspectral Images From a Snapshot Measurement”, 2019 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, Oct. 27, 2019, pp. 4058-4068. [cited by applicant]
Can et al., “An Efficient CNN for Spectral Reconstruction from RGB Images”, arxiv.org, Cornell University Library, Apr. 12, 2018, 5 pages. [cited by applicant]
Galliani et al., “Learned Spectral Super-Resolution”, arXiv:1703.09470v1, Mar. 28, 2017, 10 pages. [cited by applicant]
Mei et al., “Hyperspectral Image Spatial Super-Resolution via 3D Full Convolutional Neural Network”, Remote Sensing, vol. 9, Available Online at: https://www.researchgate.net/publication/320913192_Hyperspectral_Image_Sp… [cited by applicant]
Li et al., “Hyperspectral image super-resolution using deep convolutional neural network”, Journal of Latex Templates, Neurocomputing, Available Online at: https://www.researchgate.net/publication/317024713_Hyperspectra… [cited by applicant]