IP Library Granted Patent US 10,438,322
Granted Patent B2
US 10,438,322 · App. 15/607,321 · Granted Oct 8, 2019

Image resolution enhancement

Inventors: Moshe R. Lutz (Bellevue, WA); Vivek Pradeep (Redmond, WA)
Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC
G06T3/4046G06N3/04G06N3/08H04M1/0264H04N1/2112H04N7/18G06T2207/10004G06T2207/20084G06T2207/20212H04N5/23203
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,438,322
App. No.
15/607,321
Granted
Oct 8, 2019
Kind
B2
Abstract

Resolution enhancement techniques are described. An apparatus may receive first image data at a first resolution, and second image data at a resolution less than the first resolution. The second image data may be scaled to the first resolution and compared to the first image data. Application of a neural network may scale the first image data to a resolution higher than the first resolution. The application of the neural network may incorporate signals based on the scaled second image data. The signals may include information obtained by comparing the scaled second image data to the resolution of the first image data.

Claims (40)

1. An apparatus, comprising:

a first camera configured to obtain first image data of a first resolution;

operatively coupled to the first camera, an image processing system including a resolution-expanding neural network configured to

scale the first image data to a resolution greater than the first resolution, wherein the neural network is trained by using the neural network to scale, to the first resolution, second image data acquired by a second camera at a second resolution lower than the first resolution, with comparison of the first image data versus the second image data scaled to the first resolution and subsequent refinement of neural-network parameters.

2. The apparatus of claim 1 , wherein the first image data and the second image data are each scaled by an equivalent scaling factor.

3. The apparatus of claim 1 , wherein the first camera has a maximum resolution greater than a maximum resolution of the second camera.

4. The apparatus of claim 1 , wherein the image-processing system is configured to:

apply the neural network to the second image data;

calculate an error value based on a comparison of the scaled second image data to the first image data; and

train the neural network based on the error value.

5. The apparatus of claim 1 , wherein the neural-network parameters include a plurality of weights associated with the neural network.

6. The apparatus of claim 1 ,

wherein the first image data and the second image data are acquired contemporaneously.

7. The apparatus of claim 1 , wherein the first image data and the second image data correspond to a common region of an image frame acquired by each of the first and second cameras contemporaneously.

8. A method comprising:

receiving first image data from a first camera at a first resolution; and

scaling the first image data to a resolution higher than the first resolution by application of a non-linear function to the first image data, wherein one or more coefficients of the non-linear function are selected by using the non-linear function to scale, to the first resolution, second image data of a second resolution lower than the first resolution, by comparison of the first image data versus the second image data scaled to the first resolution and subsequent refinement of the one or more coefficients.

9. The method of claim 8 , wherein each of the first image data and the second image data is scaled by a common scaling factor.

10. The method of claim 8 , further comprising:

receiving the second image data from a second camera, wherein the second camera has a maximum resolution less than a maximum resolution of the first camera.

11. The method of claim 8 , further comprising:

scaling the second image data by applying the non-linear function to the second image data, unsealed from the second camera.

12. The method of claim 8 , wherein the non-linear function is a first non-linear function, the method further comprising:

scaling the second image data by applying a second non-linear function trained independently of the first non-linear function.

13. The method of claim 8 , wherein the second image data is obtained from a second camera.

14. The method of claim 13 , wherein the second image data is obtained from the first camera.

15. An apparatus, comprising:

a first camera;

a second camera;

at least one processor; and

computer memory having stored thereon instructions that, when executed by the at least one processor, cause the apparatus to:

obtain first image data from the first camera, the first image data having a first resolution;

obtain second image data from the second camera, the second image data having a second resolution, lower than the first resolution;

use a resolution-expanding neural network to scale the second image data to the first resolution;

train the resolution-expanding neural network based on a comparison of the first image data versus the second image data scaled to the first resolution; and

use the trained resolution-expanding neural network to scale the first image data to a resolution higher than the first resolution.

16. The apparatus of claim 15 , wherein the first image data and the second image data are each scaled by a common scaling factor.

17. The apparatus of claim 15 , wherein the first camera has a maximum resolution greater than the maximum resolution of the second camera.

18. The apparatus of claim 15 , wherein the first image data and the second image data are acquired contemporaneously.

19. The apparatus of claim 15 , wherein the first image data and the second image data correspond to a common region of an image frame acquired by each of the first and second cameras contemporaneously.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 26, 2017
From: LUTZ, MOSHE R.; PRADEEP, VIVEK
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 042520/0358 →
Continuity (1)
Related Publication 20180342044A1 · Nov 29, 2018