IP Library Granted Patent US 10,616,482
Granted Patent B2
US 10,616,482 · App. 15/455,446 · Granted Apr 7, 2020

Image quality assessment

Inventors: Antoine Meler (Chapareillan, FR); Bruno Cesar Douady-Pleven (Bures-sur-Yvette, FR)
Assignee: GoPro, Inc.
H04N5/23238G06T3/4038G06T7/13G06T7/70H04N5/247H04N5/265G06T2207/20081
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,616,482
App. No.
15/455,446
Granted
Apr 7, 2020
Kind
B2
Abstract

Systems and methods are disclosed for image signal processing. For example, methods may include receiving a first image from a first image sensor; receiving a second image from a second image sensor; stitching the first image and the second image to obtain a stitched image; identifying an image portion of the stitched image that is positioned on a stitching boundary of the stitched image; and inputting the image portion to a machine learning module to obtain a score, wherein the machine learning module has been trained using training data that included image portions labeled to reflect an absence of stitching and image portions labeled to reflect a presence of stitching, wherein the image portions labeled to reflect a presence of stitching included stitching.

Claims (53)

1. A system comprising:

a first image sensor configured to capture a first image;

a second image sensor configured to capture a second image; and

a processing apparatus that is configured to:

receive the first image from the first image sensor;

receive the second image from the second image sensor;

stitch the first image and the second image to obtain a stitched image;

identify an image portion of the stitched image that is positioned on a stitching boundary of the stitched image;

input the image portion to a machine learning module to obtain a score, wherein the machine learning module has been trained using training data that included image portions labeled to reflect an absence of stitching and image portions labeled to reflect a presence of stitching, wherein the image portions labeled to reflect a presence of stitching included stitching boundaries of stitched images;

select a parameter of a stitching algorithm based at least in part on the score;

determining whether to re-stitch based on the score and a threshold;

responsive to determination, re-stitching, using the parameter, the first image and the second image to obtain a composite image; and

store, display, or transmit an output image based on the composite image.

2. The system of claim 1 , in which the processing apparatus is configured to:

identify one or more additional image portions within the stitched image that occur along the stitching boundary of the stitched image;

input the one or more additional image portions to the machine learning module to obtain one or more additional scores; and

select the parameter based on the one or more additional scores.

3. The system of claim 1 , in which the machine learning module comprises a feature extraction submodule that is configured to determine features based on the image portion.

4. The system of claim 1 , in which stitching to obtain the stitched image is performed such that individual pixels of the stitched image are respectively based on either the first image or the second image, but not both; and

in which stitching to obtain the composite image is performed such that at least one pixel of the composite image is based on both the first image and the second image.

5. The system of claim 1 , in which stitching to obtain the stitched image is performed such that individual pixels of the stitched image are respectively based on either the first image or the second image, but not both; and

in which the image portions of the training data labeled to reflect a presence of stitching included stitching boundaries of stitched images that were stitched without blending.

6. The system of claim 1 , in which the parameter specifies whether one dimensional parallax correction or two dimensional parallax correction will be applied to stich the first image and the second image.

7. The system of claim 1 , in which the image portion is a block of pixels from the stitched image that includes pixels on both sides of the stitching boundary, where the block of pixels has a resolution less than the resolution of the first image.

8. The system of claim 1 , in which the machine learning module includes a convolutional neural network.

9. The system of claim 1 , in which the machine learning module includes a neural network and the image portion is a block of pixels from the stitched image that includes pixels on both sides of the stitching boundary and all pixel values from the block of pixels are input to a first layer of the neural network.

10. A method comprising:

receiving a first image from a first image sensor;

receiving a second image from a second image sensor;

stitching the first image and the second image to obtain a stitched image;

identifying an image portion of the stitched image that is positioned on a stitching boundary of the stitched image; and

inputting the image portion to a machine learning module to obtain a score, wherein the machine learning module has been trained using training data that included image portions labeled to reflect an absence of stitching and image portions labeled to reflect a presence of stitching, wherein the image portions labeled to reflect a presence of stitching included stitching boundaries of stitched images;

select a parameter of a stitching algorithm based at least in part of the score;

determining whether to re-stitch based on the score and a threshold;

responsive to determination, re-stitching, using the parameter, the first image and the second image to obtain a composite image;

and store, display, or transmit an output image based on the composite image.

11. The method of claim 10 , comprising:

storing, displaying, or transmitting the score or a composite score based in part on the score.

12. The method of claim 10 , comprising:

identifying one or more additional image portions within the stitched image that occur along the stitching boundary of the stitched image;

inputting the one or more additional image portions to the machine learning module to obtain one or more additional scores; and

generating a histogram of the score and the one or more additional scores.

13. The method of claim 10 , comprising:

training the machine learning module, wherein the training data includes image portions detected with a single image sensor that are labeled to reflect an absence of stitching.

14. The method of claim 10 , comprising:

training the machine learning module, wherein the training data includes image portions labeled with subjective scores provided by humans for images from which the image portions are taken.

15. The method of claim 10 , comprising:

selecting a parameter of a stitching algorithm based on the score.

16. The method of claim 10 , in which the image portion is a block of pixels from the stitched image that includes pixels on both sides of the stitching boundary, where the block of pixels has a resolution less than the resolution of the first image.

17. The method of claim 10 , in which the machine learning module includes a neural network that receives pixel values from pixels in the image portion and outputs the score.

18. The method of claim 10 , comprising:

obtaining a plurality of scores from the machine learning module for a plurality of image portions from along the stitching boundary of the stitched image; and

determining a composite score for the stitched image based on the plurality of scores.

Assignments (7)
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: WELLS FARGO BANK, NATIONAL ASSOCIATION, AS AGENT
Reel/Frame 072358/0001 →
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: FARALLON CAPITAL MANAGEMENT, L.L.C., AS AGENT
Reel/Frame 072340/0676 →
RELEASE OF PATENT SECURITY INTEREST Recorded Jan 25, 2021
From: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
To: GOPRO, INC.
Reel/Frame 055106/0434 →
SECURITY INTEREST Recorded Oct 19, 2020
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 054113/0594 →
CORRECTIVE ASSIGNMENT TO CORRECT THE SCHEDULE TO REMOVE APPLICATION 15387383 AND REPLACE WITH 15385383 PREVIOUSLY RECORDED ON REEL 042665 FRAME 0065. ASSIGNOR(S) HEREBY CONFIRMS THE SECURITY INTEREST. Recorded Oct 23, 2019
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 050808/0824 →
SECURITY INTEREST Recorded Jun 1, 2017
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 042665/0065 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 10, 2017
From: MELER, ANTOINE; DOUADY-PLEVEN, BRUNO CESAR
To: GOPRO, INC.
Reel/Frame 041538/0378 →
Continuity (1)
Related Publication 20180262683A1 · Sep 13, 2018
Cited By (1)
US 12,718,318