IP Library Granted Patent US 10,970,552
Granted Patent B2
US 10,970,552 · App. 16/142,608 · Granted Apr 6, 2021

Scene classification for image processing

Inventors: Sylvain Leroy (Bures-sur-Yvette, FR); Adrien Cariou (Paris, FR); Miguel Marques (Bois d'Arcy, FR); Julien Caron (Crepy-en-Valois, FR); Bruno Faure (Fresnes, FR); Stéphane Belardi (Meudon, FR)
Assignee: GoPro, Inc.
G06K9/00684G06K9/4652H04N9/735
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,970,552
App. No.
16/142,608
Granted
Apr 6, 2021
Kind
B2
Abstract

Image analysis includes obtaining, from an image signal processor, image processing information corresponding to a previously processed image, obtaining scene classification information for an input image based on the image processing information, generating a processed image by processing the input image based on the scene classification information, and outputting the processed image. The image processing information includes automatic white balance correction information and obtaining the scene classification information includes obtaining the scene classification information based on the automatic white balance correction information.

Claims (67)

1. A non-transitory computer-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, the operations including operations to:

generate, by an image signal processor, a previously processed image, wherein generating the previously processed image includes generating corresponding image processing information;

obtain, from the image signal processor, the image processing information corresponding to the previously processed image;

obtain scene classification information for an input image based on the image processing information, wherein to obtain the scene classification information includes to:

obtain color component ratio information corresponding to a pixel from the previously processed image;

obtain a pixel classification for the pixel by classifying the pixel based on the color component ratio information corresponding to the pixel; and

classify the scene based on the pixel classification;

generate a processed image by processing the input image based on the scene classification information; and

output the processed image.

2. The non-transitory computer-readable storage medium of claim 1 , wherein the image processing information includes automatic white balance correction information, and wherein to obtain the scene classification information includes to obtain the scene classification information based on the automatic white balance correction information.

3. The non-transitory computer-readable storage medium of claim 1 , wherein to obtain the scene classification information includes to obtain scene classification information indicating whether a scene captured by the input image is an underwater scene.

4. The non-transitory computer-readable storage medium of claim 1 , wherein to classify the pixel includes to:

in response to a determination that a ratio of a green component of the pixel to a red component of the pixel exceeds a corresponding defined threshold, classify the pixel as corresponding to an underwater scene classification; and

in response to a determination that the ratio of the green component of the pixel to the red component of the pixel is within the corresponding defined threshold, classify the pixel as corresponding to a scene classification other than the underwater scene classification.

5. The non-transitory computer-readable storage medium of claim 1 , wherein to obtain the color component ratio information includes to obtain color component ratio values corresponding to the pixel, and to classify the pixel includes to:

obtain a location corresponding to the color component ratio values in a corresponding color component ratio space;

in response to a determination that the location is within a defined portion of the color component ratio space, classify the pixel as corresponding to an underwater scene classification; and

in response to a determination that the location is in a portion of the color component ratio space other than the defined portion, classify the pixel as corresponding to a scene classification other than the underwater scene classification.

6. The non-transitory computer-readable storage medium of claim 1 , wherein:

to obtain the pixel classification includes to obtain a respective pixel classification for each pixel from a plurality of pixels from the previously processed image; and

to classify the scene includes to:

obtain a respective cardinality of pixels corresponding to candidate scene classifications based on the pixel classification; and

identify a candidate scene classification having a largest cardinality of pixels from the candidate scene classifications as the scene classification.

7. The non-transitory computer-readable storage medium of claim 6 , wherein to classify the scene includes to:

obtain a scene classification confidence level indicating a ratio of the cardinality of pixels corresponding to the candidate scene classification having the largest cardinality of pixels to a cardinality of pixels corresponding to the other candidate scene classifications; and

identify the candidate scene classification having the largest cardinality of as the scene classification in response to a determination that the scene classification confidence level exceeds a scene classification confidence threshold.

8. The non-transitory computer-readable storage medium of claim 1 , wherein to obtain the scene classification information includes to:

in response to a determination that the scene classification information indicates a scene classification that differs from a previously identified scene classification corresponding to a frame sequentially preceding the previously processed frame, obtain scene classification transition information indicating a scene transition.

9. A method comprising:

generating, by an image signal processor, a previously processed image, wherein generating the previously processed image includes generating corresponding image processing information;

obtaining, from the image signal processor, the image processing information corresponding to the previously processed image;

obtaining scene classification information for an input image based on the image processing information, wherein obtaining the scene classification information includes in response to a determination that the scene classification information indicates a scene classification that differs from a previously identified scene classification corresponding to a frame sequentially preceding the previously processed frame, obtaining scene classification transition information indicating a scene transition;

generating a processed image by processing the input image based on the scene classification information; and

outputting the processed image.

10. The method of claim 9 , wherein the image processing information includes automatic white balance correction information, and wherein obtaining the scene classification information includes obtaining the scene classification information based on the automatic white balance correction information.

11. The method of claim 10 , wherein obtaining the scene classification information includes:

obtaining color component ratio information corresponding to a pixel from the previously processed image;

obtaining a pixel classification for the pixel by classifying the pixel based on the color component ratio information corresponding to the pixel; and

classifying the scene based on the pixel classification.

12. The method of claim 11 , wherein classifying the pixel includes:

in response to a determination that a ratio of a green component of the pixel to a red component of the pixel exceeds a corresponding defined threshold, classifying the pixel as corresponding to an underwater scene classification; and

in response to a determination that the ratio of the green component of the pixel to the red component of the pixel is within the corresponding defined threshold, classifying the pixel as corresponding to a scene classification other than the underwater scene classification.

13. The method of claim 11 , wherein obtaining the color component ratio information includes obtaining color component ratio values corresponding to the pixel, and classifying the pixel includes:

obtaining a location corresponding to the color component ratio values in a corresponding color component ratio space;

in response to a determination that the location is within a defined portion of the color component ratio space, classifying the pixel as corresponding to an underwater scene classification; and

in response to a determination that the location is in a portion of the color component ratio space other than the defined portion, classifying the pixel as corresponding to a scene classification other than the underwater scene classification.

14. The method of claim 11 , wherein:

obtaining the pixel classification includes obtaining a respective pixel classification for each pixel from a plurality of pixels from the previously processed image; and

classifying the scene includes:

obtaining a respective cardinality of pixels corresponding to candidate scene classifications based on the pixel classification; and

identifying a candidate scene classification having a largest cardinality of pixels from the candidate scene classifications as the scene classification.

15. The method of claim 14 , wherein classifying the scene includes:

obtaining a scene classification confidence level indicating a ratio of the cardinality of pixels corresponding to the candidate scene classification having the largest cardinality of pixels to a cardinality of pixels corresponding to the other candidate scene classifications; and

identifying the candidate scene classification having the largest cardinality of as the scene classification in response to a determination that the scene classification confidence level exceeds a scene classification confidence threshold.

16. The method of claim 9 , wherein obtaining the scene classification information includes obtaining scene classification information indicating whether a scene captured by the input image is an underwater scene.

17. An apparatus comprising:

an image sensor;

a processor configured to:

generate a previously processed image, wherein generating the previously processed image includes generating corresponding image processing information;

obtain the image processing information corresponding to the previously processed image;

obtain scene classification information for an input image based on the image processing information, wherein to obtain the scene classification information includes to:

obtain color component ratio information corresponding to a pixel from the previously processed image;

obtain a pixel classification for the pixel by classifying the pixel based on the color component ratio information corresponding to the pixel; and

classify the scene based on the pixel classification;

generate a processed image by processing the input image based on the scene classification information; and

output the processed image.

18. The apparatus of claim 17 , wherein the processor is configured to obtain the image processing information such that the image processing information includes automatic white balance correction information, and wherein the processor is configured to obtain the scene classification information based on the automatic white balance correction information.

Assignments (6)
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: FARALLON CAPITAL MANAGEMENT, L.L.C., AS AGENT
Reel/Frame 072340/0676 →
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: WELLS FARGO BANK, NATIONAL ASSOCIATION, AS AGENT
Reel/Frame 072358/0001 →
RELEASE OF PATENT SECURITY INTEREST Recorded Jan 25, 2021
From: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
To: GOPRO, INC.
Reel/Frame 055106/0434 →
SECURITY INTEREST Recorded Mar 5, 2019
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 048508/0728 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 15, 2019
From: LEROY, SYLVAIN; CARIOU, ADRIEN; MARQUES, MIGUEL; CARON, JULIEN; FAURE, BRUNO; BELARDI, STÉPHANE
To: GOPRO, INC.
Reel/Frame 048004/0885 →
SECURITY INTEREST Recorded Dec 3, 2018
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 047713/0309 →
Continuity (2)
Provisional Application 62564431 · Sep 28, 2017
Related Publication 20190095713A1 · Mar 28, 2019
Cited By (1)
US 12,676,944