IP Library Granted Patent US 10,853,704
Granted Patent B2
US 10,853,704 · App. 16/223,947 · Granted Dec 1, 2020

Model-based image labeling and/or segmentation

Inventors: Yanan Jian (New York, NY); Matthew Zeiler (New York, NY); Marshall Jones (New York, NY)
Assignee: Clarifai, Inc.
G06K9/66G06N3/04G06T7/174
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,853,704
App. No.
16/223,947
Granted
Dec 1, 2020
Kind
B2
Abstract

In some embodiments, reduction of computational resource usage related to image labeling and/or segmentation may be facilitated. In some embodiments, a collection of images may be used to train one or more prediction models. Based on a presentation of an image on a user interface, an indication of a target quantity of superpixels for the image may be obtained. The image may be provided to a first prediction model to cause the prediction model to predict a quantity of superpixels for the image. The target quantity of superpixels may be provided to the first model to update the first model's configurations based on (i) the predicted quantity and (ii) the target quantity. A set of superpixels may be generated for the image based on the target quantity, and segmentation information related to the superpixels set may be provided to a second prediction model to update the second model's configurations.

Claims (83)

1. A method of facilitating reduction of computational resource usage related to image labeling and/or segmentation, the method being implemented by a computer system that comprises one or more processors executing computer program instructions that, when executed, perform the method, the method comprising:

obtaining a collection of images comprising 1000 or more images; and

with respect to each image of the 1000 or more images:

causing presentation of the image on a user interface;

obtaining, based on the presentation of the image, an indication of a target quantity of superpixels for the image;

providing the image to a first neural network to cause the first neural network to predict a quantity of superpixels for the image; and

providing the target quantity of superpixels as reference feedback to the first neural network to cause the first neural network to assess the predicted quantity of superpixels against the target quantity of superpixels, the first neural network updating one or more configurations of the first neural network based on the first neural network's assessment of the predicted quantity of superpixels.

2. The method of claim 1 , further comprising, with respect to each image of the 1000 or more images:

obtaining a first indication of a first target quantity of superpixels for a first region of the image and a second indication of a second target quantity of superpixels for a second region of the image that is different from the first region of the image;

providing the first target quantity of superpixels as reference feedback to the first neural network, the first neural network updating the one or more configurations based on (i) a first predicted quantity of superpixels for the first region and (ii) the first target quantity of superpixels for the first region; and

providing the second target quantity of superpixels as reference feedback to the first neural network, the first neural network updating the one or more configurations based on (i) a second predicted quantity of superpixels for the second region and (ii) the second target quantity of superpixels for the second region.

3. The method of claim 2 , further comprising, with respect to each image of the 1000 or more images:

generating an initial set of superpixels for the image;

causing presentation of the initial set of superpixels over the image; and

obtaining, based on the presentation of the initial set of superpixels, the first indication of the first target quantity of superpixels for the first region.

4. The method of claim 3 , further comprising:

generating a first set of superpixels for the first region based on the first target quantity of superpixels for the first region;

causing presentation of the first set of superpixels over the first region of the image; and

subsequent to the presentation of the first set of superpixels, obtaining the second indication of the second target quantity of superpixels for the second region.

5. The method of claim 4 , further comprising:

subsequent to the second target quantity being provided to the first neural network, generating a second set of superpixels for the second region based on the second target quantity of superpixels for the second region; and

causing presentation of the second set of superpixels over the second region of the image.

6. The method of claim 5 , wherein (i) the second region is a subset of the first region such that the first region comprises the second region and one or more other regions or (ii) the first and second regions of the image comprises at least one common region of the image.

7. The method of claim 5 , wherein the first and second regions of the image are mutually exclusive of one another.

8. The method of claim 1 , further comprising:

subsequent to the updating of the one or more configurations of the first neural network, providing a subsequent image related to a concept to the first neural network to cause the first neural network to predict a quantity of superpixels for the subsequent image;

generating a set of superpixels for the subsequent image based on the predicted quantity of superpixels for the subsequent image;

providing the subsequent image to a second neural network to cause the second neural network to predict one or more segments related to the concept in the subsequent image; and

providing segmentation information related to the set of superpixels for the subsequent image as reference feedback to the second neural network to cause the second neural network to assess the one or more predicted segments against the segmentation information, the second neural network updating one or more configurations of the second neural network based on the second neural network's assessment of the one or more predicted segments.

9. The method of claim 1 , further comprising, with respect to each image of the 1000 or more images:

generating a set of superpixels for the image based on the target quantity of superpixels for the image;

causing presentation of the set of superpixels over the image;

obtaining user feedback indicating multiple superpixels of the set of superpixels that reflect a concept in the image;

providing the image to a second neural network to cause the second neural network to predict one or more segments related to the concept in the image; and

providing, based on the user feedback, segmentation information related to the concept to the second neural network to cause the second neural network to assess the one or more predicted segments against the segmentation information, the second neural network updating one or more configurations of the second neural network based on the second neural network's assessment of the one or more predicted segments.

10. The method of claim 1 , further comprising, with respect to each image of the 1000 or more images:

generating a set of superpixels for the image based on the predicted quantity of superpixels for the image;

causing presentation of the set of superpixels over the image;

providing the image to a second neural network to cause the second neural network to predict one or more segments related to a concept in the image;

obtaining a first user selection of at least one superpixel of the set of superpixels as a positive example of the concept in the image and a second user selection of at least another superpixel of the set of superpixels as a negative example of the concept in the image; and

providing, based on the first and second user selections, segmentation information related to the concept to the second neural network to cause the second neural network to assess the one or more predicted segments against the segmentation information, the second neural network updating one or more configurations of the second neural network based on the second neural network's assessment of the one or more predicted segments.

11. A system comprising:

a computer system that comprises one or more processors programmed with computer program instructions that, when executed, cause the computer system to:

obtain an indication of a target quantity of segments for an image;

provide the image to a prediction model to cause the prediction model to predict a quantity of segments for the image; and

provide the target quantity of segments as reference feedback to the prediction model to cause the prediction model to assess the predicted quantity of segments against the target quantity of segments, the prediction model updating one or more portions of the prediction model based on the prediction model's assessment of the predicted quantity of segments.

12. The system of claim 11 , wherein the computer system is caused to:

obtain a first indication of a first target quantity of segments for a first region of the image and a second indication of a second target quantity of segments for a second region of the image that is different from the first region of the image;

provide the first target quantity of segments as reference feedback to the prediction model, the prediction model updating the one or more portions of the prediction model based on (i) a first predicted quantity of segments for the first region and (ii) the first target quantity of segments for the first region; and

provide the second target quantity of segments as reference feedback to the prediction model, the prediction model updating the one or more portions of the prediction model based on (i) a second predicted quantity of segments for the second region and (ii) the second target quantity of segments for the second region.

13. The system of claim 12 , wherein the computer system is caused to:

generate an initial set of superpixels for the image;

cause presentation of the initial set of superpixels over the image; and

obtain, based on the presentation of the initial set of superpixels, the first indication of the first target quantity of superpixels for the first region.

14. The system of claim 13 , wherein the computer system is caused to:

generate a first set of superpixels for the first region based on the first target quantity of superpixels for the first region;

cause presentation of the first set of superpixels over the first region of the image;

subsequent to the presentation of the first set of superpixels, obtain the second indication of the second target quantity of superpixels for the second region;

subsequent to the second target quantity being provided to the prediction model, generate a second set of superpixels for the second region based on the second target quantity of superpixels for the second region; and

cause presentation of the second set of superpixels over the second region of the image.

15. A method implemented by one or more processors executing computer program instructions that, when executed, perform the method, the method comprising:

obtaining an image related to a concept;

obtaining an indication of a quantity of superpixels for the image;

generating a set of superpixels for the image based on the quantity of superpixels for the image;

providing the image to a neural network to cause the neural network to predict one or more segments related to the concept in the image; and

providing segmentation information related to the set of superpixels for the image as reference feedback to the neural network to cause the neural network to assess the one or more predicted segments against the segmentation information, the neural network updating one or more portions of the neural network based on the neural network's assessment of the one or more predicted segments.

16. The method of claim 15 , further comprising:

generating an initial set of superpixels for the image;

causing presentation of the initial set of superpixels over the image; and

obtaining, based on the presentation of the initial set of superpixels, the indication of the quantity of superpixels for the image.

17. The method of claim 16 ,

wherein obtaining the indication of the quantity of superpixels for the image comprises:

obtaining, based on the presentation of the initial set of superpixels, a first indication of a first quantity of superpixels for a first region of the image;

generating a first set of superpixels for the first region based on the first quantity of superpixels for the first region;

causing presentation of the first set of superpixels over the first region of the image; and

subsequent to the presentation of the first set of superpixels, obtaining a second indication of a second quantity of superpixels for a second region of the image that is different from the first region of the image, and

wherein generating the set of superpixels for the image comprises generating the set of superpixels for the image based on the first and second quantities of superpixels.

18. The method of claim 15 , further comprising:

obtaining user feedback indicating one or more superpixels of the set of superpixels that reflect the concept in the image; and

providing, based on the user feedback, the segmentation information to the neural network to cause the neural network to assess the one or more predicted segments against the segmentation information.

19. The method of claim 15 , further comprising:

obtaining a first user selection of at least one superpixel of the set of superpixels as a positive example of the concept in the image and a second user selection of at least another superpixel of the set of superpixels as a negative example of the concept in the image; and

providing, based on the first and second user selections, the segmentation information the neural network to cause the neural network to assess the one or more predicted segments against the segmentation information.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 11, 2026
From: CLARIFAI, INC.
To: NEBIUS BV
Reel/Frame 075712/0109 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 30, 2019
From: JIAN, YANAN; ZEILER, MATTHEW; JONES, MARSHALL
To: CLARIFAI, INC.
Reel/Frame 048196/0745 →
Continuity (1)
Related Publication 20200193246A1 · Jun 18, 2020