Granted Patent B1

US 12,266,209 · App. 18/587,267 · Granted Apr 1, 2025

Inventors: Jason B. Bryslawskyj (San Diego, CA); Yi Zhang (Santa Clara, CA); Emanoel Daryoush (San Jose, CA); Ari Azarafrooz (Rancho Santa Margarita, CA); Wayne Xin (Santa Clara, CA); Yihua Liao (Palo Alto, CA); Niranjan Koduri (Pleasanton, CA)

Assignee: Netskope, Inc.

G06V40/1365G06V10/762G06V20/70G06V40/1347

Abstract

Claims (100)

1. A computer-implemented method of determining a quality an image classifier, the method comprising:

receiving a plurality of images provided by a user and a user-defined label for the plurality of images;

generating, with an image fingerprint model, a plurality of image fingerprint embeddings, wherein each image fingerprint embedding of the plurality of image fingerprint embeddings represents an associated image of the plurality of images;

creating the image classifier using each image fingerprint embedding in the plurality of image fingerprint embeddings and a plurality of negative image fingerprint embeddings;

creating a plurality of test classifiers, wherein the creating comprises creating one test classifier of the plurality of test classifiers for each image fingerprint embedding in the plurality of image fingerprint embeddings, wherein creating each test classifier of the plurality of test classifiers comprises:

selecting an image fingerprint embedding of the plurality of image fingerprint embeddings as a test embedding for which to create the respective test classifier, and

creating the respective test classifier using the plurality of negative image fingerprint embeddings and each of the plurality of image fingerprint embeddings except the selected image fingerprint embedding;

applying each of the plurality of test classifiers to its respective test embedding to obtain a plurality of recall results; and

assessing the quality of the image classifier based on the plurality of recall results.

2. The computer-implemented method of claim 1 , wherein the creating the respective test classifier for a respective image fingerprint embedding comprises:

generating an index and an approximate nearest neighbor model associated with the index using the plurality of negative image fingerprint embeddings and each of the plurality of image fingerprint embeddings except the respective image fingerprint embedding;

clustering the plurality of image fingerprint embeddings in the index into a user embedding cluster; and

applying the user-defined label to the user embedding cluster.

3. The computer-implemented method of claim 2 , wherein the applying each of the plurality of test classifiers to its respective test embedding comprises:

providing the respective test embedding as input to the respective approximate nearest neighbor model trained to retrieve a label of a most similar image fingerprint embedding from the associated index and generate a score indicating a similarity of the associated image fingerprint embedding to the most similar image fingerprint embedding; and

receiving, from the respective approximate nearest neighbor model in response to the input, the label and the score.

4. The computer-implemented method of claim 3 , wherein the applying each of the plurality of test classifiers to its respective test embedding further comprises:

providing a second plurality of negative image fingerprint embeddings as input to the respective approximate nearest neighbor model; and

receiving, from the respective approximate nearest neighbor model in response to the input, the label and the score for each of the second plurality of negative image fingerprint embeddings.

5. The computer-implemented method of claim 2 , wherein the creating the respective test classifier for a respective image fingerprint embedding further comprises:

clustering the plurality of negative image fingerprint embeddings into a negative embedding cluster; and

applying a negative label to the negative embedding cluster.

6. The computer-implemented method of claim 1 , wherein the assessing the quality of the image classifier comprises:

for each test classifier of the plurality of test classifiers, after applying the test classifier to its respective test embedding, calculating a recall value of the respective test classifier; and

generating an average recall value using recall values generated based on each of the plurality of test classifiers.

7. The computer-implemented method of claim 6 , wherein the assessing the quality of the image classifier further comprises:

in response to determining the average recall value falls below a threshold quality value, transmitting a notification to the user providing an indication of the quality of the image classifier.

8. The computer-implemented method of claim 7 , wherein the notification further comprises one of:

a suggestion to provide additional images that can be classified with the user-defined label;

a suggestion to provide higher quality images associated with the user-defined label;

a suggestion to reduce a threshold value used to determine whether subject images are classified with the user-defined label; or

a combination thereof.

9. The computer-implemented method of claim 1 , wherein the creating the image classifier comprises:

generating an index and an approximate nearest neighbor model associated with the index using the plurality of image fingerprint embeddings;

clustering the plurality of image fingerprint embeddings into a user embedding cluster; and

applying the user-defined label to the user embedding cluster.

10. The computer-implemented method of claim 9 , wherein the creating the image classifier further comprises:

clustering the plurality of negative image fingerprint embeddings into a negative embedding cluster; and

applying a negative label to the negative embedding cluster.

11. The computer-implemented method of claim 9 , further comprising:

identifying outliers of the plurality of the images, the identifying the outliers comprising:

calculating a centroid of the user embedding cluster; and

for each image fingerprint embedding of the plurality of image fingerprint embeddings:

calculating a distance between the respective image fingerprint embedding and the centroid,

calculating a z-score of the distance for the respective image fingerprint embedding, and

based on a comparison of the z-score to a threshold score:

identifying the respective image fingerprint embedding as an outlier when the z-score meets or exceeds the threshold score; and

identifying the respective image fingerprint embedding as an inlier when the z-score is below the threshold score.

12. The computer-implemented method of claim 11 , further comprising:

based on identifying the outliers, transmitting a notification to the user indicating images of the plurality of images associated with the image fingerprint embeddings that are identified as the outliers.

13. The computer-implemented method of claim 11 , further comprising:

recreating the image classifier using the plurality of image fingerprint embeddings after removing image fingerprint embeddings that are identified as the outliers from the plurality of image fingerprint embeddings.

14. The computer-implemented method of claim 1 , wherein the image fingerprint model is trained to generate an image fingerprint embedding comprising a multi-dimensional feature vector representing visual aspects of the associated image input to the image fingerprint model.

15. A system for determining a quality of an image classifier, the system comprising:

one or more processors; and

one or more memories having stored thereon instructions that, upon execution by the one or more processors, cause the one or more processors to:

receive, via a user interface, a plurality of images provided by a user and a user-defined label for the plurality of images;

generate, with an image fingerprint model of the system, a plurality of image fingerprint embeddings, wherein each image fingerprint embedding of the plurality of image fingerprint embeddings represents an associated image of the plurality of images;

create the image classifier using each image fingerprint embedding in the plurality of image fingerprint embeddings and a plurality of negative image fingerprint embeddings;

create a plurality of test classifiers comprising one test classifier of the plurality of test classifiers for each image fingerprint embedding in the plurality of image fingerprint embeddings, wherein the instructions to create each test classifier of the plurality of test classifiers comprises instructions that further cause the one or more processors to:

select an image fingerprint embedding of the plurality of image fingerprint embeddings as a test embedding for which to create the respective test classifier, and

create the respective test classifier using the plurality of negative image fingerprint embeddings and each of the plurality of image fingerprint embeddings except the selected image fingerprint embedding;

apply each of the plurality of test classifiers to its respective test embedding to obtain a plurality of recall results; and

assess the quality of the image classifier based on the recall results.

16. The system of claim 15 , wherein:

the instructions to create the respective test classifier for a respective image fingerprint embedding comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:

generate an index and an approximate nearest neighbor model associated with the index using the plurality of negative image fingerprint embeddings and each of the plurality of image fingerprint embeddings except the respective image fingerprint embedding,

cluster the plurality of image fingerprint embeddings in the index into a user embedding cluster, and

apply the user-defined label to the user embedding cluster; and

the instructions to apply each of the plurality of test classifiers to its respective test embedding comprise further instructions that, upon execution by the one or more processors cause the one or more processors to:

provide the respective test embedding as input to the respective approximate nearest neighbor model trained to retrieve a label of a most similar image fingerprint embedding from the associated index and generate a score indicating a similarity of the associated image fingerprint embedding to the most similar image fingerprint embedding, and

receive, from the respective approximate nearest neighbor model in response to the input, the label and the score.

17. The system of claim 15 , wherein the instructions to assess the quality of the image classifier comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:

for each test classifier of the plurality of test classifiers, after applying the test classifier to its respective test embedding, calculate a recall value of the respective test classifier;

generate an average recall value using recall values generated based on each of the plurality of test classifiers; and

in response to determining the average recall value falls below a threshold value, transmitting a notification to the user providing an indication of the quality of the image classifier, wherein the notification further comprises one of:

a suggestion to provide additional images that can be classified with the user-defined label;

a suggestion to provide higher quality images associated with the user-defined label;

a suggestion to reduce a threshold value used to determine whether subject images are classified with the user-defined label; or

a combination thereof.

18. The system of claim 15 , wherein the instructions to create the image classifier comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:

generate an index and an approximate nearest neighbor model associated with the index using the plurality of image fingerprint embeddings;

cluster the plurality of image fingerprint embeddings into a user embedding cluster; and

apply the user-defined label to the user embedding cluster.

19. The system of claim 18 , wherein the instructions comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:

identify outliers of the plurality of the images, the instructions to identify the outliers comprising further instructions that, upon execution by the one or more processors, cause the one or more processors to:

calculate a centroid of the user embedding cluster, and

for each image fingerprint embedding of the plurality of image fingerprint embeddings:

calculate a distance between the respective image fingerprint embedding and the centroid;

calculate a z-score of the distance for the respective image fingerprint embedding; and

based on a comparison of the z-score to a threshold score:

identify the respective image fingerprint embedding as an outlier when the z-score meets or exceeds the threshold score, and

identify the respective image fingerprint embedding as an inlier when the z-score is below the threshold score; and

based on identifying the outliers, perform one of:

transmit a notification to the user indicating images of the plurality of images associated with the image fingerprint embeddings that are identified as the outliers,

recreate the image classifier using the plurality of image fingerprint embeddings after removing image fingerprint embeddings that are identified as the outliers from the plurality of image fingerprint embeddings, or

a combination thereof.

20. The system of claim 18 , wherein the instructions to create the image classifier further comprise instructions that, upon execution by the one or more processors, cause the one or more processors to:

cluster the plurality of negative image fingerprint embeddings into a negative embedding cluster; and

apply a negative label to the negative embedding cluster.

Assignments (1)

References Cited (152)

Cited By (1)

US 12,511,925