IP Library Granted Patent US 12,148,245
Granted Patent B2
US 12,148,245 · App. 17/557,545 · Granted Nov 19, 2024

Ranking images in an image group

Inventors: Shaomin Xiong (Newark, CA); Toshiki Hirano (San Jose, CA)
Assignee: Sandisk Technologies, Inc.
G06V40/172G06V10/774G06V10/776G06V40/169
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,148,245
App. No.
17/557,545
Granted
Nov 19, 2024
Kind
B2
Abstract

Systems, methods, and data storage devices for image grouping in an end user device using trained machine learning group classifiers are described. The end user device may include an image group classifier configured to classify new image data objects using an image classification algorithm and set of machine learning parameters previously trained for a specific image group. The end user device may determine embeddings that quantify features of the target image object and use those embeddings and the image group classifier to selectively associate group identifiers with each new image data object received or generated by the end user device. Calibration, including selection and training, of the image group classifiers and ranking of classified images are also described.

Claims (102)

1. A system comprising:

at least one processor;

at least one memory;

a storage medium configured to store user image data;

an image manager, stored in the at least one memory for execution by the at least one processor, configured to index image data objects in the user image data;

at least one image group classifier, stored in the at least one memory for execution by the at least one processor, configured to:

determine a first set of embeddings quantifying a plurality of features in a set of image data objects, wherein the first set of embeddings includes a plurality of feature vectors for at least one human face;

process the first set of embeddings for each image data object in the set of image data objects through a set of machine learning parameters assigned to a first group identifier, wherein the set of machine learning parameters is based on training a first image group classifier using a first training set of embeddings for a plurality of image data objects corresponding to the first group identifier; and

determine a subset of image data objects in the set of image data objects corresponding to the first group identifier; and

at least one ranking engine, stored in the at least one memory for execution by the at least one processor, configured to:

process the subset of image data objects to train an embedding model corresponding to the first group identifier;

process, using the embedding model corresponding to the first group identifier, the subset of image data objects to determine, for each image data object in the subset of image data objects, a second set of embeddings;

process, using a ranking algorithm based on relationships among the second sets of embeddings, the second set of embeddings for each image data object in the subset of image data objects; and

return, based on the ranking algorithm, a ranked list of the subset of image data objects, wherein the image manager is further configured to display, based on the ranked list, at least one image data object from the subset of image data objects on a graphical user interface.

2. The system of claim 1 , further comprising:

a face detector, stored in the at least one memory for execution by the at least one processor, configured to:

detect, in the set of image data objects, at least one human face; and

calculate, for the at least one human face, the first set of embeddings quantifying the plurality of features of the at least one human face using a first embedding model corresponding to the plurality of feature vectors for the at least one human face.

3. The system of claim 2 , wherein:

the at least one human face includes a plurality of human faces in the set of image data object;

the first set of embeddings quantifies the plurality of features for each human face of the plurality of human faces; and

the first group identifier corresponds to a social group corresponding to a plurality of people.

4. The system of claim 2 , wherein:

each image data object in the subset of image data objects includes the at least one human face corresponding to the first group identifier; and

the at least one ranking engine is further configured to calculate, for the at least one human face in each image data object in the subset of image data objects, the second set of embeddings quantifying the plurality of features of the at least one human face using the embedding model corresponding to the first group identifier.

5. The system of claim 2 , wherein:

the at least one image group classifier is further configured to:

determine, for each image data object in the set of image data objects, a confidence score based on the set of machine learning parameters and the first set of embeddings for that image data object; and

compare the confidence score to a group confidence threshold to determine the subset of image data objects in the set of image data objects corresponding to the first group identifier; and

the at least one ranking engine is further configured to use the confidence score for each image data object in the subset of image data objects corresponding to the first group identifier.

6. The system of claim 2 , wherein the at least one ranking engine is further configured to use a graph-based ranking algorithm to process the second set of embeddings to determine the ranked list of the subset of image data objects.

7. The system of claim 6 , wherein the graph-based ranking algorithm is selected from:

a PageRank algorithm;

a hyperlink-induced topic search algorithm; and

a SimRank algorithm.

8. The system of claim 6 , wherein the graph-based ranking algorithm is configured to:

tokenize each embedding of the second set of embeddings for each image data object in the subset of image data objects; and

selectively relate, based on at least one weight value, the tokenized embeddings for a first image data object to corresponding tokenized embeddings for each other image data object in the subset of image data objects to determine the ranked list of the subset of image data objects.

9. The system of claim 8 , wherein the at least one ranking engine is further configured to:

receive at least one user tag for at least one image data object in the subset of image data objects; and

modify at least one weight value corresponding to the at least one image data object associated with the at least one user tag.

10. The system of claim 2 , wherein the at least one ranking engine is further configured to:

for each image data object in the subset of image data objects:

determine, based on the first set of embeddings, at least one image quality metric;

determine, based on the set of machine learning parameters and the first set of embeddings for that image data object, at least one confidence score; and

determine, based on a graph-based ranking algorithm, at least one relationship rank value; and

determine the ranked list of the subset of image data objects based on, for each image data object in the subset of image data objects:

the at least one image quality metric;

the at least one confidence score; and

the at least one relationship rank value.

11. A computer-implemented method, comprising:

determining an index of image data objects in user image data stored in a storage medium;

determining, by at least one image group classifier, a first set of embeddings quantifying a plurality of features in a set of image data objects, wherein the first set of embeddings includes a plurality of feature vectors for at least one human face;

processing, by the at least one image group classifier, the first set of embeddings for each image data object in the set of image data objects through a set of machine learning parameters assigned to a first group identifier, wherein the set of machine learning parameters are based on training a first image group classifier using a first training set of embeddings for a plurality of image data objects corresponding to the first group identifier;

determining, by the at least one image group classifier, a subset of image data objects in the set of image data objects corresponding to the first group identifier;

processing the subset of image data objects to train an embedding model corresponding to the first group identifier;

processing, using the embedding model corresponding to the first group identifier, the subset of image data objects to determine, for each image data object in the subset of image data objects, a second set of embeddings;

processing, using a ranking algorithm based on relationships among the second sets of embeddings, the second set of embeddings for each image data object in the subset of image data objects to determine a ranked list of the subset of image data objects; and

displaying, based on the ranked list, at least one image data object on a graphical user interface.

12. The computer-implemented method of claim 11 , further comprising:

detecting, in the set of image data objects, at least one human face; and

calculating, for the at least one human face, the first set of embeddings quantifying the plurality of features of the at least one human face using a first embedding model corresponding to the plurality of feature vectors for the at least one human face, wherein each image data object in the subset of image data objects includes the at least one human face corresponding to the first group identifier.

13. The computer-implemented method of claim 12 , wherein:

the at least one human face includes a plurality of human faces in the set of image data object;

the first set of embeddings quantifies the plurality of features for each human face of the plurality of human faces; and

the first group identifier corresponds to a social group corresponding to a plurality of people.

14. The computer-implemented method of claim 12 , further comprising:

calculating, for the at least one human face in each image data object in the subset of image data objects, the second set of embeddings quantifying the plurality of features of the at least one human face using the embedding model corresponding to the first group identifier that is different than the first embedding model.

15. The computer-implemented method of claim 12 , further comprising:

determining, for each image data object in the set of image data objects, a confidence score based on the set of machine learning parameters and the first set of embeddings for that image data object;

comparing the confidence score to a group confidence threshold to determine the subset of image data objects in the set of image data objects corresponding to the first group identifier; and

using the confidence score for each image data object in the subset of image data objects to determine the ranked list of the subset of image data objects.

16. The computer-implemented method of claim 12 , further comprising:

using a graph-based ranking algorithm to process the second set of embeddings to determine the ranked list of the subset of image data objects.

17. The computer-implemented method of claim 16 , further comprising:

tokenizing each embedding of the second set of embeddings for each image data object in the subset of image data objects; and

selectively relating, based on at least one weight value of the graph-based ranking algorithm, tokenized embeddings from a first image data object to corresponding tokenized embeddings for each other image data object in the subset of image data objects to determine the ranked list of the subset of image data objects.

18. The computer-implemented method of claim 17 , further comprising:

receiving at least one user tag for at least one image data object in the subset of image data objects; and

modifying at least one weight value corresponding to the at least one image data object associated with the at least one user tag.

19. The computer-implemented method of claim 18 , further comprising:

for each image data object in the subset of image data objects:

determining, based on the first set of embeddings, at least one image quality metric;

determining, based on the set of machine learning parameters and the first set of embeddings for that image data object, at least one confidence score; and

determining, based on a graph-based ranking algorithm, at least one relationship rank value; and

determining the ranked list of the subset of image data objects based on, for each image data object in the subset of image data objects:

the at least one image quality metric;

the at least one confidence score; and

the at least one relationship rank value.

20. An end-user computing device, comprising:

at least one processor;

at least one memory;

a camera configured to capture user image data;

a storage medium configured to store user image data;

means for determining an index of image data objects in user image data stored in the storage medium;

means for determining, by at least one image group classifier, a first set of embeddings quantifying a plurality of features in a set of image data objects, wherein the first set of embeddings includes a plurality of feature vectors for at least one human face;

means for processing, by the at least one image group classifier, the first set of embeddings for each image data object in the set of image data objects through a set of machine learning parameters assigned to a first group identifier, wherein the set of machine learning parameters is based on training a first image group classifier using a first training set of embeddings for a plurality of image data objects corresponding to the first group identifier;

means for determining, by the at least one image group classifier, a subset of image data objects in the set of image data objects corresponding to the first group identifier;

means for processing the subset of image data objects to train an embedding model corresponding to the first group identifier;

means for processing, using the embedding model corresponding to the first group identifier, the subset of image data objects to determine, for each image data object in the subset of image data objects, a second set of embeddings;

means for processing, using a ranking algorithm based on relationships among the second sets of embeddings, the second set of embeddings for each image data object in the subset of image data objects to determine a ranked list of the subset of image data objects; and

means for displaying, based on the ranked list, at least one image data object on a graphical user interface.

Assignments (8)
PARTIAL RELEASE OF SECURITY INTERESTS Recorded Apr 25, 2025
From: JPMORGAN CHASE BANK, N.A., AS AGENT
To: SANDISK TECHNOLOGIES, INC.
Reel/Frame 071382/0001 →
SECURITY AGREEMENT Recorded Apr 25, 2025
From: SANDISK TECHNOLOGIES, INC.
To: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT
Reel/Frame 071050/0001 →
PATENT COLLATERAL AGREEMENT Recorded Aug 23, 2024
From: SANDISK TECHNOLOGIES, INC.
To: JPMORGAN CHASE BANK, N.A., AS THE AGENT
Reel/Frame 068762/0494 →
CHANGE OF NAME Recorded Jun 27, 2024
From: SANDISK TECHNOLOGIES, INC.
To: SANDISK TECHNOLOGIES, INC.
Reel/Frame 067982/0032 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 29, 2024
From: WESTERN DIGITAL TECHNOLOGIES, INC.
To: SANDISK TECHNOLOGIES, INC.
Reel/Frame 067567/0682 →
PATENT COLLATERAL AGREEMENT - DDTL LOAN AGREEMENT Recorded Aug 21, 2023
From: WESTERN DIGITAL TECHNOLOGIES, INC.
To: JPMORGAN CHASE BANK, N.A.
Reel/Frame 067045/0156 →
PATENT COLLATERAL AGREEMENT - A&R LOAN AGREEMENT Recorded Aug 21, 2023
From: WESTERN DIGITAL TECHNOLOGIES, INC.
To: JPMORGAN CHASE BANK, N.A.
Reel/Frame 064715/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 22, 2021
From: XIONG, SHAOMIN; HIRANO, TOSHIKI
To: WESTERN DIGITAL TECHNOLOGIES, INC.
Reel/Frame 058457/0283 →
Continuity (1)
Related Publication 20230196832A1 · Jun 22, 2023
Cited By (2)
US 12,412,419 US 12,694,651