Granted Patent B2

US 12,367,200 · App. 16/668,252 · Granted Jul 22, 2025

Inventors: Mustafa Canim (Ossining, NY); Bishwaranjan Bhattacharjee (Yorktown Heights, NY); Alfio Massimiliano Gliozzo (Brooklyn, NY)

Assignee: International Business Machines Corporation

G06F16/24578G06F16/2438

Abstract

Claims (105)

1. A method comprising:

receiving an image from an image source;

extracting a plurality of image tags from the image;

receiving, from a model database, identification of base models of a base model set comprising a plurality of machine learning models;

identifying, from the identification, a plurality of terms comprising, for each respective machine learning model of the base models in the base model set, a respective set of one or more terms associated with the respective machine learning model;

initializing a counter for each base model in the base model set;

selecting a subset of non-matching image tags from the plurality of image tags, wherein each respective image tag in the subset of non-matching image tags is not present in the plurality of terms and is not present in the sets of the one or more terms associated with the machine learning models of the base models;

identifying a set of associated terms for a first tag in the subset of non-matching tags by processing the first tag using an embedding machine learning model, wherein the set of associated terms comprises an expanded base of related word terms for the first tag;

querying a knowledge database to determine a set of parent terms for the set of associated terms;

updating the associated counters for one or more base models of the base model set based on query results from the knowledge database, comprising determining whether each parent term of the set of parent terms are present in the plurality of terms; and

ranking the base models of the updated base model set based on the plurality of image tags, the query results, the plurality of terms, and the associated base model counters, wherein the ranking indicates relevance of each of the base models with respect to the image.

2. The method of claim 1 , further comprising

receiving the base model set from the image source.

3. The method of claim 2 , further comprising:

prior to using the embedding machine learning model, determining a matching associated base model from the base model set for a second tag of the plurality of image tags;

incrementing a counter for the matching associated base model; and

selecting a next image tag from the plurality of image tags.

4. The method of claim 2 , wherein identifying the set of associated terms further comprises:

receiving a first set of associated terms for the first tag;

determining a subset of the set of associated terms; and

wherein querying the knowledge database comprises:

selecting a first term from the set of associated terms;

querying the knowledge database using the first term; and

incrementing a counter for a first term associated base model in the base model set when a query to the knowledge database returns a parent term directly associated with the first term associated base model.

5. The method of claim 4 , further comprising:

when a parent term is not directly associated with a base model in the base model set, determining a next parent term from the knowledge database; and

incrementing a counter for a second term associated base model in the base model set when a query to the knowledge database returns a next parent term directly associated with the second term associated base model.

6. The method of claim 4 , further comprising:

upon determining that a parent term is not directly associated with a base model in base model set, determining a next parent term from the knowledge database;

determining that the next parent term is not directly associated with a base model in the base model set;

determining from the knowledge database the next parent term is a limited parent term; and

selecting a next term in the first set of associated terms.

7. The method of claim 1 , wherein ranking the base models further comprises:

determining a rank for each of the base models based on associated base model counters; and

transmitting the base models to the image source.

8. A system comprising:

one or more computer processors; and

a memory containing a program which when executed by the computer processors performs an operation comprising:

receiving an image from an image source;

extracting a plurality of image tags from the image;

receiving, from a model database, identification of base models of a base model set comprising a plurality of machine learning models;

identifying, from the identification, a plurality of terms comprising, for each respective machine learning model of the base models in the base model set, a respective set of one or more terms associated with the respective machine learning model;

selecting a subset of non-matching image tags from the plurality of image tags, wherein each respective image tag in the subset of non-matching image tags is not present in the plurality of terms and is not present in the sets of the one or more terms associated with the machine learning models of the base models;

identifying a set of associated terms for a first tag in the subset of non-matching tags by processing the first tag using an embedding machine learning model, wherein the set of associated terms comprises an expanded base of related word terms for the first tag;

querying a knowledge database to determine a set of parent terms for the set of associated terms;

updating the base model set based on query results from the knowledge database, comprising determining whether each parent term of the set of parent terms are present in the plurality of terms;

ranking the base models of the updated base model set based on the plurality of image tags, the query results, and the plurality of terms, wherein the ranking indicates relevance of each of the base models with respect to the image; and

further training, using the image as training data, a first ranked model of the updated base model set.

9. The system of claim 8 , further comprising:

receiving the base model set from the image source; and

initializing a counter for each base model in the base model set.

10. The system of claim 9 , further comprising:

prior to using the embedding machine learning model, determining a matching associated base model from the base model set for a second tag of the plurality of image tags;

incrementing a counter for the matching associated base model; and

selecting a next image tag from the plurality of image tags.

11. The system of claim 9 , wherein identifying the set of associated terms further comprises:

receiving a first set of associated terms for the first tag;

determining a subset of the first set of associated terms; and

wherein querying the knowledge database comprises:

selecting a first term from the first set of associated terms;

querying the knowledge database using the first term; and

incrementing a counter for a first term associated base model in the base model set when a query to the knowledge database returns a parent term directly associated with the first term associated base model.

12. The system of claim 11 , further comprising:

when a parent term is not directly associated with a base model in the base model set, determining a next parent term from the knowledge database; and

incrementing a counter for a second term associated base model in the base model set when a query to the knowledge database returns a next parent term directly associated with the second term associated base model.

13. The system of claim 11 , further comprising:

upon determining that a parent term is not directly associated with a base model in the base model set, determining a next parent term from the knowledge database;

determining that the next parent term is not directly associated with a base model in the base model set;

determining from the knowledge database the next parent term is a limited parent term; and

selecting a next term in the first set of associated terms.

14. The system of claim 8 , wherein ranking the base models further comprises:

determining a rank for each of the base models based on associated base model counters; and

transmitting the base models to the image source.

15. A computer program product comprising:

a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising:

receiving an image from an image source;

extracting a plurality of image tags from the image;

receiving, from a model database, identification of base models of a base model set comprising a plurality of machine learning models;

identifying, from the identification, a plurality of terms comprising, for each respective machine learning model of the base models in the base model set, a respective set of one or more terms associated with the respective machine learning model;

initializing a counter for each base model in the base model set;

selecting a subset of non-matching image tags from the plurality of image tags, wherein each respective image tag in the subset of non-matching image tags is not present in the plurality of terms and is not present in the sets of the one or more terms associated with the machine learning models of the base models;

identifying a set of associated terms for a first tag in the subset of non-matching tags by processing the first tag using an embedding machine learning model, wherein the set of associated terms comprises an expanded base of related word terms for the first tag;

querying a knowledge database to determine a set of parent terms for the set of associated terms;

updating the associated counters for one or more base models of the base model set based on query results from the knowledge database, comprising determining whether each parent term of the set of parent terms are present in the plurality of terms; and

ranking the base models of the updated base model set based on the plurality of image tags, the query results, the plurality of terms, and the associated base model counters, wherein the ranking indicates relevance of each of the base models with respect to the image.

16. The computer program product of claim 15 , further comprising receiving the base model set from the image source.

17. The computer program product of claim 16 , further comprising:

prior to using the embedding machine learning model, determining a matching associated base model from the base model set for a second tag of the plurality of image tags;

incrementing a counter for the matching associated base model; and

selecting a next image tag from the plurality of image tags.

18. The computer program product of claim 17 , wherein identifying the set of associated terms further comprises:

receiving a first set of associated terms for the first tag;

determining a subset of the first set of associated terms; and

wherein querying the knowledge database comprises:

selecting a first term from the first set of associated terms;

querying the knowledge database using the first term; and

incrementing a counter for a first term associated base model in the base model set when a query to the knowledge database returns a parent term directly associated with the first term associated base model.

19. The computer program product of claim 18 , further comprising:

when a parent term is not directly associated with a base model in the base model set, determining a next parent term from the knowledge database; and

incrementing a counter for a second term associated base model in the base model set when a query to the knowledge database returns a next parent term directly associated with the second term associated base model.

20. The computer program product of claim 18 , further comprising:

upon determining that a parent term is not directly associated with a base model in base model set, determining a next parent term from the knowledge database;

determining that the next parent term is not directly associated with a base model in the base model set;

determining from the knowledge database the next parent term is a limited parent term; and

selecting a next term in the first set of associated terms.

Assignments (1)

Continuity (1)

References Cited (72)