IP Library › Granted Patent US 12,724,984
Granted Patent B2
US 12,724,984 · App. 18/062,208 · Granted Sep 1, 2026

Hierarchical multi-armed bandit for lexicon expansion

Inventors: Muntasir Wahed (Blacksburg, VA); Daniel Gruhl (San Jose, CA)
Assignee: International Business Machines Corporation
G06F40/47G06F40/237G06F40/30
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,724,984
App. No.
18/062,208
Granted
Sep 1, 2026
Kind
B2
Abstract

Disclosed herein are methods, systems, and computer program products for selecting an artificial intelligence (AI) model. Aspects include receiving, by a multi-armed bandit agent, candidate entities from multiple set expansion models and selecting a subset of the candidate entities for a first set expansion task, wherein a first candidate entity of the selected subset of candidate entities is selected from a first model by a subject matter expert. Aspects also include selecting, by the multi-armed bandit agent, based on the first candidate entity selected by the subject matter expert, the first model from the set expansion models for generating further candidate entities.

Claims (44)

1 . A computer-implemented method for selecting an artificial intelligence (AI) model, the computer-implemented method comprising:

receiving, by a multi-armed bandit agent, candidate entities from multiple set expansion models, wherein the multiple set expansion models comprise a first AI model and a second AI model previously trained on different datasets;

selecting a subset of the candidate entities for a first set expansion task, wherein a first candidate entity of the selected subset of candidate entities is selected from the first AI model by a subject matter expert;

selecting, by the multi-armed bandit agent, based on the first candidate entity selected by the subject matter expert, the first AI model from the set expansion models for generating further candidate entities;

in response to the first candidate entity from the first AI model being selected, selecting, without having knowledge of the different datasets and while avoiding knowledge of the training distribution of the different datasets, the second AI model configured to output the first candidate entity, wherein both the first AI model and the second AI model are configured to output the first candidate entity based on receiving input of a set of positive inputs and a set of negative inputs such that a reward is given to the first AI model and the second AI model for generating the first candidate entity, wherein the set of positive inputs and the set of negative inputs are input to the first AI model and the second AI model after the first AI model and the second AI model have been previously trained on the different datasets; and

causing the first AI model and the second AI model selected to be executed for generating an expansion to a lexicon for natural language processing (NLP).

2 . The computer-implemented method of claim 1 , wherein the NLP is executed for at least one of a query understanding and a question answering.

3 . The computer-implemented method of claim 1 , wherein the first AI model outputs a first list comprising the first candidate entity and the second AI model outputs a second list comprising the first candidate entity.

4 . The computer-implemented method of claim 1 , wherein to alleviate a cold- start problem with the multi-armed bandit agent, frequency-based boosting (Marble-B) is used to generate initial candidate entities from the expansion models.

5 . The computer-implemented method of claim 4 , wherein for the frequency-based boosting:

all AI models of the expansion models are executed in order to generate the initial candidate entities; and

in response to executing the all AI models of the expansion to generate the initial candidate entities, the initial candidate entities from the all AI models are captured and one or more most frequent candidate entities from the initial candidate entities are presented to the subject matter expert.

6 . The computer-implemented method of claim 1 , wherein to alleviate a cold- start problem with the multi-armed bandit agent, history-based boosting (Marble-H) is used to generate initial candidate entities from the expansion models.

7 . The computer-implemented method of claim 6 , wherein for the history-based boosting, in response to the subject matter expert selecting, at time t, the first candidate entity from the first AI model from the expansion models, the first AI model is rewarded and further, a second model that also generates the first candidate entity at time t+1, is also rewarded.

8 . A system comprising:

a memory; and

one or more processing units coupled with the memory, the one or more processing units configured to perform a method for selecting an artificial intelligence (AI) model, the method comprising:

receiving, by a multi-armed bandit agent, candidate entities from multiple set expansion models, wherein the multiple set expansion models comprise a first AI model and a second AI model previously trained on different datasets;

selecting a subset of the candidate entities for a first set expansion task, wherein a first candidate entity of the selected subset of candidate entities is selected from the first AI model by a subject matter expert;

selecting, by the multi-armed bandit agent, based on the first candidate entity selected by the subject matter expert, the first AI model from the set expansion models for generating further candidate entities;

in response to the first candidate entity from the first AI model being selected, selecting, without having knowledge of the different datasets and while avoiding knowledge of the training distribution of the different datasets, the second AI model; configured to output the first candidate entity, wherein both the first AI model and the second AI model are configured to output the first candidate entity based on receiving input of a set of positive inputs and a set of negative inputs such that a reward is given to the first AI model and the second AI model for generating the first candidate entity, wherein the set of positive inputs and the set of negative inputs are input to the first AI model and the second AI model after the first AI model and the second AI model have been previously trained on the different datasets; and

causing the first AI model and the second AI model selected to be executed for generating an expansion to a lexicon for natural language processing (NLP).

9 . The system of claim 8 , wherein the NLP is executed for at least one of a query understanding and a question answering.

10 . The system of claim 8 , wherein the first AI model outputs a first list comprising the first candidate entity and the second AI model outputs a second list comprising the first candidate entity.

11 . The system of claim 8 , wherein to alleviate a cold-start problem with the multi-armed bandit agent, frequency-based boosting (Marble-B) is used to generate initial candidate entities from the expansion models.

12 . The system of claim 11 , wherein for the frequency-based boosting:

all AI models of the expansion models are executed in order to generate the initial candidate entities; and

in response to executing the all AI models of the expansion to generate the initial candidate entities, the initial candidate entities from the all AI models are captured and one or more most frequent candidate entities from the initial candidate entities are presented to the subject matter expert.

13 . The system of claim 8 , wherein to alleviate a cold-start problem with the multi-armed bandit agent, history-based boosting (Marble-H) is used to generate initial candidate entities from the expansion models.

14 . The system of claim 13 , wherein for the history-based boosting, in response to the subject matter expert selecting, at time t, the first candidate entity from the first AI model from the expansion models, the first AI model is rewarded and further, a second model that also generates the first candidate entity at time t+1, is also rewarded.

15 . A computer program product comprising a memory with one or more computer executable instructions stored thereon, which when executed by one or more processing units cause the one or more processing units to perform a method for selecting an artificial intelligence (AI) model, the method comprising:

receiving, by a multi-armed bandit agent, candidate entities from multiple set expansion models, wherein the multiple set expansion models comprise a first AI model and a second AI model previously trained on different datasets;

selecting a subset of the candidate entities for a first set expansion task, wherein a first candidate entity of the selected subset of candidate entities is selected from the first AI model by a subject matter expert;

selecting, by the multi-armed bandit agent, based on the first candidate entity selected by the subject matter expert, the first AI model from the set expansion models for generating further candidate entities;

in response to the first candidate entity from the first AI model being selected, selecting, without having knowledge of the different datasets and while avoiding knowledge of the training distribution of the different datasets, the second AI model, configured to output the first candidate entity, wherein both the first AI model and the second AI model are configured to output the first candidate entity based on receiving input of a set of positive inputs and a set of negative inputs such that a reward is given to the first AI model and the second AI model for generating the first candidate entity, wherein the set of positive inputs and the set of negative inputs are input to the first AI model and the second AI model after the first AI model and the second AI model have been previously trained on the different datasets; and

causing the first AI model and the second AI model selected to be executed for generating an expansion to a lexicon for natural language processing (NLP).

16 . The computer program product of claim 15 , wherein the expansion models comprise AI models of different types; and

wherein the first AI model outputs a first list comprising the first candidate entity and the second AI model outputs a second list comprising the first candidate entity.

17 . The computer program product of claim 15 , wherein to alleviate a cold-start problem with the multi-armed bandit agent, frequency-based boosting (Marble-B) is used to generate initial candidate entities from the expansion models.

18 . The computer program product of claim 17 , wherein for the frequency-based boosting:

all AI models of the expansion models are executed in order to generate the initial candidate entities; and

in response to executing the all AI models of the expansion to generate the initial candidate entities, the initial candidate entities from the all AI models are captured and one or more most frequent candidate entities from the initial candidate entities are presented to the subject matter expert.

19 . The computer program product of claim 15 , wherein to alleviate a cold-start problem with the multi-armed bandit agent, history-based boosting (Marble-H) is used to generate initial candidate entities from the expansion models.

20 . The computer program product of claim 19 , wherein for the history-based boosting, in response to the subject matter expert selecting, at time t, the first candidate entity from the first AI model from the expansion models, the first AI model is rewarded and further, a second model that also generates the first candidate entity at time t+1, is also rewarded.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 6, 2022
From: WAHED, MUNTASIR; GRUHL, DANIEL
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 061995/0465 →
Continuity (1)
Related Publication 20240184999A1 · Jun 6, 2024
References Cited (69)
US 6188976B1 · Ramaswamy et al. · 2001 [cited by applicant]
US 7805288B2 · Guo et al. · 2010 [cited by applicant]
US 10032448B1 · Baughman et al. · 2018 [cited by applicant]
US 10043511B2 · Baughman et al. · 2018 [cited by applicant]
US 10740379B2 · Alba et al. · 2020 [cited by applicant]
US 11030402B2 · Ristoski et al. · 2021 [cited by applicant]
US 11163952B2 · Gentile et al. · 2021 [cited by applicant]
US 11244116B2 · Bull et al. · 2022 [cited by applicant]
US 11328121B2 · Barbosa et al. · 2022 [cited by applicant]
US 11869490B1 · Gupta · 2024 [cited by examiner]
US 20080250015A1 · Guo et al. · 2008 [cited by applicant]
US 20170220584A1 · Gruhl et al. · 2017 [cited by applicant]
US 20170337185A1 · Barbosa et al. · 2017 [cited by applicant]
US 20180197530A1 · Baughman et al. · 2018 [cited by applicant]
US 20180197531A1 · Baughman et al. · 2018 [cited by applicant]
US 20180225374A1 · Alba et al. · 2018 [cited by applicant]
US 20190197486A1 · Jersin · 2019 [cited by examiner]
US 20200035222A1 · Sypniewski · 2020 [cited by examiner]
US 20210064702A1 · Bull et al. · 2021 [cited by applicant]
US 20220374482A1 · G Rao · 2022 [cited by examiner]
US 20230186037A1 · Lee · 2023 [cited by examiner]
CN 106951526A · 2017 [cited by applicant]
WO WO2022159198A1 · 2022 [cited by examiner]
Kreutzer, Julia, David Vilar, and Artem Sokolov. “Bandits don't follow rules: Balancing multi-facet machine translation with multi-armed bandits.” arXiv preprint arXiv:2110.06997 (2021). (Year: 2021). [cited by examiner]
Chen, Wei, et al. “Combinatorial multi-armed bandit and its extension to probabilistically triggered arms.” Journal of Machine Learning Research 17.50 (2016): 1-33. (Year: 2016). [cited by examiner]
Alba et al, “Multi-lingual Concept Extraction With Linked Data and Human-in-the-Loop”, K-CAP, 2017, 8 pages. [cited by applicant]
Anonymous, “Human-in-the-Loop Set Expansion using Multi-Armed Bandits”, Princeton University, 13 pages. [cited by applicant]
Auer, “Using Confidence Bounds for Exploitation-Exploration Trade-offs”, Journal of Machine Learning Research 3, 2002, pp. 397-422. [cited by applicant]
Auer, et al, “Finite-time Analysis of the Multiarmed Bandit Problem”, Machine Learning, 2002, pp. 235-256. [cited by applicant]
Budd, et al, “A survey on active learning and human-in-the-loop deep learning for medical image analysis”, Elsevier, Medical Image Analysis, 2021, 16 pages. [cited by applicant]
Chakrabarti, et al, “Mortal Multi-Armed Bandits”, Advances in Neural Information Processing Systems, 2008, 8 bages. [cited by applicant]
Chen, et al, “Combinatorial Multi-Armed Bandit: General Framework, Results and Applications”, Proceedings of the 30th International Conference on Machine Learning, vol. 28, 2013, 9 pages. [cited by applicant]
Chen, et al, “Entity Set Expansion with Semantic Features of Knowledge Graphs”, Journal of Web Semantics, 2018, 15 pages. [cited by applicant]
Coden, et al, “Spot the drug! An unsupervised pattern matching method to extract drug names from very large clinical corpora”, 2012 IEEE Second Conference on Healthcare Informatics, Imaging and Systems Biology, 2012, pp… [cited by applicant]
Dooley, et al, “Foodon: a harmonized food ontology to increase global food traceability, quality control and data Integration”, npj Science of Food, 2018, pp. 1-10. [cited by applicant]
Felicio, et al, “A Multi-Armed Bandit Model Selection for Cold-Start User Recommendation”, ACM, 2017, 9 pages. [cited by applicant]
Fouilloy, et al, “Solar irradiation prediction with machine learning: Forecasting models selection method depending on weather variability”, Elsevier, Energy, 165, 2018, pp. 620-629. [cited by applicant]
Gentile, et al. “Explore and Exploit. Dictionary Expansion with Human-in-the-Loop”, European Semantic Web Conference. Springer, Cham, 2019, pp. 131-145. [cited by applicant]
Ghahramani, et al, “Bayesian Sets”, In Proceedings of the 18th International Conference on Neural Information Processing Systems (NIPS), 2005, 8 pages. [cited by applicant]
Gurulingappa, et al, “Development of a Bench Mark Corpus to Support the Automatic Extraction of Drug-Related Adverse Effects from Medical Case Reports”, Journal of Biomedical Informatics, vol. 45, 2012, pp. 885-892. [cited by applicant]
Kohita, et al, “Interactive Construction of User-Centric Dictionary for Text Analytics”, In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 789-799. [cited by applicant]
Lee, et al, “Overcoming Small-size Training Set Problem in Content-Based Recommendation: A Collaboration-based Training Set Expansion Approach”, In Proceedings of the 11th International Conference on Electronic Commerce… [cited by applicant]
Li, et al, “Contrastive Learning with Hard Negative Entities for Entity Set Expansion”, arXiv preprint, 2022, 10 pages. [cited by applicant]
Lika, et al, “Facing the Cold Start Problem in Recommender Systems”, Expert Systems with Applications, vol. 41, 2014, pp. 2065-2073. [cited by applicant]
Liu, et al, “Lexicon Modeling for Query Understanding”, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2011, pp. 5604-5607. [cited by applicant]
Miller, “WordNET: A Lexical Database for English”, Communications of the ACM, vol. 38, 1995, pp. 39-41. [cited by applicant]
Pantel, et al, “Web-scale Distributional Similarity and Entity Set Expansion”, In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2009, pp. 938-947. [cited by applicant]
Rastogi, et al, “Neural Variational Entity Set Expansion for Automatically Populated Knowledge Graphs”, Information Retrieval Journal, 2019, pp. 232-255. [cited by applicant]
Sadamitsu, et al, “Entity Set Expansion Using Interactive Topic Information”, In Proceedings of the 26th Pacific Asia Conference on Language, Information, and Computation, 2012, pp. 108-116. [cited by applicant]
Sarmento, et al, “More like These: Growing Entity Classes from Seeds”, In Proceedings of the Sixteenth ACM Conference on Information and Knowledge Management (CIKM), 2007, pp. 959-962. [cited by applicant]
Seldin, et al, “Evaluation and Analysis of the Performance of the EXP3 Algorithm in Stochastic Environments”, In European Workshop on Reinforcement Learning, JMLR: Workshop and Conference Proceedings, vol. 24, 2012, 14 … [cited by applicant]
Shen, et al, “SynSetExpan: An Iterative Framework for Joint Entity Set Expansion and Synonym Discovery”, arXiv preprint, 2020, 16 pages. [cited by applicant]
Shi, et al, “Entity set expansion in knowledge graph: a heterogeneous information network perspective”, Frontiers of Computer Science, 2021, pp. 1-12. [cited by applicant]
Sutton, et al, “Multi-arm Bandits”, Reinforcement Learning: An Introduction, Chapter 2, MIT press, 2014, pp. 31-52. [cited by applicant]
Thompson, “On the Likelihood That One Unknown Probability Exceeds Another in View of the Evidence of Two Samples”, Biometrika, vol. 25, 1933; pp. 285-294. [cited by applicant]
Velardi, et al, “OntoLearn Reloaded: A Graph-Based Algorithm for Taxonomy Induction”, Association for Computational Linguistics, vol. 39, 2013, pp. 665-707. [cited by applicant]
Wang, et al, “Automatic Set Expansion for List Question Answering”, In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, 2008, pp. 947-954. [cited by applicant]
Wang, et al, “Iterative Set Expansion of Named Entities Using the Web”, I., In 2008 eighth IEEE International Conference on Data Mining (ICDM), 2008; pp. 1091-1096. [cited by applicant]
Wang, et al, “Language-Independent Set Expansion of Named Entities Using the Web”, In Seventh IEEE international conference on data mining (ICDM), 2007, pp. 342-350. [cited by applicant]
Wang, et al, “Semi-Supervised Learning of Semantic Classes for Query Understanding—from the Web and for the Web”, In Proceedings of the 18th ACM Conference on Information and Knowledge Management, 2009, pp. 37-46. [cited by applicant]
Wu, et al, “A Survey of Human-In-The-Loop for Machine Learning”, arXiv preprint, 2022, 22 pages. [cited by applicant]
Yan, et al, “End-to-End Bootstrapping Neural Network for Entity Set Expansion”, In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, 2020, pp. 9402-9409. [cited by applicant]
Yan, et al, “Global Bootstrapping Neural Network for Entity Set Expansion”, In Findings of the Association for Computational Linguistics: EMNLP, 2020, pp. 3705-3714. [cited by applicant]
Yan, et al, “Learning to Bootstrap for Entity Set Expansion”, In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proc… [cited by applicant]
Zhang, et al, “Empower Entity Set Expansion Via Language Model Probing”, arXiv preprint, 2020, 11 pages. [cited by applicant]
Zhang, et al, “Entity set Expansion in Opinion Documents”, In Proceedings of the 22nd ACM conference on Hypertext and Hypermedia, 2011, pp. 281-289. [cited by applicant]
Zhang, et al, “Entity Set Expansion via Knowledge Graphs”, In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2017, 4 pages. [cited by applicant]
Zheng, et al, “Entity Set Expansion with Meta Path in Knowledge Graph”, In Pacific-Asia conference on knowledge discovery and data mining, Springer (PAKDD), 2017, pp. 317-329. [cited by applicant]
Zulkifly, et al, “Improved Machine Learning Model Selection Techniques for Solar Energy Forecasting Applications”, International Journal of Renewable Energy Research (IJRER), vol. 11, 2021, pp. 308-319. [cited by applicant]