IP Library Granted Patent US 12,591,813
Granted Patent B1
US 12,591,813 · App. 19/274,355 · Granted Mar 31, 2026

Using machine learning models to cause operator actions

Inventors: Ramnath Mahale (Chadds Ford, PA); Promiti Dutta (New York, NY); Thomas Joseph Snyder (Hoboken, NJ); Murli D. Buluswar (Tenafly, NJ)
Assignee: Citibank, N.A.
G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,591,813
App. No.
19/274,355
Granted
Mar 31, 2026
Kind
B1
Abstract

Methods and systems are described for utilizing machine learning models to generate a response. The system receives an operator input requesting actions to prompt a desired user interaction. The system selects user-specific machine learning models trained on interactions associated with each user of a plurality of users. Based on the operator input, the system determines a model subset and selects a candidate operator action to test. The system inputs the candidate action into each model of the subset to obtain a predicted user interaction. The system then assesses whether the candidate action causes the models to predict the desired interaction. If the candidate action is effective, the system executes the candidate action in relation to the user subset.

Claims (75)

1 . A system for utilizing a plurality of machine learning models to generate a response, comprising:

a storage device; and

one or more processors in a network system communicatively coupled to the storage device, wherein the one or more processors execute instructions that are stored in the storage device to cause the system to:

receive an operator input for one or more actions that cause a user interaction to be performed;

select, based on the operator input, a plurality of user-specific machine learning models, wherein each of the plurality of user-specific machine learning models is trained based on user interactions received as real time data from a corresponding user of a plurality of users and corresponding actions associated with the user interactions;

select, based on the operator input, a model subset of the plurality of user-specific machine learning models, wherein the model subset corresponds to a user subset of the plurality of users;

select a candidate operator action for the user subset;

input, into each user-specific machine learning model of the model subset of the plurality of user-specific machine learning models, the candidate operator action to cause each user-specific machine learning model of the model subset to predict a corresponding user interaction, wherein inputting the candidate operator action causes each user-specific machine learning model of the model subset to simulate execution of the candidate operator action within each user-specific machine learning model in real-time without actual execution;

determine whether the candidate operator action causes at least a threshold number of models of the model subset to predict the user interaction;

based on determining that the candidate operator action causes at least the threshold number of models of the model subset to predict the user interaction, generate, based on the candidate operator action, one or more instructions for executing the candidate operator action in relation to the user subset; and

execute the candidate operator action in relation to each user of the user subset.

2 . The system of claim 1 , wherein the instructions for selecting, based on the operator input, the model subset of the plurality of user-specific machine learning models further cause the one or more processors to:

identify, within the operator input, one or more parameters representing one or more user characteristics of the plurality of users;

match the one or more parameters with user parameters associated with the plurality of users; and

select the user subset based on the user parameters that match the one or more parameters.

3 . The system of claim 1 , wherein the instructions for selecting the candidate operator action for the user subset further cause the one or more processors to:

retrieve, from a training dataset used to train one or more machine learning models of the plurality of user-specific machine learning models, a plurality of operator actions leading to a plurality of user actions; and

select the candidate operator action from the plurality of operator actions.

4 . The system of claim 1 , wherein the instructions further cause the one or more processors to receive, from each user-specific machine learning model of the model subset, the corresponding user interaction determined by each user-specific machine learning model.

5 . The system of claim 4 , wherein the instructions for determining whether the candidate operator action causes at least the threshold number of models of the model subset to predict the user interaction further cause the one or more processors to:

determine, from each corresponding user interaction predicted by each user-specific machine learning model, a number of user interactions that match the user interaction received with the operator input; and

determine whether the candidate operator action causes the model subset of the plurality of user-specific machine learning models to predict the user interaction based on the number of user interactions that match.

6 . The system of claim 1 , wherein the instructions for selecting, based on the operator input, the plurality of user-specific machine learning models further cause the one or more processors to identify user-specific machine learning models where a target feature corresponds to the user interaction within the operator input.

7 . The system of claim 1 , wherein the instructions further cause the one or more processors to:

based on determining that the candidate operator action does not cause at least the threshold number of models of the model subset to predict the user interaction:

select a second candidate operator action from a plurality of operator actions;

input, into each user-specific machine learning model of the model subset of the plurality of user-specific machine learning models, the second candidate operator action to cause each user-specific machine learning model of the model subset to predict the corresponding user interaction;

determine that the second candidate operator action causes at least the threshold number of models of the model subset to predict the user interaction; and

generate, based on the second candidate operator action, one or more commands for executing the candidate operator action in relation to the user subset.

8 . A method for utilizing a plurality of machine learning models to generate a response, the method comprising:

receiving an operator input for one or more actions that cause a user interaction to be performed;

selecting, based on the operator input, a plurality of user-specific machine learning models, wherein each of the plurality of user-specific machine learning models is trained based on user interactions associated with a corresponding user of a plurality of users;

selecting, based on the operator input, a model subset of the plurality of user-specific machine learning models;

selecting a candidate operator action for a user subset of the plurality of users corresponding to the model subset;

inputting, into each user-specific machine learning model of the model subset of the plurality of user-specific machine learning models, the candidate operator action to cause each user-specific machine learning model of the model subset to predict a corresponding user interaction, wherein inputting the candidate operator action causes each user-specific machine learning model of the model subset to simulate execution of the candidate operator action within each user-specific machine learning model in real-time without actual execution;

determining whether the candidate operator action causes at least a threshold number of models of the model subset to predict the user interaction; and

based on determining that the candidate operator action causes at least the threshold number of models of the model subset to predict the user interaction, executing the candidate operator action in relation to the user subset.

9 . The method of claim 8 , wherein selecting, based on the operator input, the model subset of the plurality of user-specific machine learning models further comprises:

identifying, within the operator input, one or more parameters representing one or more user characteristics of the plurality of users;

matching the one or more parameters with user parameters associated with the plurality of users; and

selecting the user subset based on the user parameters that match the one or more parameters.

10 . The method of claim 8 , wherein selecting the candidate operator action for the user subset further comprises:

retrieving, from a training dataset used to train one or more machine learning models of the plurality of user-specific machine learning models, a plurality of operator actions leading to a plurality of user actions; and

selecting the candidate operator action from the plurality of operator actions.

11 . The method of claim 8 , further comprising receiving, from each user-specific machine learning model of the model subset, the corresponding user interaction determined by each user-specific machine learning model.

12 . The method of claim 11 , wherein determining whether the candidate operator action causes at least the threshold number of models of the model subset to predict the user interaction further comprises:

determining, from each corresponding user interaction predicted by each user-specific machine learning model, a number of user interactions that match the user interaction received with the operator input; and

determining whether the candidate operator action causes the model subset of the plurality of user-specific machine learning models to output the user interaction based on the number of user interactions that match.

13 . The method of claim 8 , wherein selecting, based on the operator input, the plurality of user-specific machine learning models further comprises identifying user-specific machine learning models where a target feature corresponds to the user interaction within the operator input.

14 . The method of claim 8 , further comprising:

based on determining that the candidate operator action does not cause at least the threshold number of models of the model subset to predict the user interaction:

selecting a second candidate operator action from a plurality of operator actions;

inputting, into each user-specific machine learning model of the model subset of the plurality of user-specific machine learning models, the second candidate operator action to cause each user-specific machine learning model of the model subset to predict the corresponding user interaction;

determining that the second candidate operator action causes at least the threshold number of models of the model subset to predict the user interaction; and

generating, based on the second candidate operator action, one or more instructions for executing the candidate operator action in relation to the user subset.

15 . One or more non-transitory computer-readable media storing instructions thereon, wherein the instructions cause one or more processors to perform operations comprising:

receiving an operator input for one or more actions that cause a user interaction to be performed;

selecting, based on the operator input, a plurality of user-specific machine learning models, wherein each of the plurality of user-specific machine learning models is trained based on user interactions associated with a corresponding user of a plurality of users;

selecting, based on the operator input, a model subset of the plurality of user-specific machine learning models;

selecting a candidate operator action for a user subset of the plurality of users corresponding to the model subset;

inputting, into each user-specific machine learning model of the model subset of the plurality of user-specific machine learning models, the candidate operator action to cause each user-specific machine learning model of the model subset to predict a corresponding user interaction, wherein inputting the candidate operator action causes each user-specific machine learning model of the model subset to simulate execution of the candidate operator action within each user-specific machine learning model in real-time without actual execution;

determining whether the candidate operator action causes at least a threshold number of models of the model subset to predict the user interaction; and

based on determining that the candidate operator action causes at least the threshold number of models of the model subset to predict the user interaction, executing the candidate operator action in relation to the user subset.

16 . The one or more non-transitory computer-readable media of claim 15 , wherein the instructions for selecting, based on the operator input, the model subset of the plurality of user-specific machine learning models further cause the one or more processors to perform operations comprising:

identifying, within the operator input, one or more parameters representing one or more user characteristics of the plurality of users;

matching the one or more parameters with user parameters associated with the plurality of users; and

selecting the user subset based on the user parameters that match the one or more parameters.

17 . The one or more non-transitory computer-readable media of claim 15 , wherein the instructions for selecting the candidate operator action for the user subset further cause the one or more processors to perform operations comprising:

retrieving, from a training dataset used to train one or more machine learning models of the plurality of user-specific machine learning models, a plurality of operator actions leading to a plurality of user actions; and

selecting the candidate operator action from the plurality of operator actions.

18 . The one or more non-transitory computer-readable media of claim 15 , wherein the instructions further cause the one or more processors to receive, from each user-specific machine learning model of the model subset, the corresponding user interaction determined by each user-specific machine learning model.

19 . The one or more non-transitory computer-readable media of claim 18 , wherein the instructions for determining whether the candidate operator action causes at least the threshold number of models of the model subset to predict the user interaction further cause the one or more processors to perform operations comprising:

determining, from each corresponding user interaction predicted by each user-specific machine learning model, a number of user interactions that match the user interaction received with the operator input; and

determining whether the candidate operator action causes the model subset of the plurality of user-specific machine learning models to output the user interaction based on the number of user interactions that match.

20 . The one or more non-transitory computer-readable media of claim 15 , wherein the instructions for selecting, based on the operator input, the plurality of user-specific machine learning models further cause the one or more processors to identify user-specific machine learning models where a target feature corresponds to the user interaction within the operator input.

Continuity (1)
Continuation In Part 17541975 · Dec 3, 2021
References Cited (21)
US 6584464B1 · Warthen · 2003 [cited by applicant]
US 7974869B1 · Sharma et al. · 2011 [cited by applicant]
US 10990988B1 · Goyette et al. · 2021 [cited by applicant]
US 11900451B1 · King · 2024 [cited by examiner]
US 20040103017A1 · Reed et al. · 2004 [cited by applicant]
US 20130055137A1 · Choc et al. · 2013 [cited by applicant]
US 20140046757A1 · Kahn et al. · 2014 [cited by applicant]
US 20150052503A1 · Ligman et al. · 2015 [cited by applicant]
US 20150161652A1 · Schnabl et al. · 2015 [cited by applicant]
US 20160005072A1 · Robinson · 2016 [cited by applicant]
US 20160210656A1 · Chittilappilly et al. · 2016 [cited by applicant]
US 20180101771A1 · Schwarm et al. · 2018 [cited by applicant]
US 20180240145A1 · Zargham et al. · 2018 [cited by applicant]
US 20190164176A1 · Pydynowski et al. · 2019 [cited by applicant]
US 20190266621A1 · Eckman et al. · 2019 [cited by applicant]
US 20190378074A1 · Mcphatter et al. · 2019 [cited by applicant]
US 20200273052A1 · Ganti et al. · 2020 [cited by applicant]
US 20220070050A1 · D'ippolito et al. · 2022 [cited by applicant]
US 20240039882A1 · Varma · 2024 [cited by examiner]
WO 2021038759A1 · 2021 [cited by applicant]
Bohanec et al., “Explaining Machine Learning Models in Sales Predictions”, Expert Systems with Applications 71 (2017) 416-428. (Year: 2017). [cited by applicant]