Automatic alert dispositioning using artificial intelligence
A system for identifying false positives about a user and closing resulting alerts before the alert consumes system resources may use machine learning techniques including clustering and multi-labeling classification to effectively categorize prior text-based notes. The categorized notes may be used to identify and automatically close false positive alerts. Moreover, weak labeling, AI transformers/sentence transformation, and/or k-means cluster analysis may be used to condense large quantities of textual data into ML model components with improved interpretability. Customer relationship management (CRM) platform and Risk Management Supervision (RMS) note analysis captures past work and leverages it to reduce redundancies in future expert user/supervisory/customer advisory efforts. The various ML techniques disclosed herein output numerical features for improved alert triaging.
1 . A method for automated triaging of alerts generated for customer service advisors to reduce false positive alerts using artificial intelligence (AI) sentence transformers, weak labeling in machine learning, and multi-label classification of textual notes about interactions between the customer service advisors and customers, the method comprising:
receiving as input, from a customer relationship management (CRM) platform, CRM notes that comprise text related to a customer;
receiving a plurality of user-defined questions relating to each of an enumerated list of scenarios;
generating, using an AI sentence transformer, a matrix with CRM note-question pairs that captures relevance of each CRM note to a first scenario in the enumerated list of scenarios, wherein the pairs are structured as input embeddings;
aggregating, using a weak labeling system, results by applying user-defined labeling functions to the CRM notes, wherein the aggregated results provide a holistic weak label for each CRM note;
training a discriminative model using heuristics labels generated by the weak labeling system after the aggregating step;
updating, based on an output of the AI sentence transformer and the discriminative model, a machine learning (ML) model, which was trained based on k-means clustering outputs from a text-frequency matrix created from RMS notes, wherein the ML model was created by steps comprising:
filtering keywords in RMS notes to remove a first set of keywords that fail to correspond to any specific scenario among the enumerated list of scenarios; and
after the filtering of the keywords, transforming the RMS notes into a text-frequency matrix, wherein the first set of keywords are omitted from the text-frequency matrix, and wherein each row of the text-frequency matrix represents a different RMS note; and
suppressing, by the ML model, an alert about the customer based on a likelihood that the alert corresponding to the customer is false positive, wherein the alert is for a customer with a likelihood of satisfying the first scenario from among the enumerated list of scenarios,
wherein unsuppressed alerts are displayed on a graphical user interface (GUI) to a customer service advisor responsible for servicing the customer.
2 . The method of claim 1 , wherein the ML model:
infers a likelihood that the customer was previously notified of the first scenario by clustering numerical representations of the RMS notes, which represent interactions between the customer service advisor and the customer, in relation to the enumerated list of scenarios; and
outputs a multi-classification relevance score associated with the customer, wherein the multi-classification relevance score comprises a relevance score corresponding to the first scenario, and wherein each relevance score in the multi-classification relevance score is calculated as a distance from a corresponding numerical representation to a centroid of a corresponding cluster.
3 . The method of claim 1 , wherein the ML model was created by further steps comprising:
reducing a dimension of the text-frequency matrix, wherein the text-frequency matrix is outputted as numerical representations;
k-means clustering on the numerical representations outputted by the text-frequency matrix to produce a plurality of clusters, each cluster corresponding to a single scenario in the enumerated list of scenarios; and
for each RMS note, outputting a multi-classification relevance score that includes a quantity of scores matching a quantity of clusters in the plurality of clusters, wherein each score in the multi-classification relevance score is calculated as a distance from the corresponding numerical representation to a centroid of a corresponding cluster, wherein the multi-classification relevance score corresponds to a likelihood that a particular RMS note belongs to a specific scenario.
4 . The method of claim 1 , further comprising: tagging a CRM note as corresponding to the first scenario of the enumerated list of scenarios based on the output of the AI sentence transformer.
5 . The method of claim 1 , wherein the user-defined labeling functions comprise labeling based on length of a note and presence of particular keywords.
6 . The method of claim 1 , wherein the aggregating is performed by calculating an unweighted average.
7 . The method of claim 1 , further comprising: outputting an accuracy of a labeling function of the user-defined labeling functions.
8 . The method of claim 1 , wherein the customer service advisor includes an AI-chat bot.
9 . The method of claim 1 , wherein the plurality of user-defined questions used with the AI sentence transformer correspond to features for feature engineering of the ML model that predicts the likelihood that the alert corresponding to the customer is false positive.
10 . The method of claim 1 , wherein the aggregating of the results includes a majority vote of the user-defined labeling functions.
11 . The method of claim 1 , wherein the likelihood that the alert corresponding to the customer is false positive further comprises: the ML model taking as inputs prior occurrences of alerts, total assets of the customer, recent losses of the customer, types of securities traded by the customer, and birthdate of the customer.
12 . A non-transitory computer-readable medium storing computer-executable instruction, which when executed by a computer processor, cause the computer processor to perform steps comprising:
receiving as input, from a customer relationship management (CRM) platform, CRM notes that comprise text related to a customer;
receiving a plurality of user-defined questions relating to each of an enumerated list of scenarios;
generating, using an AI sentence transformer, a matrix with CRM note-question pairs that captures relevance of each CRM note to a first scenario in the enumerated list of scenarios, wherein the pairs are structured as input embeddings;
aggregating, using a weak labeling system, results by applying user-defined labeling functions to the CRM notes, wherein the aggregated results provide a holistic weak label for each CRM note;
training a discriminative model using heuristics labels generated by the weak labeling system after the aggregating step;
updating, based on an output of the AI sentence transformer and the discriminative model, a machine learning (ML) model, which was trained based on k-means soft clustering outputs from a text-frequency matrix created from RMS notes, wherein the ML model was created by steps comprising:
filtering keywords in RMS notes to remove a first set of keywords that fail to correspond to any specific scenario among the enumerated list of scenarios; and
after the filtering of the keywords, transforming the RMS notes into a text-frequency matrix, wherein the first set of keywords are omitted from the text-frequency matrix, and wherein each row of the text-frequency matrix represents a different RMS note; and
suppressing, by the ML model, an alert about the customer based on a likelihood that the alert corresponding to the customer is false positive, wherein the alert is for a customer with a likelihood of satisfying the first scenario from among the enumerated list of scenarios,
wherein unsuppressed alerts are displayed on a graphical user interface (GUI) to a customer service advisor responsible for servicing the customer.
13 . The non-transitory computer-readable medium of claim 12 , wherein the ML model:
infers a likelihood that the customer was previously notified of the first scenario by clustering numerical representations of the RMS notes, which represent interactions between the customer service advisor and the customer, in relation to the enumerated list of scenarios; and
outputs a multi-classification relevance score associated with the customer, wherein the multi-classification relevance score comprises a relevance score corresponding to the first scenario, and wherein each relevance score in the multi-classification relevance score is calculated as a distance from a corresponding numerical representation to a centroid of a corresponding cluster.
14 . The non-transitory computer-readable medium of claim 12 , further comprising: tagging a CRM note as corresponding to the first scenario of the enumerated list of scenarios based on the output of the AI sentence transformer, and wherein the plurality of user-defined questions used with the AI sentence transformer correspond to features for feature engineering of the ML model that predicts the likelihood that the alert corresponding to the customer is false positive.
15 . The non-transitory computer-readable medium of claim 12 , wherein the user-defined labeling functions comprise labeling based on length of a note and presence of particular keywords, and the computer-executable instructions, which when executed by the computer processor, cause the computer processor to perform steps comprising: outputting an accuracy of a labeling function of the user-defined labeling functions.
16 . The non-transitory computer-readable medium of claim 12 , wherein the aggregating of the results includes a majority vote of the user-defined labeling functions.
17 . A method for automated triaging of alerts generated for customer service advisors to reduce false positive alerts, the method comprising:
receiving as input, from a customer relationship management (CRM) platform, CRM notes that comprise text related to a customer;
aggregating, using a weak labeling system, results by applying user-defined labeling functions to the CRM notes, wherein the aggregated results provide a holistic weak label for each CRM note;
training a discriminative model using heuristics labels generated by the weak labeling system after the aggregating step;
updating, based on an output of the discriminative model and an output of an AI sentence transformer, a machine learning (ML) model, which was trained based on k-means clustering outputs from a text-frequency matrix created from RMS notes,
receiving a plurality of user-defined questions relating to each enumerated scenario of an enumerated list of scenarios; and
generating, using the AI sentence transformer, a matrix with CRM note-question pairs that captures relevance of each CRM note to a first scenario in the enumerated list of scenarios; and
suppressing, by the ML model, an alert about the customer based on a likelihood that the alert corresponding to the customer is false positive, wherein the alert is for a customer with a likelihood of satisfying the first scenario from among the enumerated list of scenarios,
wherein unsuppressed alerts are displayed on a graphical user interface (GUI) to a customer service advisor responsible for servicing the customer.
18 . The method of claim 17 , wherein the ML model:
infers a likelihood that the customer was previously notified of the first scenario by clustering numerical representations of the RMS notes, which represent interactions between the customer service advisor and the customer, in relation to the enumerated list of scenarios; and
outputs a multi-classification relevance score associated with the customer, wherein the multi-classification relevance score comprises a relevance score corresponding to the first scenario, and wherein each relevance score in the multi-classification relevance score is calculated as a distance from a corresponding numerical representation to a centroid of a corresponding cluster,
and the ML model was created by steps comprising:
filtering keywords in RMS notes to remove a first set of keywords that fail to correspond to any specific scenario among the enumerated list of scenarios;
after the filtering of the keywords, transforming the RMS notes into a text-frequency matrix, wherein the first set of keywords are omitted from the text-frequency matrix, and wherein each row of the text-frequency matrix represents a different RMS note;
reducing a dimension of the text-frequency matrix, wherein the text-frequency matrix is outputted as numerical representations;
k-means clustering on the numerical representations outputted by the text-frequency matrix to produce a plurality of clusters, each cluster corresponding to a single scenario in the enumerated list of scenarios; and
for each RMS note, outputting a multi-classification relevance score that includes a quantity of scores matching a quantity of clusters in the plurality of clusters, wherein each score in the multi-classification relevance score is calculated as a distance from the corresponding numerical representation to a centroid of a corresponding cluster, wherein the multi-classification relevance score corresponds to a likelihood that a particular RMS note belongs to a specific scenario.