Incremental clustering
Incremental clustering of similar or related messages that otherwise requires limited use of memory for the purpose of increasing scalability. Incremental clustering includes receiving, by a machine learning (ML) engine, an incoming message from an application programming interface (API) server, and scanning, by the ML engine, a plurality of clusters for one or more messages similar to that of the incoming message. Incremental clustering also includes identifying, by a clustering engine, a cluster from the plurality of clusters. The identified cluster includes the one or more messages similar to that of the incoming message. Incremental clustering further includes assigning, by the clustering engine, the incoming message to the identified cluster.
1. A computer-implemented method for clustering incoming messages, comprising:
receiving, by a machine learning (ML) engine, an incoming message from an application programming interface (API) server;
scanning, by the ML engine, a plurality of clusters for one or more messages having similar attributes or features to that of the incoming message, wherein each of the plurality of clusters comprises of the one or more messages, each of which have similar attributes or features to one another;
retrieving, by the ML engine, a nearest leader from √N leader, where N is a total number of messages in a cluster database and √N leader is a square root of the total number of messages;
performing, by the ML engine, a cosine similarity between the incoming message and one of the plurality of clusters;
performing, by the ML engine, a subsequent cosine similarity between incoming message and one or more messages within √N leader to identify a nearest message and a distance between the nearest message and the incoming message; and
identifying, by a clustering engine, a cluster from the plurality of clusters, wherein the identified cluster comprises the one or more messages similar to that of the incoming message; and
assigning, by the clustering engine, the incoming message to the identified cluster.
2. The computer implemented method of claim 1 , further comprising:
scanning, by the ML engine, each of the plurality of clusters for one or more messages duplicate to that of the incoming message,
returning, by the ML engine, a cluster identification to the API server, when a duplicate message to that of the incoming message is found in a cluster database.
3. The computer-implemented method of claim 1 , further comprising:
when the distance between the nearest message and the incoming message is 1, returning, by the ML engine, a cluster identification to the API server.
4. The computer-implemented method of claim 1 , further comprising:
when the distance between the nearest message and the incoming message is 0,
forming, by the ML engine, a new cluster and assigning the incoming message to the new cluster, and
returning a cluster identification for the newly formed cluster to the API server, or
when the distance between the nearest message and the incoming message between 1 and 0,
adding, by the ML engine, the incoming message to an existing cluster, and
returning a cluster identification for the existing cluster to the API server.
5. The computer-implemented method of claim 1 , further comprising:
searching, by the ML engine, for one or more similar clusters to that of a current cluster, wherein the current cluster is incorrectly identified; and
when a similar cluster is identified, notifying, by the ML engine, an agent of the identified similar cluster to merge the current cluster with the identified similar cluster, or
when a similar cluster is not identified, causing, by the ML engine, the clustering engine to predict a next possible cluster from the plurality of clusters in a cluster database and send the next possible cluster to the agent for feedback.
6. The computer-implemented method of claim 1 , further comprising:
receiving, by the clustering engine, positive or negative feedback from an agent; and
when the feedback received is positive, marking, by the clustering agent, the incoming message as being mapped, or
when the feedback received is negative, determining if the cluster being predicted has been predicted a predetermined number of times prior to receiving the negative feedback.
7. The computer-implemented method of claim 6 , further comprising:
when the cluster is predicted for the predetermined number of times, creating, by the clustering engine, a new cluster and assign the incoming message to the new cluster.
8. The computer-implemented method of claim 6 , further comprising:
when the cluster predicted is less than the predetermined number of times, predicting, by the clustering engine, a next possible cluster for the incoming message, and
when the predicted cluster is not predicted before, sending, by the cluster engine, cluster assignment information to the agent for feedback, or
when the predicted cluster is predicted before, predicting, by the clustering engine, another cluster for the incoming message.
9. An apparatus configured to cluster incoming messages, comprising:
at least one processor; and
memory comprising a set of instructions, wherein
the set of instructions are configured to cause the at least one processor to execute:
receiving, by a machine learning (ML) engine, an incoming message from an application programming interface (API) server;
scanning, by the ML engine, a plurality of clusters for one or more having messages similar attributes or features to that of the incoming message, wherein each of the plurality of clusters comprises of the one or more messages, each of which have similar attributes or features to one another;
retrieving, by the ML engine, a nearest leader from √N leader, where N is a total number of messages in a cluster database and √N leader is a square root of the total number of messages;
performing, by the ML engine, a cosine similarity between the incoming message and one of the plurality of clusters;
performing, by the ML engine, a subsequent cosine similarity between incoming message and one or more messages within √N leader to identify a nearest message and a distance between the nearest message and the incoming message; and
identifying, by a clustering engine, a cluster from the plurality of clusters, wherein the identified cluster comprises the one or more messages similar to that of the incoming message; and
assigning, by the clustering engine, the incoming message to the identified cluster.
10. The apparatus of claim 9 , wherein the set of instructions are further configured to cause at least one processor to execute:
scanning, by the ML engine, each of the plurality of clusters for one or more messages duplicate to that of the incoming message,
returning, by the ML engine, a cluster identification to the API server, when a duplicate message to that of the incoming message is found in a cluster database.
11. The apparatus of claim 9 , wherein the set of instructions are further configured to cause at least one processor to execute:
when the distance between the nearest message and the incoming message is 1, returning, by the ML engine, a cluster identification to the API server.
12. The apparatus of claim 9 , wherein the set of instructions are further configured to cause at least one processor to execute:
when the distance between the nearest message and the incoming message is 0,
forming, by the ML engine, a new cluster and assigning the incoming message to the new cluster, and
returning a cluster identification for the newly formed cluster to the API server, or
when the distance between the nearest message and the incoming message between 1 and 0,
adding, by the ML engine, the incoming message to an existing cluster, and
returning a cluster identification for the existing cluster to the API server.
13. The apparatus of claim 9 , wherein the set of instructions are further configured to cause at least one processor to execute:
searching, by the ML engine, for one or more similar clusters to that of a current cluster, wherein the current cluster is incorrectly identified; and
when a similar cluster is identified, notifying, by the ML engine, an agent of the identified similar cluster to merge the current cluster with the identified similar cluster, or
when a similar cluster is not identified, causing, by the ML engine, the clustering engine to predict a next possible cluster from the plurality of clusters in a cluster database and send the next possible cluster to the agent for feedback.
14. The apparatus of claim 9 , wherein the set of instructions are further configured to cause at least one processor to execute:
receiving, by the clustering engine, positive or negative feedback from an agent; and
when the feedback received is positive, marking, by the clustering agent, the incoming message as being mapped, or
when the feedback received is negative, determining if the cluster being predicted has been predicted a predetermined number of times prior to receiving the negative feedback.
15. The apparatus of claim 14 , wherein the set of instructions are further configured to cause at least one processor to execute:
when the cluster is predicted for the predetermined number of times, creating, by the clustering engine, a new cluster and assign the incoming message to the new cluster.
16. The apparatus of claim 14 , wherein the set of instructions are further configured to cause at least one processor to execute:
when the cluster predicted is less than the predetermined number of times, predicting, by the clustering engine, a next possible cluster for the incoming message, and
when the predicted cluster is not predicted before, sending, by the cluster engine, cluster assignment information to the agent for feedback, or
when the predicted cluster is predicted before, predicting, by the clustering engine, another cluster for the incoming message.
17. A system configured to cluster incoming messages, comprising:
an application programming interface (API) server configured to receive an incoming message from one or more service application, and route the incoming message for clustering;
a machine learning (ML) module configured to receive the incoming message from the API server, and scan a plurality of clusters for one or more messages having similar attributes or features to that of the incoming message, wherein the plurality of clusters comprise of the one or more messages, each of which have similar attributes or features to one another,
wherein the ML module is further configured to
retrieve a nearest leader from √N leader, where N is a total number of messages in a cluster database and √N leader is a square root of the total number of messages;
perform a cosine similarity between the incoming message and one of the plurality of clusters, and
perform a subsequent cosine similarity between incoming message and one or more messages within √N leader to identify a nearest message and a distance between the nearest message and the incoming message; and
a clustering module configured to identify a cluster from the plurality of clusters, wherein the identified cluster comprises the one or more messages similar to that of the incoming message, and assign the incoming message to the identified cluster.