Multi-service business platform system having entity resolution systems and methods
The disclosure is directed to various ways of improving the functioning of computer systems, information networks, data stores, search engine systems and methods, and other advantages. Among other things, provided herein are methods, systems, components, processes, modules, blocks, circuits, sub-systems, articles, and other elements (collectively referred to in some cases as the “platform” or the “system”) that collectively enable, in one or more datastores (e.g., where each datastore may include one or more databases) and systems, the creation, development, maintenance, and use of a set of custom objects for use in a wide range of activities, including sales activities, marketing activities, service activities, content development activities, and others, as well as improved methods and systems for sales, marketing and services that make use of such entity resolution systems and methods as well as custom objects.
1 . A method comprising:
monitoring activity of a user accessing entities represented as objects stored within a customer relationship management (CRM) database hosted by a multi-service business platform;
generating a trigger derived from a context of the activity indicating that the user is viewing entity data extracted from the objects within the CRM database; and
in response to the trigger being generated from the user accessing an object within the CRM database and viewing entity data extracted from the object, triggering an on-demand deduplication action to remove duplicate objects from the CRM database, wherein the on-demand deduplication action is performed while the user is accessing the object within the CRM database and includes:
generating, utilizing an embedding model, embeddings for entities represented by the objects stored within the CRM database, wherein the embedding model is used to limit an embedding to a count of numbers less than an embedding size;
generating candidate duplicate entity pairs by processing the embeddings using an approximate nearest neighbors algorithm that includes implementation of a locality sensitive hashing algorithm using configurable parameters with values calculated over a grid of possible locality sensitive hashing algorithm configurations;
generating deduplication probabilities for the candidate duplicate entity pairs;
specifying that a first entity and a second entity represented by a candidate duplicate entity pair are duplicate entities based upon a deduplication probability for the candidate duplicate entity pair exceeding a threshold; and
performing the on-demand deduplication action upon objects, within the CRM database, representing the first entity and the second entity.
2 . The method of claim 1 , wherein the triggering the on-demand deduplication action comprises:
triggering, based upon a determination that a user has accessed an entity within the CRM database, the on-demand deduplication action to identify and display other entities within the CRM database that are identified as being duplicate entities with respect to the entity; and
identifying and displaying the duplicate entities in real-time while the user is accessing the entity.
3 . The method of claim 1 , wherein the triggering the on-demand deduplication action comprises:
triggering, based upon a determination that the context of the activity indicates an entity import operation being performed to import a set of entities into the CRM database, the on-demand deduplication action to identify and display other entities within the CRM database that are identified as being duplicate entities with respect to the set of entities; and
identifying and displaying the duplicate entities in real-time during operation of the entity import operation.
4 . The method of claim 1 , wherein the triggering the on-demand deduplication action comprises:
triggering, based upon a determination that the context of the activity indicates a user interacting with a user interface element of a user interface, the on-demand deduplication action to identify and display the duplicate entities within the user interface.
5 . The method of claim 1 , wherein the triggering the on-demand deduplication action comprises:
triggering, based upon a determination that the context of the activity indicates an update operation being performed upon the CRM database, the on-demand deduplication action to identify and display other entities within the CRM database that are identified as being duplicate entities with respect to a set of entities of the update operation, wherein the duplicate entities are identified and displayed in real-time during operation of the update operation.
6 . The method of claim 1 , wherein the generating the candidate duplicate entity pairs comprises:
implementing the locality sensitive hashing algorithm as the approximate nearest neighbors algorithm to processing the embedding for generating the candidate duplicate entity pairs.
7 . The method of claim 1 , wherein the generating the candidate duplicate entity pairs comprises:
implementing a hierarchical navigable small worlds (HNSW) as the approximate nearest neighbors algorithm to processing the embedding for generating the candidate duplicate entity pairs.
8 . The method of claim 1 , wherein the performing the on-demand deduplication action comprises:
providing a recommendation to a user to merge the first entity with the second entity.
9 . The method of claim 1 , comprising:
setting the embedding size to be a count between 16 numbers and 190 numbers.
10 . The method of claim 1 , comprising:
utilizing a self-supervised training technique to train the embedding model.
11 . The method of claim 1 , comprising:
training the embedding model based upon a training constraint that an entity ID within the CRM database is a duplicate of itself.
12 . The method of claim 1 , comprising:
training the embedding model using a set of unlabeled entities.
13 . The method of claim 1 , comprising:
training the embedding model using a set of unlabeled entities and a set of labeled entities that are labeled as duplicates.
14 . The method of claim 1 , comprising:
training the embedding model as a Siamese network with two identical embedding structures that use weight sharing, wherein a pair of entities that were merged by a user are each passed through one of the embedding structures for comparison.
15 . The method of claim 1 , comprising:
training the embedding model using a mixture of two loss functions corresponding to N-pairs hard mining loss and Barlow-twins loss.
16 . A non-transitory machine readable medium comprising instructions for performing a method, which when executed by a machine, causes the machine to perform operations comprising:
monitoring activity of a user accessing entities represented as objects stored within a customer relationship management (CRM) database hosted by a multi-service business platform;
generating a trigger derived from a context of the activity indicating that the user is viewing entity data extracted from the objects within the CRM database; and
in response to the trigger being generated from the user accessing an object within the CRM database and viewing entity data extracted from the object, triggering an on-demand deduplication action to remove duplicate objects from the CRM database, wherein the on-demand deduplication action is performed while the user is accessing the object within the CRM database and includes:
generating, utilizing the embedding model, embeddings for entities represented by the objects stored within the CRM database, wherein the embedding model is used to limit an embedding to a count of numbers less than an embedding size;
generating candidate duplicate entity pairs by processing the embeddings using an approximate nearest neighbors algorithm that includes implementation of a locality sensitive hashing algorithm using configurable parameters with values calculated over a grid of possible locality sensitive hashing algorithm configurations;
generating deduplication probabilities for the candidate duplicate entity pairs;
specifying, upon based a deduplication probability for a candidate duplicate entity pair exceeding a threshold, that a first entity and a second entity represented by the candidate duplicate entity pair are duplicate entities; and
performing an on-demand deduplication action upon objects, within the CRM database, representing the first entity and the second entity.
17 . The non-transitory machine readable medium of claim 16 , wherein the operations comprise:
generating an embedding with embedding features corresponding to:
a first name having a character tokenization class and a string type;
a last name having the character tokenization class and the string type;
a company name having an ignored class and the string type;
a phone number having the character tokenization class and the string type;
an email having the character tokenization class and the string type; and
a zip code having the ignored class and the string type.
18 . The non-transitory machine readable medium of claim 16 , wherein the operations comprise:
generating an embedding as a set of numbers having an embedding size less than 190.
19 . The non-transitory machine readable medium of claim 16 , wherein the operations comprise:
generating an embedding as a set of numbers having an embedding size of 16.
20 . A computing device comprising
a memory comprising instructions; and
a processor coupled to the memory, the processor configured to execute the instructions to cause the processor to perform operations comprising:
monitoring activity of a user accessing entities represented as objects stored within a customer relationship management (CRM) database hosted by a multi-service business platform;
generating a trigger derived from a context of the activity indicating that the user is viewing entity data extracted from the objects within the CRM database; and
in response to the trigger being generated from the user accessing an object within the CRM database and viewing entity data extracted from the object, triggering an on-demand deduplication action to remove duplicate objects from the CRM database, wherein the on-demand deduplication action is performed while the user is accessing the object within the CRM database and includes:
generating, utilizing an embedding model, embeddings for entities represented by the objects stored within the CRM database, wherein the embedding model is used to limit an embedding to a count of numbers less than an embedding size;
generating candidate duplicate entity pairs by processing the embeddings using an approximate nearest neighbors algorithm that includes implementation of a locality sensitive hashing algorithm using configurable parameters with values calculated over a grid of possible locality sensitive hashing algorithm configurations;
generating deduplication probabilities for the candidate duplicate entity pairs;
specifying, based upon a deduplication probability for a candidate duplicate entity pair exceeding a threshold, that a first entity and a second entity represented by the candidate duplicate entity pair are duplicate entities; and
performing the on-demand deduplication action upon objects, within the CRM database, representing the first entity and the second entity.
21 . The computing device of claim 20 , wherein the operations comprise:
identifying and providing a correlation between the first entity and second entity.