Method for linking identifiers to generate a unique entity identifier for deduplicating high-speed data streams in real time
A processor-implemented method for linking identifiers to generate a unique entity identifier for deduplicating high-speed data streams in real time, the method comprising (i) obtaining one or more data streams with an identifier from independently controlled entities, wherein the one or more data streams comprises timestamp data and location indexed data that partially characterizes an activity of an entity, (ii) determining home location or internet protocol address of the entity by analyzing data obtained from the one or more data streams, (iii) clustering entity devices based on an association between an internet protocol address, a real-time event, a period of time or a location, (iv) disambiguating the clusters of entity devices into sub-clusters that resolve to an entity by analyzing data streams until a candidate pair of identifiers is obtained, (v) generating score for the candidate pair using a machine learning classifier to discern the candidate pair of identifiers into to same or different entity, (vi) filtering the candidate pair of identifiers by comparing the score with a predetermined threshold value, (vii) evaluating filtered candidate pair of identifiers to generate a unique entity identifier for the entity, (viii) refining the unique entity identifier in real-time using a feedback loop based on a test engagement activity, and (ix) deduplicating one or more data streams that are associated with the unique entity identifier in real time by validating entity attributes linked with the unique entity identifier with the attributes observed in a first data stream and a second data stream of the test engagement activity.
1. A processor-implemented method for linking device identifiers associated with data streams with a unique entity identifier for an entity that is using devices having the device identifiers, the method comprising:
obtaining, in real time, data streams with at least one device identifier of a device, from independently controlled sources, wherein the data streams comprise timestamp data and location indexed data that partially characterizes an activity of an entity;
generating, using an unsupervised machine learning model, clusters of device identifiers that are observed together in at least one of an internet protocol address, a real-time event, a period of time or a location;
generating a set of candidate pairs of device identifiers from the clusters of device identifiers based on combinations of the device identifiers in the clusters;
generating a similarity score for each of the candidate pairs of device identifiers in the clusters using a machine learning classifier based on a shared history of the device identifiers in the candidate pairs;
comparing the similarity score of the candidate pairs of device identifiers with a predetermined threshold value;
obtaining filtered pairs of device identifiers that have the similarity score greater than the predetermined threshold value, wherein the filtered pairs of device identifiers are determined to refer to the same entity;
generating a unique entity identifier that links at least one of the filtered pairs of device identifiers with the entity;
performing a test engagement activity on each device associated with at least one of the filtered pairs of device identifiers by engaging the entity that is linked with the at least one of the filtered pairs of the device identifiers; and
validating the unique entity identifier in real-time based on a positive engagement in response to the test engagement activity to obtain a validated unique entity identifier for the entity that is linked with the at least one of the filtered pairs of the device identifiers.
2. The method as claimed in claim 1 , wherein the data streams comprises at least one of (i) location pings from a plurality of applications engaged on the plurality of entity devices, (ii) access pings from wireless hot-spots, (iii) active subscriber's data & location in different geo-areas from a mobile network, or (iv) a local information from traffic sensors and public cameras.
3. The method as claimed in claim 1 , wherein the data streams comprises event streams, wherein the event streams include a web traffic to a global website and a public activity on social media services.
4. The method as claimed in claim 1 , wherein the unique entity identifier is refined in real time with respect to incoming data streams, wherein the method reruns if inconsistent data is observed.
5. The method as claimed in claim 1 , wherein personally identifiable information and non-personally identifiable information of the entity are linked with the unique entity identifier.
6. The method as claimed in claim 1 , further comprises linking the unique entity identifier of the entity to the home location of the entity for providing persistence to the unique entity identifier.
7. The method as claimed in claim 1 , further comprises filtering and ranking the candidate pair of identifiers using a data fusion and a Bayesian updating technique based on an additional contextual data corresponding to the entity identifiers and an entity behavior for enabling temporal additivity with changing entity behavior.
8. The method as claimed in claim 1 , wherein a private unique entity identifier graph is created and maintained for a new data source by combining data from the new data source with data associated with the unique entity identifier.
9. The method as claimed in claim 1 , wherein the at least one device identifier comprises at least one of an advertisement identifier, a cookie identifier or a social media identifier.
10. A system for linking device identifiers associated with data streams with a unique entity identifier for an entity that is using devices having the device identifiers, the system comprising:
an identity management server that obtains, in real time, data streams with at least one device identifier of a device, from independently controlled sources, wherein the data streams comprise timestamp data and location indexed data that partially characterizes an activity of an entity, the identity management server further comprising:
a database that stores information including attributes, identifiers, data streams;
a clustering module that generates, using an unsupervised machine learning model, clusters of device identifiers that are observed together in at least one of an internet protocol address, a real-time event, a period of time or a location;
a disambiguation module that (i) generates a set of candidate pairs of device identifiers from the clusters of device identifiers based on combinations of the device identifiers in the clusters, (ii) a similarity score for each of the candidate pairs of device identifiers in the clusters using a machine learning classifier based on a shared history of the device identifiers in the candidate pairs, (iii) comparing the similarity score of the candidate pairs of device identifiers with a predetermined threshold value, and (iv) obtains filtered pairs of device identifiers that have the similarity score greater than the predetermined threshold value, wherein the filtered pairs of device identifiers are determined to refer to the same entity and
a validation module that (i) generates a unique entity identifier that links at least one of the filtered pairs of device identifiers with the entity, (ii) performing a test engagement activity on each device associated with at least one of the filtered pairs of device identifiers by engaging the entity that is linked with the at least one of the filtered pairs of the device identifiers, and (iii) validates the unique entity identifier in real-time based on a positive engagement in response to the test engagement activity to obtain a validated unique entity identifier for the entity that is linked with the at least one of the filtered pairs of the device identifiers.
11. The system as claimed in claim 10 , wherein the data streams comprises at least one of (i) location pings from a plurality of applications engaged on the plurality of entity devices, (ii) access pings from wireless hot-spots, (iii) active subscriber's data & location in different geo-areas from a mobile network, or (iv) a local information from traffic sensors and public cameras.
12. The system as claimed in claim 10 , wherein the data streams comprises event streams, wherein the event streams include a web traffic to a global website and public activity on social media services.
13. The system as claimed in claim 10 , wherein the unique entity identifier is refined in real time with respect to incoming data streams, wherein the system reruns if inconsistent data is observed.
14. The system as claimed in claim 10 , wherein personally identifiable information and non-personally identifiable information of the entity are linked with the unique entity identifier.
15. The system as claimed in claim 10 , wherein the system links the unique entity identifier of the entity to the home location of the entity for providing persistence to the unique entity identifier.
16. The system as claimed in claim 10 , wherein the system filters and ranks candidate pair of identifiers using a data fusion and Bayesian updating technique based on an additional contextual data corresponding to the identifiers and an entity behavior for enabling temporal additivity with changing entity behavior.
17. The system as claimed in claim 10 , wherein a private unique entity identifier graph is created and maintained for a new data source by combining data from the new data source is combined with the data associated with the unique entity identifier.
18. The system as claimed in claim 10 , wherein the at least one device identifier comprises at least one of an advertisement identifier, a cookie identifier or a social media identifier.
19. A one or more non-transitory computer-readable storage mediums storing the one or more sequences of instructions, which when executed by the one or more processors, causes to perform a method for linking device identifiers associated with data streams with a unique entity identifier for an entity that is using devices having the device identifiers, wherein the method comprises:
obtaining, in real time, data streams with at least one device identifier of a device, from independently controlled sources, wherein the data streams comprise timestamp data and location indexed data that partially characterizes an activity of an entity;
generating, using an unsupervised machine learning model, clusters of device identifiers that are observed together in at least one of an internet protocol address, a real-time event, a period of time or a location;
generating a set of candidate pairs of device identifiers from the clusters of device identifiers based on combinations of the device identifiers in the clusters;
generating a similarity score for each of the candidate pairs of device identifiers in the clusters using a machine learning classifier based on a shared history of the device identifiers in the candidate pairs,
comparing the similarity score of the candidate pairs of device identifiers with a predetermined threshold value;
obtaining filtered pairs of device identifiers that have the similarity score greater than the predetermined threshold value, wherein the filtered pairs of device identifiers are determined to refer to the same entity;
generating a unique entity identifier that links at least one of the filtered pairs of device identifiers with the entity;
performing a test engagement activity on each device associated with at least one of the filtered pairs of device identifiers by engaging the entity that is linked with the at least one of the filtered pairs of the device identifiers; and
validating the unique entity identifier in real-time based on a positive engagement in response to the test engagement activity to obtain a validated unique entity identifier for the entity that is linked with the at least one of the filtered pairs of the device identifiers.