Methods and systems for associating internet devices
A data processing system performs data processing of raw or preprocessed data. In some embodiments, the data processing system includes a connectivity overlay engine comprising a data ingester, a connectivity generator, an event access control system, and a feature vector generation framework.
1 . A system comprising:
one or more hardware computing device processors configured to:
receive, from a data source, raw or preprocessed data,
process the raw or preprocessed data received from the data source, thereby resulting in processed data,
filter, using a data filter, the processed data, thereby resulting in filtered data,
normalize and sample, using a normalization and sampling operation, the processed data or the filtered data, thereby resulting in output data, and
transmit the output data to a data store;
a connectivity generation system configured to generate a connectivity overlay comprising an intra-device graph and an inter-device graph;
an event access control system configured to:
receive data from the data store, and
generate an event set, wherein the event set is based at least in part on the data received from the data store and at least one first rule; and
a feature vector generation system configured to:
determine, using a machine learning model, at least one candidate computing device pair, wherein the at least one candidate computing device pair is based at least in part on a first computing device activity history associated with a first computing device comprised in the at least one candidate computing device pair, and a second computing device activity history associated with a second computing device comprised in the at least one candidate computing device pair,
apply at least one second rule to the at least one candidate computing device pair, and
generate, based on the apply the at least one second rule to the at least one candidate computing device pair, one or more feature vectors associated with the at least one candidate computing device pair,
wherein a first feature vector comprised in the one or more feature vectors associated with the at least one candidate computing device pair comprises a first confidence result and a second confidence result, the first confidence result being used to generate the inter-device graph, and the second confidence result being used to generate the intra-device graph,
wherein the machine learning model is trained based on a set of confidence results associated with a set of candidate computing device pairs, wherein a third confidence result comprised in the set of confidence results is mapped to a first candidate computing device pair comprised in the set of candidate computing device pairs.
2 . A system comprising:
one or more hardware computing device processors;
one or more memory systems comprising code, executable by the one or more hardware computing device processors, and configured to:
receive, from a data source, raw or preprocessed data,
process the raw or preprocessed data received from the data source, thereby resulting in processed data,
filter, using a data filter, the processed data, thereby resulting in filtered data,
normalize and sample, using a normalization and sampling operation, the processed data or the filtered data, thereby resulting in output data,
transmit the output data to a data store,
generate, using a connectivity generator, a connectivity overlay,
receive first data from the data store,
generate, using an event access control operation, an event set, wherein the event set is based at least in part on the first data received from the data store and at least one first rule,
determine, using a machine learning model, at least one candidate computing device pair, wherein the at least one candidate computing device pair is based at least in part on a first computing device activity history associated with a first computing device comprised in the at least one candidate computing device pair, and a second computing device activity history associated with a second computing device comprised in the at least one candidate computing device pair,
apply at least one second rule to the at least one candidate computing device pair, and
generate, based on applying the at least one second rule to the at least one candidate computing device pair, one or more feature vectors associated with the at least one candidate computing device pair,
wherein a first feature vector comprised in the one or more feature vectors associated with the at least one candidate computing device pair comprises a first confidence result and a second confidence result, the first confidence result being used to generate an inter-device graph, and the second confidence result being used to generate an intra-device graph,
wherein the machine learning model is trained based on a set of confidence results associated with a set of candidate computing device pairs, wherein a third confidence result comprised in the set of confidence results is mapped to a first candidate computing device pair comprised in the set of candidate computing device pairs.
3 . The system of claim 2 , wherein the one or more feature vectors associated with the at least one candidate computing device pair comprises a fourth confidence result and a fifth confidence result, the fourth confidence result being used to generate the inter-device graph, and the fifth confidence result being used to generate the intra-device graph.
4 . The system of claim 2 , further comprising determining a data structure for storing information identifying computing device identifiers associated with different computing devices or users, wherein the computing device identifiers are used to target information to the different computing devices or users.
5 . The system of claim 2 , wherein at least one of the raw or preprocessed data, the processed data, the filtered data, or the output data, is used to associate computing devices or computing device identifiers with one or more network source or destination address identifiers.
6 . The system of claim 5 , wherein the code comprised in the one or more memory systems is further configured to identify computing device pairs based at least in part on the computing device identifiers or the one or more network source or destination address identifiers.
7 . The system of claim 2 , wherein the code comprised in the one or more memory systems is further configured to produce the one or more feature vectors associated with the at least one candidate computing device pair based at least in part on a third computing device activity history associated with a third computing device of the at least one candidate computing device pair.
8 . The system of claim 2 , wherein the code comprised in the one or more memory systems is further configured to: determine scores or second data to associate with the at least one candidate computing device pair, based at least in part on the one or more feature vectors associated with the at least one candidate computing device pair.
9 . The system of claim 2 , wherein the code comprised in the one or more memory systems is further configured to determine or generate at least one graph structure comprising one or more of the inter-device graph and the intra-device graph, wherein nodes within a graph structure represent computing device identifiers, including a first computing device identifier associated with the first computing device of the at least one candidate computing device pair and a second computing device identifier associated with the second computing device of the at least one candidate computing device pair.
10 . The system of claim 9 , wherein the code comprised in the one or more memory systems is further configured to identify clusters of nodes within the at least one graph structure based at least in part on one or more clustering rules or conditions.
11 . The system of claim 2 , wherein the code configured to normalize and sample, using the normalization and sampling operation, the processed data or the filtered data, comprises second code configured to normalize and sample the processed data.
12 . The system of claim 2 , wherein the code configured to normalize and sample, using the normalization and sampling operation, the processed data or the filtered data, comprises second code configured to normalize and sample the filtered data.
13 . A method comprising:
using one or more hardware computing device processors to:
receive, from a data source, raw or preprocessed data,
process the raw or preprocessed data received from the data source, thereby resulting in processed data,
filter, using a data filter, the processed data, thereby resulting in filtered data,
normalize and sample, using a normalization and sampling operation, the processed data or the filtered data, thereby resulting in output data, and
transmit the output data to a data store;
using a connectivity generation system to generate a connectivity overlay;
using an event access control system to:
receive first data from the data store, and
generate an event set, wherein the event set is based at least in part on the first data received from the data store and at least one first rule; and
using a feature vector generation system to:
determine, using a machine learning model, at least one candidate computing device pair, wherein the at least one candidate computing device pair is based at least in part on a first computing device activity history associated with a first computing device comprised in the at least one candidate computing device pair, and a second computing device activity history associated with a second computing device comprised in the at least one candidate computing device pair,
apply at least one second rule to the at least one candidate computing device pair, and
generate, based on applying the at least one second rule to the at least one candidate computing device pair, one or more feature vectors associated with the at least one candidate computing device pair,
wherein a first feature vector comprised in the one or more feature vectors associated with the at least one candidate computing device pair comprises a first confidence result and a second confidence result, the first confidence result being used to generate an inter-device graph, and the second confidence result being used to generate an intra-device graph,
wherein the machine learning model is trained based on a set of confidence results associated with a set of candidate computing device pairs, wherein a third confidence result comprised in the set of confidence results is mapped to a first candidate computing device pair comprised in the set of candidate computing device pairs.
14 . The method of claim 13 , wherein the one or more feature vectors associated with the at least one candidate computing device pair comprises a first confidence result and a fifth confidence result, the fourth confidence result being used to generate the inter-device graph, and the fifth confidence result being used to generate the intra-device graph.
15 . The method of claim 13 , further comprising storing data structures comprising computing device identifiers associated with different computing devices or users.
16 . The method of claim 15 , further comprising using the computing device identifiers to target information to the different computing devices or users.
17 . The method of claim 13 , wherein at least one of the raw or preprocessed data, the processed data, the filtered data, or the output data, is used to associate computing devices or computing device identifiers with one or more network source or destination address identifiers.
18 . The method of claim 13 , further comprising using the feature vector generation system to produce the one or more feature vectors associated with the at least one candidate computing device pair based at least in part on a third computing device activity history associated with a third computing device of the at least one candidate computing device pair.
19 . The method of claim 13 , further comprising determining a data category for finding association among computing devices, wherein the data category comprises at least one of an owner, a location, or a time period.
20 . The method of claim 13 , further comprising determining scores or second data to associate with the at least one candidate computing device pair, based at least in part on the one or more feature vectors associated with the at least one candidate computing device pair.
21 . The method of claim 13 , further comprising determining or storing a graph structure including one or more of the inter-device graph and the intra-device graph, wherein nodes within the graph structure represent computing device identifiers, including a first computing device identifier associated with the first computing device of the at least one candidate computing device pair and a second computing device identifier associated with the second computing device of the at least one candidate computing device pair.
22 . The method of claim 21 , further comprising identifying clusters of nodes within the graph structure based at least in part on one or more clustering rules or conditions.
23 . The method of claim 13 , wherein the using the one or more hardware computing device processors to normalize and sample, using the normalization and sampling operation, the processed data or the filtered data, comprises using the one or more hardware computing device processors to normalize and sample the processed data.
24 . The method of claim 13 , wherein the using the one or more hardware computing device processors to normalize and sample, using the normalization and sampling operation, the processed data or the filtered data, comprises using the one or more hardware computing device processors to normalize and sample the filtered data.
25 . The method of claim 13 , wherein the connectivity generation system, the event access control system and the feature vector generation system: are comprised in one or more hardware data systems, or comprise one or more computing operations executable by the one or more hardware data systems.