EFFICIENT CONSOLIDATION OF HIGH-VOLUME METRICS
The disclosed embodiments provide a system for processing data. During operation, the system obtains a set of records from a set of inputs, with each record containing an entity key, a partition key, and one or more attribute-value pairs. For each attribute-value pair in the records, the system maps an attribute name in the attribute-value pair to a unique identifier for the attribute name and replaces the attribute name with the unique identifier. The system then identifies a subset of the records with a matching entity key and a matching partition key and merges the subset of the records into a single record that includes the matching entity key, the matching partition key, and a single field containing a list of attribute-value pairs from the subset of the records. Finally, the system provides the single record and the mapping for use in querying from a centralized source.
1 . A method, comprising:
obtaining a set of records from a set of inputs, wherein each of the records comprises an entity key, a partition key, and one or more attribute-value pairs;
for each attribute-value pair in the set of records:
mapping an attribute name in the attribute-value pair to a unique identifier for the attribute name; and
replacing, by one or more computer systems, the attribute name within the attribute-value pair with the unique identifier;
identifying, by the one or more computer systems, a subset of the records with a matching entity key and a matching partition key;
merging, by the one or more computer systems, the subset of the records into a single record that comprises the matching entity key, the matching partition key, and a single field comprising a list of attribute-value pairs from the subset of the records; and
providing the single record and the mapping for use in querying of data in the set of inputs from a centralized source.
2 . The method of claim 1 , wherein mapping the attribute name in the attribute-value pair to the unique identifier for the attribute name comprises:
combining the attribute name with an input name of an input from which the attribute-value pair was obtained to create a combined name; and
assigning the unique identifier to the combined name.
3 . The method of claim 1 , further comprising:
filtering the subset of the records to exclude, from the single record, a portion of attribute-value pairs in the subset.
4 . The method of claim 3 , wherein filtering the subset of the records to exclude, from the single record, the portion of attribute-value pairs in the subset comprises:
omitting an attribute-value pair from the single record when a value in the attribute-value pair matches a non-meaningful value.
5 . The method of claim 4 , wherein the non-meaningful value comprises at least one of:
a null value;
a zero numeric value; and
a default value.
6 . The method of claim 1 , wherein obtaining the set of records from the set of inputs comprises:
obtaining a configuration comprising a set of input names of the inputs and a set of input locations of the inputs; and
using the input locations to load the records from the inputs.
7 . The method of claim 1 , wherein mapping the attribute name to the unique identifier for the attribute name comprises at least one of:
adding the mapping to a list of mappings of attribute names to unique identifiers; and
identifying an existing mapping of the attribute name to the unique identifier within the list of mappings.
8 . The method of claim 1 , wherein providing the single record for use in querying of data in the set of inputs from the centralized source comprises:
providing the single record in a flattened format.
9 . The method of claim 1 , wherein the entity key represents a member of an online professional network.
10 . The method of claim 1 , wherein the partition key comprises a date key.
11 . The method of claim 1 , wherein an attribute-value pair in the one or more attribute-value pairs comprises an attribute that is a metric and a value that is a measurement of the metric.
12 . An apparatus, comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
obtain a set of records from a set of inputs, wherein each of the records comprises an entity key, a partition key, and one or more attribute-value pairs;
for each attribute-value pair in the set of records:
map an attribute name in the attribute-value pair to a unique identifier for the attribute name; and
replace the attribute name within the attribute-value pair with the unique identifier;
identify a subset of the records with a matching entity key and a matching partition key;
merge the subset of the records into a single record that comprises the matching entity key, the matching partition key, and a single field comprising a list of attribute-value pairs from the subset of the records; and
provide the single record and the mapping for use in querying of data in the set of inputs from a centralized source.
13 . The apparatus of claim 12 , wherein mapping the attribute name in the attribute-value pair to the unique identifier for the attribute name comprises:
combining the attribute name with an input name of an input from which the attribute-value pair was obtained to create a combined name; and
assigning the unique identifier to the combined name..
14 . The apparatus of claim 12 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:
filter the subset of the records to exclude, from the single record, a portion of attribute-value pairs in the subset.
15 . The apparatus of claim 14 , wherein filtering the subset of the records to exclude, from the single record, the portion of attribute-value pairs in the subset comprises:
omitting an attribute-value pair from the single record when a value in the attribute-value pair matches a non-meaningful value.
16 . The apparatus of claim 15 , wherein the non-meaningful value comprises at least one of:
a null value;
a zero numeric value; and
a default value.
17 . The apparatus of claim 12 , wherein obtaining the set of records from the set of inputs comprises:
obtaining a configuration comprising a set of input names of the inputs and a set of input locations of the inputs; and
using the input locations to load the records from the inputs.
18 . The apparatus of claim 12 , wherein mapping the attribute name to the unique identifier for the attribute name comprises at least one of:
adding the mapping to a list of mappings of attribute names to unique identifiers; and
identifying an existing mapping of the attribute name to the unique identifier within the list of mappings.
19 . A system, comprising:
an analysis module comprising a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the system to:
obtain a set of records from a set of inputs, wherein each of the records comprises an entity key, a partition key, and one or more attribute-value pairs;
for each attribute-value pair in the set of records:
map an attribute name in the attribute-value pair to a unique identifier for the attribute name; and
replace the attribute name within the attribute-value pair with the unique identifier;
identify a subset of the records with a matching entity key and a matching partition key;
merge the subset of the records into a single record that comprises the matching entity key, the matching partition key, and a single field comprising a list of attribute-value pairs from the subset of the records; and
a management module comprising a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the system to provide the single record and the mapping for use in querying of data in the set of inputs from a centralized source.
20 . The system of claim 19 , wherein merging the subset of the records into the single record comprises:
omitting an attribute-value pair from the single record when a value in the attribute-value pair matches a non-meaningful value.