Multidimensional partition of data to calculate aggregation at scale
A method enables data aggregation in a multi-tenant system. The method includes receiving, at a data aggregation manager, an event from an activity tracking component, generating, by the data aggregation manager, a first hash value based on a first identifier in the event, generating, by the data aggregation manager, a second hash value based on a second identifier in the event, and storing event message information to an entry of a metric storage database in a partition according the first hash value and the second hash value.
1. A method of data aggregation in a multi-tenant system, the method comprising:
receiving, at a data aggregation manager, an event from an activity tracking component where the activity tracking component is executed by a first electronic device communicatively coupled to a second electronic device that executes the data aggregation manager;
generating, by the data aggregation manager, a first hash value based on a first identifier in the event;
generating, by the data aggregation manager, a second hash value based on a second identifier in the event;
storing, by the data aggregation manager, event message information to an entry of a metric storage database in a partition according to the first hash value and the second hash value, in response to determining the metric storage database does not include a prior copy of the event, where the entry can include event message information for multiple events having a same value of the first hash value or of the second hash value; and
updating, by the data aggregation manager, a metric count for the entry, in response to storing the event message information to the entry.
2. The method of data aggregation of claim 1 , further comprising:
discarding the event in response to determining that the metric storage database includes a prior copy of the event in the entry corresponding to the first hash value and the second hash value.
3. The method of data aggregation of claim 1 , further comprising:
aggregating a metric count across partitions in the metric storage database.
4. The method of data aggregation of claim 1 , wherein the event is generated by the activity tracking component.
5. The method of data aggregation of claim 1 , wherein the first identifier is an asset identifier that uniquely identifies an activity or engagement.
6. The method of data aggregation of claim 1 , wherein the second identifier is a target identifier to uniquely identify a target recipient of the event.
7. The method of data aggregation of claim 1 , wherein the first hash value determines a horizontal partition in the metric storage database.
8. The method of data aggregation of claim 1 , wherein the second hash value determines a vertical partition in the metric storage database.
9. The method of data aggregation of claim 1 , wherein the activity tracking component outputs a message queue including data sourced by different tenants of the multi-tenant system, the event included in a message of the message queue and sourced by a source tenant.
10. The method of data aggregation of claim 9 , wherein receiving the event from the activity tracking component comprises:
partitioning the messages of the message queue according to the source tenant, the first identifier, and the second identifier.
11. A non-transitory machine-readable storage medium that provides instructions that, if executed by a processor, will cause said processor to perform operations comprising:
receiving, at a data aggregation manager, an event from an activity tracking component where the activity tracking component is executed by a first electronic device communicatively coupled to a second electronic device that executes the data aggregation manager;
generating, by the data aggregation manager, a first hash value based on a first identifier in the event;
generating, by the data aggregation manager, a second hash value based on a second identifier in the event;
storing, by the data aggregation manager, event message information to an entry of a metric storage database in a partition according to the first hash value and the second hash value, in response to determining the metric storage database does not include a prior copy of the event, where the entry can include event message information for multiple events having a same value of the first hash value or of the second hash value; and
updating, by the data aggregation manager, a metric count for the entry, in response to storing the event message information to the entry.
12. The non-transitory machine-readable storage medium of claim 11 , the operations further comprising:
discarding the event in response to determining that the metric storage database includes a prior copy of the event in the entry corresponding to the first hash value and the second hash value.
13. The non-transitory machine-readable storage medium of claim 11 , the operations further comprising:
aggregating a metric count across partitions in the metric storage database.
14. The non-transitory machine-readable storage medium of claim 11 , wherein the event is generated by the activity tracking component.
15. The non-transitory machine-readable storage medium of claim 11 , wherein the first identifier is an asset identifier that uniquely identifies an activity or engagement.
16. The non-transitory machine-readable storage medium of claim 11 , wherein the second identifier is a target identifier to uniquely identify a target recipient of the event.
17. The non-transitory machine-readable storage medium of claim 11 , wherein the first hash value determines a horizontal partition in the metric storage database.
18. The non-transitory machine-readable storage medium of claim 11 , wherein the second hash value determines a vertical partition in the metric storage database.
19. A computer system to execute a method of data aggregation in a multi-tenant system, the computer system comprising:
a non-transitory machine-readable storage medium having stored therein a data aggregation manager; and
a processor coupled to the non-transitory machine-readable storage medium, the processor to execute the data aggregation manager, the data aggregation manager to receive an event from an activity tracking component executed by another computer system, to generate a first hash value based on a first identifier in the event, to generate a second hash value based on a second identifier in the event, to store event message information to an entry of a metric storage database in a partition according to the first hash value and the second hash value, in response to determining the metric storage database does not include a prior copy of the event, where the entry can include event message information for multiple events having a same value of the first hash value or of the second hash value, and updating a metric count for the entry, in response to storing the event message information to the entry.
20. The computer system of claim 19 , wherein the data aggregation manager is further to discard the event in response to determining that the metric storage database includes a prior copy of the event in the entry corresponding to the first hash value and the second hash value.