Distributed cardinality optimization
A cardinality query associated with a specific attribute is received. One or more in-scope attribute cardinality partitions is enforced on session record analytics. Vertical data compression is performed to eliminate a time dimension. Horizontal data compression is performed to eliminate one or more out-of-scope attributes. One or more like in-scope session records is aggregated. Magnitudes of each in-scope attribute of each enforced cardinality partition are summed. The sum is returned as a response to the cardinality query.
1. A system, comprising:
a hardware processor; and
a memory coupled to the hardware processor and configured to provide the hardware processor with instructions configured to:
receive a cardinality query associated with a specific attribute;
generate one or more cardinality partitions on session record analytics in a session datastore, wherein the one or more cardinality partitions comprise in-scope session records;
perform vertical data compression to eliminate a time dimension in the one or more cardinality partitions;
perform horizontal data compression to eliminate one or more out-of-scope attributes in the one or more cardinality partitions;
aggregate, in each of the one or more compressed cardinality partitions, like in-scope session records, wherein a given like in-scope session record matches an in-scope attribute array;
sum magnitudes of each in-scope attribute array of each of the one or more compressed cardinality partitions matching the specific attribute; and
return the sum of magnitudes as a response to the cardinality query as output.
2. The system of claim 1 , wherein the specific attribute is destination IP.
3. The system of claim 1 , wherein the specific attribute is a country.
4. The system of claim 1 , wherein the specific attribute is a maliciousness flag.
5. The system of claim 1 , wherein the cardinality query is received from a configured dashboard.
6. The system of claim 1 , wherein the cardinality query is received from a configured alert.
7. The system of claim 1 , wherein the cardinality query is received from an administrator in real time.
8. The system of claim 1 , wherein the processor is further configured to perform the horizontal data compression on a set of session records.
9. The system of claim 1 , wherein the processor is further configured to perform the vertical data compression on a set of session records.
10. A method, comprising:
receiving a cardinality query associated with a specific attribute;
generating one or more cardinality partitions on session record analytics in a session datastore, wherein the one or more cardinality partitions comprise in-scope session records;
performing vertical data compression to eliminate a time dimension in the one or more cardinality partitions;
performing horizontal data compression to eliminate one or more out-of-scope attributes in the one or more cardinality partitions;
aggregating, in each of the one or more compressed cardinality partitions, like in-scope session records, wherein a given like in-scope session record matches an in-scope attribute array;
summing magnitudes of each in-scope attribute array of each of the one or more compressed cardinality partitions matching the specific attribute; and
returning the sum of magnitudes as a response to the cardinality query as output.
11. The method of claim 10 , wherein the specific attribute is destination IP.
12. The method of claim 10 , wherein the specific attribute is a country.
13. The method of claim 10 , wherein the specific attribute is a maliciousness flag.
14. The method of claim 10 , wherein the cardinality query is received from a configured dashboard.
15. The method of claim 10 , wherein the cardinality query is received from a configured alert.
16. The method of claim 10 , wherein the cardinality query is received from an administrator in real time.
17. The method of claim 10 , further comprising performing the horizontal data compression on a set of session records.
18. The method of claim 10 , further comprising performing the vertical data compression on a set of session records.
19. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving a cardinality query associated with a specific attribute;
generating one or more cardinality partitions on session record analytics in a session datastore, wherein the one or more cardinality partitions comprise in-scope session records;
performing vertical data compression to eliminate a time dimension in the one or more cardinality partitions;
performing horizontal data compression to eliminate one or more out-of-scope attributes in the one or more cardinality partitions;
aggregating, in each of the one or more compressed cardinality partitions, like in-scope session records, wherein a given like in-scope session record matches an in-scope attribute array;
summing magnitudes of each in-scope attribute array of each of the one or more compressed cardinality partitions matching the specific attribute; and
returning the sum of magnitudes as a response to the cardinality query as output.
20. The computer program product of claim 19 , wherein the specific attribute is destination IP.