Distributed cardinality optimization
A query requesting a count of unique data values for a specific attribute is received. The received query is used to generate and transmit a plurality of non-overlapping queries to a data store. A plurality of responses is received from the data store. Results from the plurality of responses is summed and the resulting sum is returned.
1. A system, comprising:
a hardware processor configured to:
receive, from a client device, a cardinality query associated with at least one of a session dimension or a device attribute identifiable from the cardinality query requesting a count of unique data values for a specific attribute for a defined time interval;
use the received cardinality query to generate and transmit to a data store a plurality of non-overlapping queries, wherein stored session records in the data store are compressed, at least in part, by a vertical data compression engine removing irrelevant time attributes and a horizontal data compression engine removing one or more irrelevant non-time attributes;
receive a plurality of responses from the data store based on the compressed session records;
aggregate results from the plurality of responses; and
return the aggregated results to the client device; and
a memory coupled to the hardware processor and configured to provide the processor with instructions.
2. The system of claim 1 , wherein the specific attribute is destination IP.
3. The system of claim 1 , wherein the specific attribute is a country.
4. The system of claim 1 , wherein the specific attribute is a maliciousness flag.
5. The system of claim 1 , wherein the cardinality query is received from a configured dashboard associated with the client device.
6. The system of claim 1 , wherein the cardinality query is received from a configured alert associated with the client device.
7. The system of claim 1 , wherein the cardinality query is received from an administrator associated with the client device in real time.
8. A method, comprising:
receiving, from a client device, a cardinality query associated with at least one of a session dimension or a device attribute identifiable from the cardinality query requesting a count of unique data values for a specific attribute for a defined time interval;
using the received cardinality query to generate and transmit to a data store a plurality of non-overlapping queries, wherein stored session records in the data store are compressed, at least in part, by a vertical data compression engine removing irrelevant time attributes and a horizontal data compression engine removing one or more irrelevant non-time attributes;
receiving a plurality of responses from the data store based on the compressed session records;
aggregating results from the plurality of responses; and
returning the aggregated results to the client device.
9. The method of claim 8 , wherein the specific attribute is destination IP.
10. The method of claim 8 , wherein the specific attribute is a country.
11. The method of claim 8 , wherein the specific attribute is a maliciousness flag.
12. The method of claim 8 , wherein the cardinality query is received from a configured dashboard associated with the client device.
13. The method of claim 8 , wherein the cardinality query is received from a configured alert associated with the client device.
14. The method of claim 8 , wherein the cardinality query is received from an administrator associated with the client device in real time.
15. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving, from a client device, a cardinality query associated with at least one of a session dimension or a device attribute identifiable from the cardinality query requesting a count of unique data values for a specific attribute for a defined time interval;
using the received cardinality query to generate and transmit to a data store a plurality of non-overlapping queries, wherein stored session records in the data store are compressed, at least in part, by a vertical data compression engine removing irrelevant time attributes and a horizontal data compression engine removing one or more irrelevant non-time attributes;
receiving a plurality of responses from the data store based on the compressed session records;
aggregating results from the plurality of responses; and
returning the aggregated results to the client device.