Aggregating quantile metrics in multidimensional data sets

1. A method, comprising:

obtaining, by a set of processing nodes, a set of partitions comprising records spanning a time interval, wherein the records comprise a set of values for a metric and a set of dimensions associated with the set of values;

for each dimensional subset associated with the set of dimensions, aggregating, by the set of processing nodes based on the set of partitions, a corresponding subset of values for the metric into a distribution-based representation of the metric;

storing a dictionary encoding of the dimensional subset in association with the distribution-based representation of the metric;

calculating, based on the distribution-based representation, one or more quantiles associated with the dimensional subset; and

outputting the one or more quantiles for use in characterizing a distribution of the metric.

2. The method of claim 1 , wherein aggregating the corresponding subset of values for the metric into the distribution-based representation of the metric comprises:

generating a fixed number of clusters from the corresponding subset of values for the metric; and

for each cluster in the fixed number of clusters, storing a center of the cluster and a number of values associated with the cluster.

3. The method of claim 2 , wherein aggregating the corresponding subset of values for the metric into the distribution-based representation of the metric further comprises:

merging the fixed number of clusters from multiple partitions in the set of partitions into a single set of clusters representing the dimensional subset.

4. The method of claim 1 , wherein storing the dictionary encoding of the dimensional subset in association with the distribution-based representation of the metric comprises:

when a value of a dimension in the dimensional subset is first encountered in the records, assigning an encoding to the value;

storing the encoding in a position associated with the dimension within an ordering of dimensions representing the dimensional subset; and

mapping the ordering of the dimensions to the distribution-based representation of the metric for the dimensional subset.

5. The method of claim 4 , wherein storing the dictionary encoding of the dimensional subset in association with the distribution-based representation of the metric further comprises:

when another dimension is not specified in the dimensional subset, storing a default value at another position for the other dimension within the ordering of dimensions.

6. The method of claim 1 , wherein storing the dictionary encoding of the dimensional subset in association with the distribution-based representation of the metric comprises:

maintaining in-memory representations of dictionary encodings and distribution-based representations of the metric that span the time interval and one or more additional time intervals; and

when the time interval and the one or more additional time intervals spanned by the in-memory representations fill a time window, transferring one or more of the in-memory representations to storage.

7. The method of claim 6 , wherein storing the dictionary encoding of the dimensional subset in association with the distribution-based representation of the metric further comprises:

after the dictionary encoding and the distribution-based representation of the metric are transferred to the storage, receiving a late record associated with the time interval in an event stream;

matching the late record to the dimensional subset; and

updating the distribution-based representation of the metric stored in association with the dictionary encoding of the dimensional subset in the storage based on a value of the metric in the late record.

8. The method of claim 1 , further comprising:

merging, based on the stored dictionary encoding of the dimensional subset, the distribution-based representation of the metric with another distribution-based representation of the metric for the dimensional subset and a different time interval; and

calculating an additional quantile based on the merged distribution-based representation of the metric and the other distribution-based representation of the metric.

9. The method of claim 1 , wherein obtaining the set of partitions comprising the records comprises:

dividing an event stream comprising the records into the set of partitions; and

distributing the set of partitions among the set of processing nodes.

10. The method of claim 1 , wherein the set of dimensions comprises at least one of:

a location;

a browser;

an operating system;

a device type;

a protocol;

a page key; or

an environment.

11. The method of claim 1 , wherein the metric comprises at least one of:

a page load time;

a first byte time;

a render completion time;

a download time;

a function execution time; or

a request time.

12. A system, comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the system to:

obtain a set of partitions comprising records spanning a time interval, wherein the records comprise a set of values for a metric and a set of dimensions associated with the set of values;

for each dimensional subset associated with the set of dimensions, aggregate, based on the set of partitions, a corresponding subset of values for the metric into a distribution-based representation of the metric;

store a dictionary encoding of the dimensional subset in association with the distribution-based representation of the metric;

calculate, based on the distribution-based representation, one or more quantiles associated with the dimensional subset; and

output the one or more quantiles for use in characterizing a distribution of the metric.

13. The system of claim 12 , wherein aggregating the corresponding subset of values for the metric into the distribution-based representation of the metric comprises:

generating a fixed number of clusters from the corresponding subset of values for the metric; and

for each cluster in the fixed number of clusters, storing a center of the cluster and a number of values associated with the cluster.

14. The system of claim 12 , wherein storing the dictionary encoding of the dimensional subset in association with the distribution-based representation of the metric comprises:

when a value of a dimension in the dimensional subset is first encountered in the records, assigning an encoding to the value;

storing the encoding in a position associated with the dimension within an ordering of dimensions representing the dimensional subset;

mapping the ordering of the dimensions to the distribution-based representation of the metric for the dimensional subset; and

when another dimension is not specified in the dimensional subset, storing a default value at another position for the other dimension within the ordering of dimensions.

15. The system of claim 12 , wherein storing the dictionary encoding of the dimensional subset in association with the distribution-based representation of the metric comprises:

maintaining in-memory representations of dictionary encodings and distribution-based representations of the metric that span the time interval and one or more additional time intervals; and

when the time interval and the one or more additional time intervals spanned by the in-memory representations fill a time window, transferring one or more of the in-memory representations to storage.

16. The system of claim 15 , wherein storing the dictionary encoding of the dimensional subset in association with the distribution-based representation of the metric further comprises:

after the dictionary encoding and the distribution-based representation of the metric are transferred to the storage, receiving a late record associated with the time interval in an event stream associated with late records;

matching the late record to the dimensional subset; and

updating the distribution-based representation of the metric stored in association with the dictionary encoding of the dimensional subset in the storage based on a value of the metric in the late record.

17. The system of claim 12 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to:

merge, based on the stored dictionary encoding of the dimensional subset, the distribution-based representation of the metric with another distribution-based representation of the metric for the dimensional subset and a different time interval; and

calculate an additional quantile based on the merged distribution-based representation of the metric and the other distribution-based representation of the metric.

18. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, the method comprising:

obtaining a set of partitions comprising a records spanning a time interval, wherein the records comprise a set of values for a metric and a set of dimensions associated with the set of values;

for each dimensional subset associated with the set of dimensions, aggregating, based on the set of partitions, a corresponding subset of values for the metric into a distribution-based representation of the metric;

storing a dictionary encoding of the dimensional subset in association with the distribution-based representation of the metric;

calculating, based on the distribution-based representation, one or more quantiles associated with the dimensional subset; and

outputting the one or more quantiles for use in characterizing a distribution of the metric.

19. The non-transitory computer-readable storage medium of claim 18 , wherein storing the dictionary encoding of the dimensional subset in association with the distribution-based representation of the metric comprises:

when a value of a dimension in the dimensional subset is first encountered in the records, assigning an encoding to the value;

storing the encoding in a position associated with the dimension within an ordering of dimensions representing the dimensional subset; and

mapping the ordering of the dimensions to the distribution-based representation of the metric for the dimensional subset.

20. The non-transitory computer-readable storage medium of claim 18 , wherein storing the dictionary encoding of the dimensional subset in association with the distribution-based representation of the metric comprises:

maintaining in-memory representations of dictionary encodings and distribution-based representations of the metric that span the time interval and one or more additional time intervals; and

when the time interval and the one or more additional time intervals spanned by the in-memory representations fill a time window, transferring one or more of the in-memory representations to storage.