Hash bucketing of data
The disclosed embodiments include a method performed by a data intake and query system. The method includes ingesting metrics including respective key values and respective measured values, where the respective key values include a primary key value of a selected primary key. The method further includes generating a hash value for each metric by processing each primary key value with a hashing function, and indexing each metric in association with an existing hash bucket having a matching hash value.
1. A method comprising:
ingesting, by a data intake and query system, a plurality of metrics including key values and measured values, wherein a measured value includes a numerical value and represents a performance measurement of a computing resource that is measured at a point in time, and wherein each of the plurality of key values includes a primary key value of a primary key;
generating, by the data intake and query system, a particular hash value for each metric of the plurality of metrics by processing a primary key value associated with the metric, with a hashing function; and
storing, by the data intake and query system, a first metric of the plurality of metrics in a first hash bucket, wherein the first hash bucket is associated with a time-based criterion, and wherein a hash value of the first hash bucket matches the particular hash value for the first metric;
storing, by the data intake and query system, a second metric of the plurality of metrics in a second hash bucket when a time value associated with the second metric of the plurality of metrics does not correspond to the time-based criterion associated with the first hash bucket;
generating, by the data intake and query system, a new hash bucket for a third metric having a particular hash value that does not match the hash value of the first hash bucket, the new hash bucket being allocated to receive other metrics having hash values that match the hash value of the new bucket;
identifying, by the data intake and query system, an anomalous metric from among the plurality of metrics, the anomalous metric being a metric that includes an outlier key value relative to key values of other metrics; and
storing, by the data intake and query system, the anomalous metric in a quarantine bucket irrespective of the particular hash value of the anomalous metric;
obtaining, by the data intake and query system, search results satisfying a search query indicative of queried key value by searching hash buckets having hash values that match a hash value of the queried key value;
causing display, on a display device, of the search results or data indicative of the search results; and
generating, by the data intake and query system, a new hash bucket for a particular metric having a particular hash value that matches the hash value of the first hash bucket, in response to determining that a time value of the particular metric is not within a time range of the first hash bucket, wherein the new hash bucket is allocated to receive other metrics having hash values that match the hash value of the first hash bucket.
2. The method of claim 1 , wherein the primary key is selected by a user.
3. The method of claim 1 , wherein the primary key is included in a policy defining conditions for generating a new hash bucket.
4. The method of claim 1 , wherein the new hash bucket is generated in accordance with heuristics defining conditions that must be satisfied before generating the new hash bucket.
5. The method of claim 1 , wherein the primary key is a source key, a host key, or a source type key.
6. The method of claim 1 , wherein the search results are obtained by only searching the hash buckets that have hash values matching the hash value of the queried key value.
7. The method of claim 1 , wherein the primary key is not a time based key.
8. The method of claim 1 , wherein the quarantine bucket includes a plurality of anomalous metrics having different hash values.
9. The method of claim 1 , wherein the new hash bucket is generated when the first hash bucket exceeds a threshold size value.
10. The method of claim 1 , wherein the new hash bucket is generated when a time value of the particular metric is not within a time range of the first hash bucket.
11. The method of claim 1 , wherein the search results are obtained by only searching the hash buckets having hash values that match the hash value of the queried key value, and wherein the search query is input by a user and expressed in a pipelined search language.
12. The method of claim 1 , wherein each metric includes a plurality of dimensions, wherein each dimension is either a required dimension or an optional dimension, wherein each metric includes a key value for each required dimension, and wherein only some of the plurality of metrics include key values for some optional dimensions.
13. The method of claim 1 , wherein each metric is a semi-structured metric or a structured metric.
14. The method of claim 1 , wherein the measured value comprises a measure of a utilization of an electronic component, a temperature of the electronic component, or a voltage reading of the electronic component.
15. The method of claim 1 , wherein each numerical value is a floating point value.
16. A non-transitory machine-readable store medium having instructions stored thereon, execution of which in a data intake and query system cause the data intake and query system to perform operations comprising:
ingesting, by a data intake and query system, a plurality of metrics including key values and measured values, wherein a measured value includes a numerical value and represents a performance measurement of a computing resource that is measured at a point in time, and wherein each of the plurality of key values includes a primary key value of a primary key;
generating, by the data intake and query system, a particular hash value for each metric of the plurality of metrics by processing a primary key value associated with the metric, with a hashing function;
storing, by the data intake and query system, a first metric of the plurality of metrics in a first hash bucket, wherein the first hash bucket is associated with a time-based criterion, and wherein a hash value of the first hash bucket matches the particular hash value for the first metric;
storing, by the data intake and query system, a second metric of the plurality of metrics in a second hash bucket when a time value associated with the second metric of the plurality of metrics does not correspond to the time-based criterion associated with the first hash bucket;
generating, by the data intake and query system, a new hash bucket for a third metric having a particular hash value that does not match the hash value of the first hash bucket, the new hash bucket being allocated to receive other metrics having hash values that match the hash value of the new bucket;
identifying, by the data intake and query system, an anomalous metric from among the plurality of metrics, the anomalous metric being a metric that includes an outlier key value relative to key values of other metrics;
storing, by the data intake and query system, the anomalous metric in a quarantine bucket irrespective of the particular hash value of the anomalous metric;
obtaining, by the data intake and query system, search results satisfying a search query indicative of queried key value by searching hash buckets having hash values that match a hash value of the queried key value;
causing display, on a display device, of the search results or data indicative of the search results; and
generating, by the data intake and query system, a new hash bucket for a particular metric having a particular hash value that matches the hash value of the first hash bucket, in response to determining that a time value of the particular metric is not within a time range of the first hash bucket, wherein the new hash bucket is allocated to receive other metrics having hash values that match the hash value of the first hash bucket.
17. The non-transitory machine-readable storage medium of claim 16 , wherein the primary key is selected by a user.
18. The non-transitory machine-readable storage medium of claim 16 , wherein the primary key is included in a policy defining conditions for generating the new hash bucket.
19. The method non-transitory machine-readable storage medium of claim 16 , wherein the new hash bucket is generated in accordance with heuristics defining conditions that must be satisfied before generating the new hash bucket.
20. The non-transitory machine-readable storage medium of claim 16 , wherein the primary key is a source key, a host key, or a source type key.
21. The non-transitory machine-readable storage medium of claim 16 , wherein the search results are obtained by only searching the hash buckets matching the hash value of the queried key value.
22. The non-transitory machine-readable storage medium of claim 16 , wherein the primary key is not a time based key.
23. The non-transitory machine-readable storage medium of claim 16 , wherein the quarantine bucket includes a plurality of anomalous events having different hash values, and wherein the data intake and query system searches the quarantine bucket in addition to only the hash buckets matching the hash value of the queried key value to obtain the search results.
24. The non-transitory machine-readable storage medium of claim 16 , wherein the new hash bucket is generated even if the first hash bucket has a matching hash value when the first hash bucket exceeds a threshold size value.
25. A data intake and query system comprising:
a processor; and
memory containing instructions that, when executed by the data intake and query system, cause the data intake and query system to:
ingest a plurality of metrics including key values and measured values, wherein a measured value includes a numerical value and represents a performance measurement of a computing resource that is measured at a point in time, and wherein each of the plurality of key values includes a primary key value;
generate a particular hash value for each metric of the plurality of metrics by processing a primary key value associated with the metric, with a hashing function; and
store a first metric of the plurality of metrics in a first hash bucket, wherein the first hash bucket is associated with a time-based criterion, and wherein a hash value of the first hash bucket matches the particular hash value for the first metric;
store a second metric of the plurality of metrics in a second hash bucket when a time value associated with the second metric of the plurality of metrics does not correspond to the time-based criterion associated with the first hash bucket;
generate a new hash bucket for a third metric having a particular hash value that does not match the hash value of the first hash bucket, the new hash bucket being allocated to receive other metrics having hash values that match the hash value of the new bucket;
identify an anomalous metric from among the plurality of metrics, the anomalous metric being a metric that includes an outlier key value relative to key values of other metrics;
store the anomalous metric in a quarantine bucket irrespective of the particular hash value of the anomalous metric;
obtain search results satisfying a search query indicative of queried key value by searching hash buckets having hash values that match a hash value of the queried key value;
cause display, on a display device, of the search results or data indicative of the search results; and
generate a new hash bucket for a particular metric having a particular hash value that matches the hash value of the first hash bucket, in response to determining that a time value of the particular metric is not within a time range of the first hash bucket, wherein the new hash bucket is allocated to receive other metrics having hash values that match the hash value of the first hash bucket.