Collecting and supporting querying of high-cardinality time-series data
A method and apparatus for collecting and supporting querying of multi-dimensional data pertaining to usage of software and/or hardware to service tenant requests in a multi-tenant cloud computing system where the multi-dimensional data is initially captured on a per request basis and recorded in objects of a first type that store data pertaining to a specific request, specific tenant, specific host and specific time. The objects of the first type are combined by time windows to form objects of a second type. The objects of a second type are stored in another system as separate text files. Responsive to a query for multi-dimensional data for a specific tenant that spans an interval of multiple time windows, the objects of the second type for the specific tenant and time interval are combined across all hosts to generate a query result, and the query result is returned.
1. An article of manufacture comprising:
a non-transitory machine-readable storage medium that provides instructions that, if executed by a set of one or more processors implemented in an electronic device, are configurable to cause the electronic device to perform operations for collecting and supporting, for one or more tenants in a multi-tenant cloud computing system, querying of data pertaining to usage of software and/or hardware to service requests made by those tenants in the multi-tenant cloud computing system, wherein such data is high-cardinality and multi-dimensional, where the data is initially captured on a per request basis such that there is an object of a first type for each request, objects of the first type store multi-dimensional data pertaining to usage of software and hardware for a specific request made by a specific tenant at a specific host at a specific time, the operations comprising:
within each host, combining the objects of the first type for each tenant by time windows to form objects of a second type, wherein the combining includes separately combining each like dimension of the multi-dimensional data, such that the objects of the second type store, for each dimension of the multi-dimensional data, data that represents the totality of requests made by each tenant in each time window at a specific host;
storing the objects of the second type in another system as separate text files as opposed to indexed records in a database; and
responsive to a query regarding usage of software and/or hardware regarding a specific one of the tenants during a specific time interval that spans multiple consecutive time windows:
retrieving, from the files, objects of the second type that pertain to the specific tenant and the specific time interval,
combining the retrieved objects of the second type by time windows to generate a query result, wherein the combining includes separately combining each like dimension of the multi-dimensional data, such that the query result includes, for each dimension of the multi-dimensional data, data that represents the totality of requests made to the hosts by the specific tenant in each of the time windows in the specific time interval, and
returning the query result.
2. The article of manufacture of claim 1 , wherein combining the objects of the first type and combining objects of second type includes aggregating one or more first dimensions by applying a first aggregation operation to respective first dimension values.
3. The article of manufacture of claim 2 , wherein combining the objects of the first type and combining objects of second type includes aggregating one or more second dimensions by applying a second aggregation operation, different than the first aggregation operation, to respective second dimension values.
4. The article of manufacture of claim 1 , wherein the objects of the first type and the objects of the second type are stored in structured object notation format.
5. The article of manufacture of claim 4 , wherein the objects of the first type and the objects of the second type are stored in JavaScript Object Notation (JSON) format.
6. The article of manufacture of claim 1 , the operations further comprising:
causing the query result to be displayed on a graphical user interface of a client device.
7. The article of manufacture of claim 1 , wherein the time window is in the range of 1 ms-500 ms.
8. A method for collecting and supporting, for one or more tenants in a multi-tenant cloud computing system, querying of data pertaining to usage of software and/or hardware to service requests made by those tenants in the multi-tenant cloud computing system, wherein such data is high-cardinality and multi-dimensional, where the data is initially captured on a per request basis such that there is an object of a first type for each request, objects of the first type store multi-dimensional data pertaining to usage of software and/or hardware for a specific request made by a specific tenant at a specific host at a specific time, the method comprising:
within each host, combining the objects of the first type for each tenant by time windows to form objects of a second type, wherein the combining includes separately combining each like dimension of the multi-dimensional data, such that the objects of the second type store, for each dimension of the multi-dimensional data, data that represents the totality of requests made by each tenant in each time window at a specific host;
storing the objects of the second type in another system as separate text files as opposed to indexed records in a database; and
responsive to a query regarding usage of software and/or hardware regarding a specific one of the tenants during a specific time interval that spans multiple consecutive time windows:
retrieving, from the files, objects of the second type that pertain to the specific tenant and the specific time interval,
combining the retrieved objects of the second type by time windows to generate a query result, wherein the combining includes separately combining each like dimension of the multi-dimensional data, such that the query result includes, for each dimension of the multi-dimensional data, data that represents the totality of requests made to the hosts by the specific tenant in each of the time windows in the specific time interval, and
returning the query result.
9. The method of claim 8 , wherein combining the objects of the first type and combining objects of second type includes aggregating one or more first dimensions by applying a first aggregation operation to respective first dimension values.
10. The method of claim 9 , wherein combining the objects of the first type and combining objects of second type includes aggregating one or more second dimensions by applying a second aggregation operation, different than the first aggregation operation, to respective second dimension values.
11. The method of claim 8 , wherein the objects of the first type and the objects of the second type are stored in structured object notation format.
12. The method of claim 11 , wherein the objects of the first type and the objects of the second type are stored in JavaScript Object Notation (JSON) format.
13. The method of claim 8 , further comprising:
causing the query result to be displayed on a graphical user interface of a client device.
14. The method of claim 8 , wherein the time window is in the range of 1 ms-500 ms.
15. An apparatus comprising:
a set of one or more processors; and
a non-transitory machine-readable storage medium that provides instructions that, if executed by the set of one or more processors, are configurable to cause the apparatus to perform operations for collecting and supporting, for one or more tenants in a multi-tenant cloud computing system, querying of data pertaining to usage of software and/or hardware to service requests made by those tenants in the multi-tenant cloud computing system, wherein such data is high-cardinality and multi-dimensional, where the data is initially captured on a per request basis such that there is an object of a first type for each request, objects of the first type store multi-dimensional data pertaining to usage of software and/or hardware for a specific request made by a specific tenant at a specific host at a specific time, the operations comprising:
within each host, combining the objects of the first type for each tenant by time windows to form objects of a second type, wherein the combining includes separately combining each like dimension of the multi-dimensional data, such that the objects of the second type store, for each dimension of the multi-dimensional data, data that represents the totality of requests made by each tenant in each time window at a specific host;
storing the objects of the second type in another system as separate text files as opposed to indexed records in a database; and
responsive to a query regarding usage of software and/or hardware regarding a specific one of the tenants during a specific interval that spans multiple consecutive time windows:
retrieving, from the files, objects of the second type that pertain to the specific tenant and the specific time interval,
combining the retrieved objects of the second type by time windows to generate a query result, wherein the combining includes separately combining each like dimension of the multi-dimensional data, such that the query result includes, for each dimension of the multi-dimensional data, data that represents the totality of requests made to the hosts by the specific tenant in each of the time windows in the specific time interval, and
returning the query result.
16. The apparatus of claim 15 , wherein combining the objects of the first type and combining objects of second type includes aggregating one or more first dimensions by applying a first aggregation operation to respective first dimension values.
17. The apparatus of claim 16 , wherein combining the objects of the first type and combining objects of second type includes aggregating one or more second dimensions by applying a second aggregation operation, different than the first aggregation operation, to respective second dimension values.
18. The apparatus of claim 15 , wherein the objects of the first type and the objects of the second type are stored in structured object notation format.
19. The apparatus of claim 15 , wherein the objects of the first type and the objects of the second type are stored in JavaScript Object Notation (JSON) format.
20. The apparatus of claim 15 , wherein the operations further comprise:
causing the query result to be displayed on a graphical user interface of a client device.