Storage of mass data for monitoring
Disclosed are various embodiments for processing and storing mass data, where the data may include metrics generated based on performance of an event in a monitored system. Metrics describing a state of a monitored system may be received, accessed, and aggregated to generate a data model that describes performance of the monitored system. The metrics utilized in generating the data model may be disregarded after the data model has been generated. An output describing the state of the monitored system may be generated based on the data model, and the output may be communicated over a network, for example, to a requesting service.
1. A system, comprising:
at least one computing device; and
program instructions executable in the at least one computing device that, when executed, cause the at least one computing device to:
access a plurality of metrics generated based on individual ones of a plurality of events performed by a monitored system, wherein the plurality of metrics describe a state of the monitored system with respect to a plurality of timestamps generated at an occurrence of the plurality of events;
generate a data model that describes a performance of the monitored system by aggregating the plurality of metrics, the data model being generated for at least one of a plurality of periods of time utilizing individual ones of the plurality of metrics having a respective timestamp falling within the at least one of the plurality of periods of time, wherein the data model consumes less storage space in memory than the plurality of metrics;
disregard the individual ones of the plurality of metrics utilized in generating the data model after the data model has been generated;
generate an output describing the state of the monitored system based at least in part on the data model; and
send the output describing the monitored system.
2. The system of claim 1 , wherein individual ones of the plurality of metrics are associated with at least one dimension.
3. The system of claim 1 , wherein individual ones of the plurality of metrics are generated by the monitored system in an extensible language markup (XML) format.
4. The system of claim 1 , wherein the data model that describes the performance of the monitored system is generated using at least one aggregate metric.
5. The system of claim 4 , wherein the at least one aggregate metric comprises at least one of: a total count of the plurality of metrics, an average of a plurality of values associated with the plurality of metrics, and a sum of the plurality of values associated with the plurality of metrics.
6. The system of claim 4 , wherein the at least one aggregate metric comprises at least one of: a maximum one of a plurality of values associated with the plurality of metrics and a minimum one of the plurality of values associated with the plurality of metrics.
7. The system of claim 1 , wherein the data model is generated based at least in part on a specified degree of aggregation pertaining to a degree of detail in information maintained in the data model relative to the plurality of metrics.
8. The system of claim 1 , wherein the data model is generated based at least in part on a previously generated data model.
9. A computer-implemented method, comprising:
accessing, by at least one computing device, a plurality of metrics generated based on individual ones of a plurality of events performed by a monitored system, wherein the plurality of metrics describe a state of the monitored system with respect to a plurality of timestamps generated at an occurrence of the plurality of events;
generating, by the at least one computing device, an aggregate metric that describes a performance of the monitored system by aggregating the plurality of metrics, the aggregate metric being generated for at least one of a plurality of periods of time utilizing individual ones of the plurality of metrics having a respective timestamp falling within the at least one of the plurality of periods of time, wherein the aggregate metric consumes less storage space in memory than the plurality of metrics;
disregarding, by the at least one computing device, the individual ones of the plurality of metrics utilized in generating the aggregate metric after the aggregate metric has been generated;
generating, by the at least one computing device, an output describing the state of the monitored system based at least in part on the aggregate metric; and
sending, by the at least one computing device, the output describing the monitored system.
10. The computer-implemented method of claim 9 , wherein individual ones of the plurality of metrics are associated with at least one dimension.
11. The computer-implemented method of claim 9 , wherein individual ones of the plurality of metrics are generated by the monitored system in an extensible language markup (XML) format.
12. The computer-implemented method of claim 9 , wherein the aggregate metric comprises a total count of the plurality of metrics, an average of a plurality of values associated with the plurality of metrics, or a sum of the plurality of values associated with the plurality of metrics.
13. The computer-implemented method of claim 9 , wherein the aggregate metric comprises a maximum one of a plurality of values associated with the plurality of metrics or a minimum one of the plurality of values associated with the plurality of metrics.
14. The computer-implemented method of claim 9 , wherein the aggregate metric is one of a plurality of aggregate metrics.
15. The computer-implemented method of claim 14 , further comprising generating, by the at least one computing device, a data model using the plurality of aggregate metrics.
16. A non-transitory computer-readable medium comprising program code executable in at least one computing device that, when executed by the at least one computing device, causes the at least one computing device to:
access a plurality of metrics generated based on individual ones of a plurality of events performed by a monitored system, wherein the plurality of metrics describe a state of the monitored system with respect to a plurality of timestamps generated at an occurrence of the plurality of events;
generate a performance metric indicative of performance of the monitored system by aggregating the plurality of metrics, the performance metric being generated for at least one of a plurality of periods of time utilizing individual ones of the plurality of metrics having a respective timestamp falling within the at least one of the plurality of periods of time, wherein the performance metric consumes less storage space in memory than the plurality of metrics;
discard the individual ones of the plurality of metrics utilized in generating the performance metric after the performance metric has been generated; and
generate an output describing the state of the monitored system based at least in part on the performance metric.
17. The non-transitory computer-readable medium of claim 16 , wherein the performance metric is an aggregate metric.
18. The non-transitory computer-readable medium of claim 17 , wherein the aggregate metric comprises a total count of the plurality of metrics, an average of a plurality of values associated with the plurality of metrics, or a sum of the plurality of values associated with the plurality of metrics, a maximum one of the plurality of values associated with the plurality of metrics, or a minimum one of the plurality of values associated with the plurality of metrics.
19. The non-transitory computer-readable medium of claim 17 , further comprising program code that, when executed, causes the at least one computing device to generate a data model using the aggregate metric.
20. The non-transitory computer-readable medium of claim 16 , wherein individual ones of the plurality of metrics are associated with at least one dimension.
21. The non-transitory computer-readable medium of claim 16 , wherein individual ones of the plurality of metrics are generated by the monitored system in an extensible language markup (XML) format.