System capacity heatmap
The methods and systems described herein can be used for generating a graphical representation, such as a heatmap, for displaying and visualizing a given metric associated with one or more distributed resources in a distributed environment. A heatmap generally refers to a graphical display of data in which values are located in a matrix and indicated by the intensity of a color, hue, shading or some other gradient, or by geometric shape or other format. Heatmaps provide a way of visualizing metrics of a large number of resources easily for capacity management, among other uses. Generally, and as set forth in greater detail, the disclosed subject matter provided herein includes methods and systems for collecting, processing, and visualizing operational data of distributed resources via a heatmap by a monitoring system in a distributed environment.
1. A method for collecting, processing, and visualizing data of distributed resources in a distributed environment by a monitoring system, the monitoring system comprising at least a collecting server, a processing server, and a terminal device configured to communicate over a network, and wherein the collecting server is configured to communicate with one or more distributed resources over the network, comprising:
receiving, by the collecting server, data associated with a plurality of variables monitored over a period of time at the one or more distributed resources over the network, wherein the data associated with the plurality of variables is included in one or more batch files;
determining, by the one or more distributed resources, whether an error occurred during transmission of the one or more batch files;
generating, in the monitoring system, an error warning associated with the error;
retransmitting, by the one or more distributed resources, the one or more batch files based on the error; and
receiving, by the collecting server, the one or more retransmitted batch files;
storing, by the collecting server, the data associated with the plurality of variables;
calculating, by the processing server, metric values based on the stored data, wherein the metric values correspond to a plurality of time intervals during the period of time;
generating, by the processing server, a graphical representation of the metric values corresponding to the time intervals for each distributed resource, wherein the generating comprises:
determining, based on the metric values, visual indicators to represent each of the metric values at the time intervals for each distributed resource, wherein each of the visual indicators corresponds with one of a plurality of threshold values, and
mapping the visual indicators to a set of cells along a time axis within a window of the graphical representation;
displaying the graphical representation on a terminal device, wherein the graphical representation is a heatmap and wherein the graphical representation includes interactive elements to selectively display a desired metric;
rendering and displaying, on the graphical representation, the metric values corresponding to each of the visual indicators on the terminal device, wherein the metric values are displayed on a portion of the graphical representation corresponding to each metric value respectively; and
recording, by the process server, one or more data breaches of the one or more distributed resources and information corresponding to the one or more data breaches to an information request window of a problem management system, wherein the corresponding information includes at least one of an urgency rating, a status, an impact, an investigation driver, a summary, a note, and a due date, and wherein the problem management system is illustrated within the graphical representation;
monitoring, by the process server, data breaches of the one or more distributed resources recorded in the problem management system;
transferring the recorded data breach or information corresponding to the one or more data breaches between at least the problem management system and a second system or a second program, wherein the second system or the second program is distinct from the problem management system; and
rendering a breach indicator on the graphical representation corresponding to the one or more data breaches, wherein the breach indicator includes an interactive component which enables performance of a plurality of data breach management functions.
2. The method of claim 1 , wherein the metric values include at least one of CPU usage of a first resource of the one or more distributed resources, CPU usage of an individual process on a second resource of the one or more distributed resources, memory usage, memory availability, network usage, network congestion, network latency, or disk latency.
3. The method of claim 1 , wherein the visual indicators representing each of the metric values at said time intervals for each distributed resource include a color, hue, shade, pattern, or marking.
4. The method of claim 1 , wherein the metric values correspond with pre-determined percentiles of usage of each of the one or more distributed resources.
5. A computer system for collecting, processing, and visualizing data of distributed resources in a distributed environment, comprising:
a monitoring system having at least a collecting server, a processing server, and a terminal device configured to communicate over a network, and wherein the collecting server is configured to communicate with one or more distributed resources over the network;
the monitoring system further having a processor and a memory storing instructions that, when executed by the processor, cause the computer system to:
receive, by the collecting server, data associated with a plurality of variables monitored over a period of time at the one or more distributed resources over the network, wherein the data associated with the plurality of variables is included in one or more batch files;
determine, by the one or more distributed resources, whether an error occurred during transmission of the one or more batch files;
generate, in the monitoring system, an error warning associated with the error;
retransmit, by the one or more distributed resources, the one or more batch files based on the errors; and
receive, by the collecting server, the one or more retransmitted batch files;
store, by the collecting server, the data associated with the plurality of variables;
calculate, by the processing server, metric values based on the stored data, wherein the metric values correspond to a plurality of time intervals during the period of time;
generate, by the processing server, a graphical representation of the metric values corresponding to the time intervals for each distributed resource, wherein the generating comprises:
determining, based on the metric values, visual indicators to represent each of the metric values at the time intervals for each distributed resource, wherein each of the visual indicators corresponds with one of a plurality of threshold values, and
mapping the visual indicators to a set of cells along a time axis within a window of the graphical representation;
displaying the graphical representation on a terminal device, wherein the graphical representation is a heatmap and wherein the graphical representation includes interactive elements to selectively display a desired metric;
rendering and displaying, on the graphical representation, the metric values corresponding to each of the visual indicators on the terminal device, wherein the metric values are displayed on a portion of the graphical representation corresponding to each metric value respectively; and
record, by the process server, one or more data breaches of the one or more distributed resources and information corresponding to the one or more data breaches to an information request window of a problem management system, wherein the corresponding information includes at least one of an urgency rating, a status, an impact, an investigation driver, a summary, a note, and a due date, and wherein the problem management system is illustrated within the graphical representation;
monitor, by the process server, data breaches of the one at more distributed resources recorded in the problem management system;
transfer the recorded data breach or information corresponding to the one or more data breaches between at least the problem management system and a second system or a second program, wherein the second system or the second program is distinct from the problem management system; and
render a breach indicator on the graphical representation corresponding to the one or more data breaches, wherein the breach indicator includes an interactive component which enables performance of one of a plurality of data breach management functions.
6. The computer system of claim 5 , wherein the metric values include at least one of CPU usage of a first resource of the one or more distributed resources, CPU usage of an individual process on a second resource of the one or more distributed resources, memory usage, memory availability, network usage, network congestion, network latency, or disk latency.
7. The computer system of claim 5 , wherein the visual indicators representing each of the metric values at said time intervals for each distributed resource include a color, hue, shade, pattern, or marking.
8. The computer system of claim 5 , wherein the metric values correspond with pre-determined percentiles of usage of each of the one or more distributed resources.
9. The computer system of claim 5 , wherein the monitoring component of the monitoring system includes a central server operatively connected to one or more individual servers.
10. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause a computer system to collect, process, and visualize data of distributed resources in a distributed environment by a monitoring system, the monitoring system comprising at least a collecting server, a processing server, and a terminal device configured to communicate over a network, and wherein the collecting server is configured to communicate with one or more distributed resources over the network, by performing the steps of:
receiving, by the collecting server, data associated with a plurality of variables monitored over a period of time at the one or more distributed resources over the network, wherein the data associated with the plurality of variables is included in one or more batch files;
determining, by the one or more distributed resources, whether an error occurred during transmission of the one or more batch files;
generating, in the monitoring system, an error warning associated with the error;
retransmitting, by the one or more distributed resources, the one or more batch files based on the error; and
receiving, by the collecting server, the one or more retransmitting batch files;
storing, by the collecting server, the data associated with the plurality of variables;
calculating, by the processing server, metric values based on the stored data, wherein the metric values correspond to a plurality of time intervals during the period of time;
generating, by the processing server, a graphical representation of the metric values corresponding to the time intervals for each distributed resource, wherein the generating comprises;
determining, based on the metric values, visual indicators to represent each of the metric values at the time intervals for each distributed resource, wherein each of the visual indicators corresponds with one of a plurality of threshold values, and
mapping the visual indicators to a set of cells along a time axis within a window of the graphical representation;
displaying the graphical representation on a terminal device, wherein the graphical representation is a heatmap and wherein the graphical representation includes interactive elements to selectively display a desired metric;
rendering and displaying, on the graphical representation, the metric values corresponding to each of the visual indicators on the terminal device, wherein the metric values are displayed on a portion of the graphical representation corresponding to each metric value respectively; and
recording, by the process server, one or more data breaches of the one or more distributed resources and information corresponding to the one or more data breaches to an information request window of a problem management system, wherein the corresponding information includes at least one of an urgency rating, a status, an impact, an investigation driver, a summary, a note, and a due date, and wherein the problem management system is illustrated within the graphical representation;
monitoring, by the process server, data breaches of the one or more distributed resources recorded in the problem management system;
transferring the recorded data breach or information corresponding to the one or more data breaches between at least the problem management system and a second system or a second program, wherein the second system or the second program is distinct from the problem management system; and
rendering a breach indicator on the graphical representation corresponding to the one or more data breaches, wherein the breach indicator includes an interactive component which enables performance of a plurality of data breach management functions.
11. The non-transitory computer-readable medium of claim 10 , wherein the metric values include at least one of CPU usage of a first resource of the one or more distributed resources, CPU usage of an individual process on a second resource of the one or more distributed resources, memory usage, memory availability, network usage, network congestion, network latency, or disk latency.
12. The non-transitory computer-readable medium of claim 10 , wherein the visual indicators representing each of the metric values at said time intervals for each distributed resource include a color, hue, shade, pattern, or marking.
13. The non-transitory computer-readable medium of claim 10 , wherein the metric values correspond with pre-determined percentiles of usage of each of the one or more distributed resources.