Fault prevention
View Patent ↗Methods, systems, and apparatus, including computer programs encoded on computer storage media, for identifying a plurality of software components within a cluster of computing nodes, each component operating on one or more respective nodes within the cluster to process a workload; determining, for each identified component, a respective resource usage trend for the component having a respective range of predicted values; comparing, for each identified component, the respective range of predicted values to a respective range of suitable values; and generating, for each component with the respective range of predicted values that is outside of the respective range of suitable values, a respective recommendation for configuring the component to cause the component to generate an updated respective resource usage trend having an updated respective range of predicted values that are inside of the respective range of suitable values.
1. A method, performed by at least one computing device, comprising:
receiving metrics from a cluster of computer nodes relating to a heap size of a NameNode of a Hadoop Distributed File System on the cluster, the metrics including
a number of directories over a preceding period,
a number of predicted files over the preceding period,
a number of predicted blocks over the preceding period,
a number of predicted snapshot objects over the preceding period,
a size per metadata object (O),
a number of transactions (T), and
an average memory needed per transaction (M);
predicting, by linear regression, corresponding values for the received metrics over a future period, the corresponding values including
a number of directories over the future period (D),
a number of predicted files over the future period (F),
a number of predicted blocks over the future period (B), and
a number of predicted snapshot objects over the future period (S);
determining a value for the heap size of the NameNode over the future period, including
computing an amount of memory MD needed for the metadata objects according to MD=(D+F+B+S)*O,
computing an amount of memory TM needed for the number of transactions according to TM=T*M, and
computing the value for the heap size as MD+TM times a safety factor; and
outputting fault preventive recommendations based at least on the determined value for the heap size.
2. The method of claim 1 , wherein the safety factor is 1.2.
3. The method of claim 1 , wherein the preceding period is six months or twelve months.
4. A system comprising a cluster of computer nodes running a Hadoop Distributed File System, the system comprising:
at least one hardware computing device;
a metrics analyzer configured to be executing by the at least one hardware computing device to perform operations comprising:
receiving metrics from the cluster of computer nodes relating to a heap size of a NameNode of the Hadoop Distributed File System on the cluster, the metrics including
a number of directories over a preceding period,
a number of predicted files over the preceding period,
a number of predicted blocks over the preceding period,
a number of predicted snapshot objects over the preceding period,
a size per metadata object (O),
a number of transactions (T), and
an average memory needed per transaction (M);
a recommendation engine configured to be executing by the at least one hardware computing device to perform operations comprising:
predicting, by linear regression, corresponding values for the received metrics over a future period, the corresponding values including
a number of directories over the future period (D),
a number of predicted files over the future period (F),
a number of predicted blocks over the future period (B), and
a number of predicted snapshot objects over the future period (S);
determining a value for the heap size of the NameNode over the future period, including
computing an amount of memory MD needed for the metadata objects according to MD=(D+F+B+S)*O,
computing an amount of memory TM needed for the number of transactions according to TM=T*M, and
computing the value for the heap size as MD+TM times a safety factor; and
outputting fault preventive recommendations based at least on the determined value for the heap size.
5. The system of claim 4 , wherein the safety factor is 1.2.
6. The system of claim 4 , wherein the preceding period is six months or twelve months.
7. A non-transitory medium storing computer program instructions configured to cause a system to perform operations comprising:
receiving metrics from a cluster of computer nodes relating to a heap size of a NameNode of a Hadoop Distributed File System on the cluster, the metrics including
a number of directories over a preceding period,
a number of predicted files over the preceding period,
a number of predicted blocks over the preceding period,
a number of predicted snapshot objects over the preceding period,
a size per metadata object (O),
a number of transactions (T), and
an average memory needed per transaction (M);
predicting, by linear regression, corresponding values for the received metrics over a future period, the corresponding values including
a number of directories over the future period (D),
a number of predicted files over the future period (F),
a number of predicted blocks over the future period (B), and
a number of predicted snapshot objects over the future period (S);
determining a value for the heap size of the NameNode over the future period, including
computing an amount of memory MD needed for the metadata objects according to MD=(D+F+B+S)*O,
computing an amount of memory TM needed for the number of transactions according to TM=T*M, and
computing the value for the heap size as MD+TM times a safety factor; and
outputting fault preventive recommendations based at least on the determined value for the heap size.
8. The non-transitory medium of claim 7 , wherein the safety factor is 1.2.
9. The non-transitory medium of claim 7 , wherein the preceding period is six months or twelve months.