Progress estimation of iterative hierarchical clustering algorithms
An example method includes initiating training of an hierarchical clustering algorithm using training data. The method further includes determining a first factor, the first factor being a number of analyzed nodes compared to a number of discovered nodes. The method further includes determining a second factor, the second factor being a first time elapsed compared to a first typical training duration for environments with a data set size substantially similar to a data set size of the training data. The method further includes determining a third factor, the third factor being a second time elapsed compared to a second typical training duration for environments with a data having a uniformity substantially similar to a uniformity of the training data. The method further includes estimating a progress of the training of the hierarchical clustering algorithm based at least in part on the first factor, the second factor, and the third factor.
1 . A computer-implemented method for progress estimation for training hierarchical clustering algorithms, the computer-implemented method comprising:
initiating training of an hierarchical clustering algorithm for an application using training data;
calculating, while the hierarchical clustering algorithm is being trained, a first factor, the first factor being a number of analyzed nodes compared to a number of discovered nodes, wherein the analyzed nodes and the discovered nodes are continuously updated throughout training of the hierarchical clustering algorithm;
determining a second factor, the second factor being a first time elapsed compared to a first typical training duration for environments with a data set size substantially similar to a data set size of the training data, wherein the first time elapsed is continuously updated;
determining a third factor, the third factor being a second time elapsed compared to a second typical training duration for environments with a data having a uniformity substantially similar to a uniformity of the training data, wherein the second time elapsed is continuously updated; and
estimating a progress of the training of the hierarchical clustering algorithm based at least in part on the first factor, the second factor, and the third factor.
2 . The computer-implemented method of claim 1 , wherein the first factor is determined during the training of the hierarchical clustering algorithm, and wherein the second factor and the third factor are determined in part prior to initiating the training of the hierarchical clustering algorithm.
3 . The computer-implemented method of claim 1 , wherein the second factor and the third factor are determined by gathering data about duration, data volume, and data compression from a plurality of deployments where the training has been performed previously.
4 . The computer-implemented method of claim 1 , wherein the second factor and the third factor are scaled based at least in part on a hardware configuration of a device performing the training.
5 . The computer-implemented method of claim 1 , wherein the progress of the training is based at least in part on a minimum percentage reported by each of the first factor, the second factor, and the third factor.
6 . The computer-implemented method of claim 1 , further comprising updating the progress of the training over time to a new progress responsive to determining that the new progress is greater than the progress.
7 . The computer-implemented method of claim 1 , further comprising iteratively updating the first factor, the second factor, and the third factor while training the hierarchical clustering algorithm.
8 . A system comprising:
a memory comprising computer readable instructions; and
a processing device for executing the computer readable instructions, the computer readable instructions controlling the processing device to perform operations for progress estimation for training hierarchical clustering algorithms, the operations comprising:
initiating training of an hierarchical clustering algorithm for an application using training data;
calculating, while the hierarchical clustering algorithm is being trained, a first factor, the first factor being a number of analyzed nodes compared to a number of discovered nodes, wherein the analyzed nodes and the discovered nodes are continuously updated throughout training of the hierarchical clustering algorithm;
determining a second factor, the second factor being a first time elapsed compared to a first typical training duration for environments with a data set size substantially similar to a data set size of the training data, wherein the first time elapsed is continuously updated;
determining a third factor, the third factor being a second time elapsed compared to a second typical training duration for environments with a data having a uniformity substantially similar to a uniformity of the training data, wherein the second time elapsed is continuously updated; and
estimating a progress of the training of the hierarchical clustering algorithm based at least in part on the first factor, the second factor, and the third factor.
9 . The system of claim 8 , wherein the first factor is determined during the training of the hierarchical clustering algorithm, and wherein the second factor and the third factor are determined in part prior to initiating the training of the hierarchical clustering algorithm.
10 . The system of claim 8 , wherein the second factor and the third factor are determined by gathering data about duration, data volume, and data compression from a plurality of deployments where the training has been performed previously.
11 . The system of claim 8 , wherein the second factor and the third factor are scaled based at least in part on a hardware configuration of a device performing the training.
12 . The system of claim 8 , wherein the progress of the training is based at least in part on a minimum percentage reported by each of the first factor, the second factor, and the third factor.
13 . The system of claim 8 , wherein the operations further comprise updating the progress of the training over time to a new progress responsive to determining that the new progress is greater than the progress.
14 . The system of claim 8 , wherein the operations further comprise iteratively updating the first factor, the second factor, and the third factor while training the hierarchical clustering algorithm.
15 . A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform operations for progress estimation for training hierarchical clustering algorithms, the operations comprising:
initiating training of an hierarchical clustering algorithm for an application using training data;
calculating, while the hierarchical clustering algorithm is being trained, a first factor, the first factor being a number of analyzed nodes compared to a number of discovered nodes, wherein the analyzed nodes and the discovered nodes are continuously updated throughout training of the hierarchical clustering algorithm;
determining a second factor, the second factor being a first time elapsed compared to a first typical training duration for environments with a data set size substantially similar to a data set size of the training data, wherein the first time elapsed is continuously updated;
determining a third factor, the third factor being a second time elapsed compared to a second typical training duration for environments with a data having a uniformity substantially similar to a uniformity of the training data, wherein the second time elapsed is continuously updated; and
estimating a progress of the training of the hierarchical clustering algorithm based at least in part on the first factor, the second factor, and the third factor.
16 . The computer program product of claim 15 , wherein the first factor is determined during the training of the hierarchical clustering algorithm, and wherein the second factor and the third factor are determined in part prior to initiating the training of the hierarchical clustering algorithm.
17 . The computer program product of claim 15 , wherein the second factor and the third factor are determined by gathering data about duration, data volume, and data compression from a plurality of deployments where the training has been performed previously.
18 . The computer program product of claim 15 , wherein the second factor and the third factor are scaled based at least in part on a hardware configuration of a device performing the training.
19 . The computer program product of claim 15 , wherein the progress of the training is based at least in part on a minimum percentage reported by each of the first factor, the second factor, and the third factor.
20 . The computer program product of claim 15 , wherein the operations further comprise updating the progress of the training over time to a new progress responsive to determining that the new progress is greater than the progress.