Hyperparameter tuning in autoregressive integrated moving average (ARIMA) models
A system and method include tuning hyperparameters for an ARIMA model using a derivative free approach by determining a set of initial hyperparameter values, fitting an ARIMA model to the set of initial hyperparameter values, selecting a tuning method for the set of hyperparameters, responsive to selecting a single-objective method, computing a first objective function value from time-series data applied to the ARIMA model based on the set of initial hyperparameter values, or responsive to selecting a multi-objective method, computing at least a second objective function value and a third objective function value from the time-series data applied to the ARIMA model based on the set of initial hyperparameter values, determining whether a stopping criterion for tuning the set of hyperparameters has reached, responsive to determining that the stopping criteria has reached, outputting a set of tuned hyperparameter values.
1. A non-transitory computer-readable medium comprising computer-readable instructions stored thereon that when executed by a first processor associated with a main machine and a second processor associated with each of a plurality of worker machines of a distributed computing system, cause the first processor and the second processor to:
receive, via a user interface associated with the distributed computing system, a time series dataset;
tune a set of hyperparameters for the time series dataset in a plurality of iterations, wherein the plurality of worker machines execute at least some of the plurality of iterations in parallel, and wherein to tune the set of hyperparameters, the computer-readable instructions further cause the first processor and the second processor to:
(A) determine, by each worker machine of the plurality of worker machines, an initial hyperparameter value for each hyperparameter in the set of hyperparameters to be tuned to generate a set of initial hyperparameter values for the time series dataset;
(B) fit, by each worker machine of the plurality of worker machines, an Auto-Regressive Integrated Moving Average (ARIMA) model to the set of initial hyperparameter values;
(C) receive selection, at the main machine or at each of the plurality of worker machines, of a tuning method for the set of hyperparameters via the user interface of the distributed computing system, wherein the tuning method comprises either a single-objective method or a multi-objective method;
(D) responsive to receiving selection of the single-objective method, compute, by each worker machine of the plurality of worker machines, a first objective function value from time-series data applied to the ARIMA model based on the set of initial hyperparameter values, or responsive to receiving selection of the multi-objective method, compute, by each worker machine of the plurality of worker machines, at least a second objective function value and a third objective function value from the time-series data applied to the ARIMA model based on the set of initial hyperparameter values, wherein to compute the third objective function value, the computer-readable instructions further cause each worker machine of the plurality of worker machines to compute a Model Residual Validity (MRV) score measuring how close an array of model residuals of the ARIMA model are to white noise, wherein the MRV score is computed as:
MRV=(|U acf |+|U pacf |+|U wn |)/3n, where U acf ={1≤i≤n|ACF([i])≥tol 1 ×(standard error of ACF([i]))}, U pacf ={1≤i≤n|PACF([i])≥tol 2 ×(standard error of PACF([i]))}, U wn ={1≤i≤n|(WNP(i)≥tol 3 }, tol 1 , tol 2 , tol 3 are predetermined threshold values, n is the number of lags, ACF is an auto-correlation function of ε, PACF is a partial auto-correlation function of ε; WNP is a white noise probability function of ε, and ε are model residuals;
(E) send, by each worker machine of the plurality of worker machines, the computed first objective function value for the single-objective method or the at least the second objective function value and the third objective function value for the multi-objective method to the main machine;
(F) determine, at the main machine, whether a stopping criterion for tuning the set of hyperparameters has reached; and
(G) responsive to determining, at the main machine, that the stopping criterion has not reached, send instructions by the main machine to each worker machine of the plurality of worker machines to update the initial hyperparameter value for each hyperparameter in the set of hyperparameters to an updated hyperparameter value to generate a set of updated hyperparameter values, and repeat (B) to (G) with the set of updated hyperparameter values instead of the set of initial hyperparameter values, or responsive to determining that the stopping criteria has reached,
determine, by the main machine, a set of tuned hyperparameter values comprising a tuned hyperparameter value for each hyperparameter in the set of hyperparameters, wherein:
responsive to determining that the single-objective method is received in (C), the main machine compares the first objective function value of each hyperparameter of the set of hyperparameters from each of the plurality of iterations and selects a lowest value of the first objective function value as the tuned hyperparameter value for that hyperparameter, or
responsive to determining that the multi-objective method is received in (C), the main machine determines the tuned hyperparameter value of each hyperparameter of the set of hyperparameters based on either the second objective function value or the third objective function value from the plurality of iterations; and
display the set of tuned hyperparameter values on the user interface associated with the distributed computing system.
2. The non-transitory computer-readable medium of claim 1 , wherein to generate the set of initial hyperparameter values, the computer-readable instructions further cause at least one of the first processor or the second processor to:
receive a value range of each hyperparameter in the set of hyperparameters; and
select a value from the value range of each hyperparameter as the initial hyperparameter value for that hyperparameter.
3. The non-transitory computer-readable medium of claim 2 , wherein to generate the set of updated hyperparameter values, the computer-readable instructions further cause at least one of the first processor or the second processor to select another value from the value range of each hyperparameter as the updated hyperparameter value for that hyperparameter.
4. The non-transitory computer-readable medium of claim 1 , wherein the ARIMA model comprises a full ARIMA model.
5. The non-transitory computer-readable medium of claim 1 , wherein the ARIMA model comprises a subset ARIMA model.
6. The non-transitory computer-readable medium of claim 1 , wherein to compute the first objective function value, the computer-readable instructions further cause the second processor to:
compute a first function value from in-sample data selected from the time-series data; or
compute a second function value from holdout data selected from the time-series data.
7. The non-transitory computer-readable medium of claim 6 , wherein the first function value comprises an Akaike's information criterion (AICc) value, and wherein the second function value comprises a root mean square error value.
8. The non-transitory computer-readable medium of claim 1 , wherein:
the multi-objective method comprises a first multi-objective method or a second multi-objective method;
each of the first multi-objective method and the second multi-objective method comprises the second objective function value, the third objective function value, and a fourth objective function value;
to compute the second objective function value, the computer-readable instructions further cause the second processor to compute a first function value on holdout data selected from the time-series data;
and
the set of updated hyperparameter values that are output as the set of tuned hyperparameter values are based on either the second objective function value, the third objective function value, or the fourth objective function value.
9. The non-transitory computer-readable medium of claim 8 , wherein to compute the fourth objective function value for the second multi-objective method, the computer-readable instructions further cause the second processor to compute a second function value from in-sample data selected from the time-series data.
10. The non-transitory computer-readable medium of claim 9 , wherein the first function value comprises a root mean square error value and the second function value comprises an Akaike's information criterion (AICc) value.
11. The non-transitory computer-readable medium of claim 8 , wherein:
to compute the fourth objective function value for the first multi-objective method, the computer-readable instructions further cause the second processor to compute a complexity value indicating a number of hyperparameters in the set of hyperparameters to estimate for the ARIMA model;
the complexity value for a full ARIMA model is computed as p+q+P+Q, where p is a non-seasonal autoregressive order, q is a non-seasonal moving average order, P is a seasonal autoregressive order, Q is a seasonal moving average order; and the complexity value for a subset ARIMA model is computed as |S p |+|S q |+|S P |+|S Q |, where S p , S P , S q , and S Q are cardinalities of subsets of non-seasonal AR orders, seasonal AR orders, non-seasonal MA orders, and seasonal MA models, respectively, in the subset ARIMA model.
12. The non-transitory computer-readable medium of claim 8 , wherein to output the set of updated hyperparameter values as the set of tuned hyperparameter values based on either the second objective function value, the third objective function value, or the fourth objective function value, the computer-readable instructions further cause at least one of the first processor or the second processor to:
generate a set of Pareto front outputs based on the second objective function value, the third objective function value, and the fourth objective function value;
compute a third function value from out-of-sample data in the time-series data for each Pareto front output in the set of Pareto front outputs; and
select the set of updated hyperparameter values corresponding to the Pareto front output having an optimal value of the third function value as the set of tuned hyperparameter values.
13. The non-transitory computer-readable medium of claim 12 , wherein the optimal value comprises either a highest value of the third function value or a lowest value of the third function value.
14. The non-transitory computer-readable medium of claim 12 , wherein the third function value comprises a root mean square error value.
15. The non-transitory computer-readable medium of claim 1 , wherein the computer-readable instructions further cause at least one of the first processor or the second processor to predict a future data point in the time-series data by applying the set of tuned hyperparameter values to the ARIMA model.
16. The non-transitory computer-readable medium of claim 1 , wherein the plurality of iterations comprises one or more computing sessions.
17. A system comprising:
a memory having computer-readable instructions stored thereon; and
a first processor associated with a main machine and a second processor associated with each of a plurality of worker machines of a distributed computing system that execute the computer-readable instructions to:
receive, via a user interface associated with the distributed computing system, a time series dataset;
tune a set of hyperparameters for the time series dataset in a plurality of iterations, wherein the plurality of worker machines execute at least some of the plurality of iterations in parallel, and wherein to tune the set of hyperparameters, the computer-readable instructions further cause the first processor and the second processor to:
(A) determine, by each worker machine of the plurality of worker machines, an initial hyperparameter value for each hyperparameter in the set of hyperparameters to be tuned to generate a set of initial hyperparameter values for the time series dataset;
(B) fit, by each worker machine of the plurality of worker machines, an Auto-Regressive Integrated Moving Average (ARIMA) model to the set of initial hyperparameter values;
(C) receive selection, at the main machine or at each of the plurality of worker machines, of a tuning method for the set of hyperparameters via the user interface of the distributed computing system, wherein the tuning method comprises either a single-objective method or a multi-objective method;
(D) responsive to receiving selection of the single-objective method, compute, by each worker machine of the plurality of worker machines, a first objective function value from time-series data applied to the ARIMA model based on the set of initial hyperparameter values, or responsive to receiving selection of the multi-objective method, compute, by each worker machine of the plurality of worker machines, at least a second objective function value and a third objective function value from the time-series data applied to the ARIMA model based on the set of initial hyperparameter values, wherein to compute the third objective function value, the computer-readable instructions further cause each worker machine of the plurality of worker machines to compute a Model Residual Validity (MRV) score measuring how close an array of model residuals of the ARIMA model are to white noise, wherein the MRV score is computed as:
MRV=(|U acf |+|U pacf |+|U wn |)/3n, where U acf ={1≤i≤n|ACF([i])≥tol 1 ×(standard error of ACF([i]))}, U pacf ={1≤i≤n|PACF([i])≥tol 2 ×(standard error of PACF([i]))}, U wn ={1≤i≤n|(WNP(i)≥tol 3 }, tol 1 , tol 2 , tol 3 are predetermined threshold values, n is the number of lags, ACF is an auto-correlation function of ε, PACF is a partial auto-correlation function of ε; WNP is a white noise probability function of ε, and a are model residuals;
(E) send, by each worker machine of the plurality of worker machines, the computed first objective function value for the single-objective method or the at least the second objective function value and the third objective function value for the multi-objective method to the main machine:
(F) determine, at the main machine, whether a stopping criterion for tuning the set of hyperparameters has reached; and
(G) responsive to determining, at the main machine, that the stopping criterion has not reached, send instructions by the main machine to each worker machine of the plurality of worker machines to update the initial hyperparameter value for each hyperparameter in the set of hyperparameters to an updated hyperparameter value to generate a set of updated hyperparameter values, and repeat (B) to (G) with the set of updated hyperparameter values instead of the set of initial hyperparameter values, or responsive to determining that the stopping criteria has reached,
determine, by the main machine, a set of tuned hyperparameter values comprising a tuned hyperparameter value for each hyperparameter in the set of hyperparameters, wherein:
responsive to determining that the single-objective method is received in (C), the main machine compares the first objective function value of each hyperparameter of the set of hyperparameters from each of the plurality of iterations and selects a lowest value of the first objective function value as the tuned hyperparameter value for that hyperparameter, or
responsive to determining that the multi-objective method is received in (C), the main machine determines the tuned hyperparameter value of each hyperparameter of the set of hyperparameters based on either the second objective function value or the third objective function value from the plurality of iterations; and
display the set of tuned hyperparameter values on the user interface associated with the distributed computing system.
18. The system of claim 17 , wherein to generate the set of initial hyperparameter values, the computer-readable instructions further cause at least one of the first processor or the second processor to:
receive a value range of each hyperparameter in the set of hyperparameters; and
select a value from the value range of each hyperparameter as the initial hyperparameter value for that hyperparameter.
19. The system of claim 18 , wherein to generate the set of updated hyperparameter values, the computer-readable instructions further cause the at least one of the first processor or the second processor to select another value from the value range of each hyperparameter as the updated hyperparameter value for that hyperparameter.
20. The system of claim 17 , wherein the ARIMA model comprises a full ARIMA model or a subset ARIMA model.
21. The system of claim 17 , wherein to compute the first objective function value, the computer-readable instructions further cause the second processor to:
compute a first function value from in-sample data selected from the time-series data; or
compute a second function value from holdout data selected from the time-series data.
22. The system of claim 21 , wherein the first function value comprises an Akaike's information criterion (AICc) value, and wherein the second function value comprises a root mean square error value.
23. The system of claim 17 , wherein:
the multi-objective method comprises a first multi-objective method or a second multi-objective method;
each of the first multi-objective method and the second multi-objective method comprises the second objective function value, the third objective function value, and a fourth objective function value;
to compute the second objective function value, the computer-readable instructions further cause the second processor to compute a root mean square error value on holdout data selected from the time-series data;
to compute the fourth objective function value for the second multi-objective method, the computer-readable instructions further cause the second processor to compute an Akaike's information criterion (AICc) value from in-sample data selected from the time-series data or to compute the fourth objective function value for the first multi-objective method, the computer-readable instructions further cause the second processor to compute a complexity value indicating a number of hyperparameters in the set of hyperparameters to estimate for the ARIMA model, wherein the complexity value for a full ARIMA model is computed as p+q+P+Q, where p is a non-seasonal autoregressive order, q is a non-seasonal moving average order, P is a seasonal autoregressive order, Q is a seasonal moving average order, and wherein the complexity value for a subset ARIMA model is computed as |S p |+|S q |+|S P |+|S Q |, where S p , S P , S q , and S Q are cardinalities of subsets of non-seasonal AR orders, seasonal AR orders, non-seasonal MA orders, and seasonal MA models, respectively, in the subset ARIMA model; and
the set of updated hyperparameter values that are output as the set of tuned hyperparameter values are based on either the second objective function value, the third objective function value, or the fourth objective function value.
24. The system of claim 23 , wherein to output the set of updated hyperparameter values as the set of tuned hyperparameter values based on either the second objective function value, the third objective function value, or the fourth objective function value, the computer-readable instructions further cause at least one of the first processor or the second processor to:
generate a set of Pareto front outputs based on the second objective function value, the third objective function value, and the fourth objective function value;
compute a third function value from out-of-sample data in the time-series data for each Pareto front output in the set of Pareto front outputs; and
select the set of updated hyperparameter values corresponding to the Pareto front output having an optimal value of the third function value as the set of tuned hyperparameter values,
wherein the optimal value comprises either a highest value of the third function value or a lowest value of the third function value; and
wherein the third function value comprises a root mean square error value.
25. The system of claim 17 , wherein the computer-readable instructions further cause at least one of the first processor or the second processor to predict a future data point in the time-series data by applying the set of tuned hyperparameter values to the ARIMA model.
26. The system of claim 17 , wherein the plurality of iterations comprises one or more computing sessions.
27. A method comprising:
receiving, via a user interface associated with a distributed computing system having a first processor associated with a main machine and a second processor associated with each of a plurality of worker machines, a time series dataset;
tuning, by the first processor and the second processor executing computer-readable instructions stored on a memory, a set of hyperparameters for the time series dataset in a plurality of iterations, wherein the plurality of worker machines execute at least some of the plurality of iterations in parallel, and wherein to tune the set of hyperparameters, the computer-readable instructions further cause the first processor and the second processor to:
(A) determine, by each worker machine of the plurality of worker machines, an initial hyperparameter value for each hyperparameter in the set of hyperparameters to be tuned to generate a set of initial hyperparameter values for the time series dataset;
(B) fit, by each worker machine of the plurality of worker machines, an Auto-Regressive Integrated Moving Average (ARIMA) model to the set of initial hyperparameter values;
(C) receive, at the main machine or at each of the plurality of worker machines, selection of a tuning method for the set of hyperparameters via the user interface of the distributed computing system, wherein the tuning method comprises either a single-objective method or a multi-objective method;
(D) responsive to receiving selection of the single-objective method, compute, by each worker machine of the plurality of worker machines, a first objective function value from time-series data applied to the ARIMA model based on the set of initial hyperparameter values, or responsive to receiving selection of the multi-objective method, compute, by each worker machine of the plurality of worker machines, at least a second objective function value and a third objective function value from the time-series data applied to the ARIMA model based on the set of initial hyperparameter values, wherein to compute the third objective function value, the computer-readable instructions further cause each worker machine of the plurality of worker machines to compute a Model Residual Validity (MRV) score measuring how close an array of model residuals of the ARIMA model are to white noise, wherein the MRV score is computed as:
MRV=(|U acf +|U pacf |+|U wn |)/3n, where U acf ={1≤i≤n|ACF([i])≥tol 1 ×(standard error of ACF([i]))}, U pacf ={1≤i≤n|PACF([i])≥tol 2 ×(standard error of PACF([i]))}, U wn ={1≤i≤n|(WNP(i)≥tol 3 }, tol 1 , tol 2 , tol 3 are predetermined threshold values, n is the number of lags, ACF is an auto-correlation function of ε, PACF is a partial auto-correlation function of ε; WNP is a white noise probability function of ε, and a are model residuals;
(E) send, by each worker machine of the plurality of worker machines, the computed first objective function value for the single-objective method or the at least the second objective function value and the third objective function value for the multi-objective method to the main machine;
(F) determine, at the main machine, whether a stopping criterion for tuning the set of hyperparameters has reached; and
(G) responsive to determining, at the main machine, that the stopping criterion has not reached, send instructions by the main machine to each worker machine of the plurality of worker machines for updating the initial hyperparameter value for each hyperparameter in the set of hyperparameters to an updated hyperparameter value to generate a set of updated hyperparameter values, and repeat, by the processor, (B) to (G) with the set of updated hyperparameter values instead of the set of initial hyperparameter values, or responsive to determining that the stopping criteria has reached,
determine, by the main machine, a set of tuned hyperparameter values comprising a tuned hyperparameter value for each hyperparameter in the set of hyperparameters, wherein:
responsive to determining that the single-objective method is received in (C), compare, by the main machine, the first objective function value of each hyperparameter of the set of hyperparameters from each of the plurality of iterations and select a lowest value of the first objective function value as the tuned hyperparameter value for that hyperparameter, or
responsive to determining that the multi-objective method is received in (C), determine, by the main machine, the tuned hyperparameter value of each hyperparameter of the set of hyperparameters based on either the second objective function value or the third objective function value from the plurality of iterations; and
display the set of tuned hyperparameter values on the user interface associated with the distributed computing system.
28. The method of claim 27 , further comprising predicting, by at least one of the first processor or the second processor, a future data point in the time-series data by applying the set of tuned hyperparameter values to the ARIMA model.
29. The method of claim 27 , wherein the multi-objective method comprises a first multi-objective method or a second multi-objective method, wherein each of the first multi-objective method and the second multi-objective method comprises the second objective function value, the third objective function value, and a fourth objective function value, and wherein the method further comprises:
computing, by the second processor, a root mean square error value on holdout data selected from the time-series data for computing the second objective function value;
computing, by the second processor, an Akaike's information criterion (AICc) value from in-sample data selected from the time-series data for computing the fourth objective function value for the second multi-objective method or computing, by the second processor, a complexity value indicating a number of hyperparameters in the set of hyperparameters to estimate for the ARIMA model for computing the fourth objective function value for the first multi-objective method, wherein the complexity value for a full ARIMA model is computed as p+q+P+Q, where p is a non-seasonal autoregressive order, q is a non-seasonal moving average order, P is a seasonal autoregressive order, Q is a seasonal moving average order, and wherein the complexity value for a subset ARIMA model is computed as |S p |+|S q |+|S P |+|S Q |, where S p , S P , S q , and S Q are cardinalities of subsets of non-seasonal AR orders, seasonal AR orders, non-seasonal MA orders, and seasonal MA models, respectively, in the subset ARIMA model;
generating, by at least one of the first processor or the second processor, a set of Pareto front outputs based on the second objective function value, the third objective function value, and the fourth objective function value;
computing, by at least one of the first processor or the second processor, a third function value from out-of-sample data in the time-series data for each Pareto front output in the set of Pareto front outputs; and
selecting, by at least one of the first processor or the second processor, the set of updated hyperparameter values corresponding to the Pareto front output having an optimal value of the third function value as the set of tuned hyperparameter values,
wherein the optimal value comprises either a highest value of the third function value or a lowest value of the third function value; and
wherein the third function value comprises a root mean square error value.
30. The method of claim 27 , wherein for computing the first objective function value, the method further comprises:
computing, by the second processor, a first function value from in-sample data selected from the time-series data; or
computing, by the second processor, a second function value from holdout data selected from the time-series data,
wherein the first function value comprises an Akaike's information criterion (AICc) value, and
wherein the second function value comprises a root mean square error value.