Tool for hyperparameter tuning

1. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a computing device to:

receive factor information indicating multiple factors, the multiple factors for designing a system by assigning one of respective possible options to each of the multiple factors;

receive range information indicating initial ranges with a respective range for each of the respective possible options,

wherein at least one of the multiple factors is a continuous factor that defines continuous values within a given range of the initial ranges;

wherein the initial ranges define a design space; and

wherein the range information indicates a first range of the initial ranges for an experimental factor of the multiple factors;

obtain a space-filling design for the design space,

wherein the space-filing design indicates selected design points in the design space,

wherein each of the selected design points represents assigned options assigned to the multiple factors, the assigned options assigned from the initial ranges, and

wherein the selected design points are separated from one another in the design space based on a criterion for separating the selected design points in the design space;

generate, based on the space-filling design, an initial design suite that provides initial design cases corresponding to one or more of the selected design points, wherein each element of a respective design case of the initial design suite is one of the assigned options represented by a selected design point of the one or more of the selected design points;

generate evaluations of the initial design cases by modeling respective initial responses of the system for each of the initial design cases, wherein each of the respective initial responses corresponds to an operation of the system defined by each element of a given respective initial design case of the initial design suite; and

output, based on the evaluations of the initial design cases, an indication of a selected design case by:

determining, based on the evaluations of the initial design cases, a first design case with a first design option for the experimental factor;

determining an updated range for the experimental factor, wherein the updated range comprises the first design option and the updated range is a subset of the first range;

generating an updated design suite that provides updated design cases for the system according to the updated range for the experimental factor, wherein each element of an updated design case corresponds to one of the multiple factors, and wherein one element of each of the updated design cases corresponds to the experimental factor and is selected from the updated range for the experimental factor;

generating evaluations of the updated design cases by modeling respective updated responses of the system for one or more of the updated design cases, wherein each of the respective updated responses corresponds to an operation of the system defined by each element of a given updated design case of the updated design suite; and

outputting, based on the evaluations of the one or more of the updated design cases, the indication of the selected design case.

2. The computer-program product of claim 1 , wherein the instructions are operable to cause a computing device to determine, based on the evaluations of the initial design cases, the first design case by generating a generated design case different than all of the initial design cases of the initial design suite based on a correlation of individual design options of the assigned options for factors of the multiple factors and the evaluations of the initial design cases.

3. The computer-program product of claim 1 , wherein the instructions are operable to cause a computing device to determine the updated range for the experimental factor by:

determining a respective updated range for each factor of the multiple factors, wherein a given respective updated range is a subset of an initial range of the initial ranges corresponding to a same factor of the multiple factors; and

generating an updated design suite that provides design cases for the system according to the respective updated range for each factor of the multiple factors.

4. The computer-program product of claim 1 ,

wherein the design space is an initial design space and the space-filling design is an initial space-filling design for the initial design space; and

wherein the instructions are operable to cause a computing device to generate the updated design suite by:

determining an updated design space defined by the updated range for the experimental factor;

obtaining an updated space-filling design based on a criterion for separating selected design points in the updated design space; and

generating the updated design suite based on the updated space-filling design.

5. The computer-program product of claim 1 , wherein the instructions are operable to cause a computing device to:

receive the range information by obtaining a predefined percentage for each respective initial range; and

determine the updated range for the experimental factor by using the predefined percentage to determine the subset of the first range around the first design option.

6. The computer-program product of claim 1 , wherein the instructions are operable to cause a computing device to:

determine, based on the evaluations of the initial design cases, the first design case with the first design option for the experimental factor of the multiple factors by selecting a subset of the initial design cases;

determine a respective updated range for each factor of each design case of the subset of the initial design cases, wherein a given respective updated range corresponding a same factor in each design case is different;

generate respective updated design suites for each design case of the subset;

evaluate respective responses of the system defined by updated options of each of the design cases of the respective updated design suites; and

select, based on the evaluation of the updated design suite, a design case from or generated based on one of the respective updated design suites.

7. The computer-program product of claim 1 , wherein the instructions are operable to cause a computing device to:

for N iterations, where N is a numeral greater than 0:

set a refined range that is a subset of a range of a previous updated design suite;

generate a further updated design suite that provides design cases for the system according to the refined range;

generate evaluations of the further updated design cases by modeling respective responses of the system for each of the further updated design cases, wherein each respective response corresponds to an operation of the system defined by each element of a given further updated design case of the further updated suite;

select, based on the evaluations of the further updated design cases, a design case; and

determine whether to update the refined range based on the selected design case; and

output, based on the updated design suite, an indication of a selected design case by outputting an indication of a selected design from the further updated design cases.

8. The computer-program product of claim 7 , wherein the instructions are operable to set the N iterations by:

obtaining an indication of a number of iterations for updating the initial design suite, or obtaining a threshold for evaluating a response of the system defined by updated options of each of the design cases of the further updated design suite; and

setting N based on the indication or threshold.

9. The computer-program product of claim 7 ,

wherein a combination of design options for all elements of the first design case are different from any combination of design options for elements of the design cases of the initial design suite; and

wherein the instructions are operable to cause a computing device to set the N iterations by:

determining an evaluation of the first design case by modeling a response of the system according to the first design case;

computing a comparison result by comparing the evaluation of the first design case to each of the evaluations of the initial design cases; and

setting N based on the comparison result.

10. The computer-program product of claim 1 , wherein the instructions are operable to cause a computing device to obtain the space-filling design by:

mapping the design space onto a matrix with rows and columns;

determining primary clusters for the design space, each containing a different set of representative design points that is mutually exclusive of representative design points in other primary clusters of the design space;

selecting the selected design points by selecting a representative design point from each primary cluster that minimizes a criterion based on:

min

D

⁢

⁢

ψ

⁡

(

D

)

=

{

1

(

n

2

)

⁢

∑

i

=

1

n

-

1

⁢

∑

j

=

i

+

1

n

⁢

1

∏

l

=

1

p

⁢

(

x

il

-

x

jl

)

2

}

1

/

p

where:

ψ(D) is the criterion;

i, j, l are integer counters;

n is an integer number of primary clusters for the design space;

p is an integer number of continuous variables for the design space; and

x ab is an entry in row a and column b of the matrix.

11. The computer-program product of claim 1 , wherein the instructions are operable to cause a computing device to:

receive restriction information indicating disallowed options within a given respective range of the range information; and

obtain the space-filling design by restricting potential design points of the design space that represent disallowed options in the design space.

12. The computer-program product of claim 1 , wherein the instructions are operable to cause a computing device to:

generate evaluations of the initial design cases by computing a coefficient of determination for each of the respective initial responses of the system; and

select the selected design case based on an optimality criterion for the computed coefficient of determination for each of the respective initial responses of the system.

13. The computer-program product of claim 1 , wherein the modeling the system comprises generating a model using a gradient boosted tree, a Gaussian process, or a neural network.

14. The computer-program product of claim 1 ,

wherein the multiple factors comprise a categorical factor defining discrete design options for the system or a partitioned factor that is a continuous factor partitioned into partitions using equivalence partitioning;

wherein the instructions are operable to cause a computing device to assign level values to each of the discrete design options or partitions; and

wherein the level values are within a respective initial range for the categorical factor or partitioned factor.

15. The computer-program product of claim 1 , wherein the instructions are operable to cause a computing device to:

display, on a display device, a graphical user interface for user entry or modification of user information in the graphical user interface, the user information comprising one or more of the factor information, the range information; and a number of design cases for a given design suite; and

receive, from a user of the graphical user interface, via one or more input devices, user input indicating the user information.

16. The computer-program product of claim 1 , wherein:

the system comprises a machine learning system; and

the multiple factors are hyperparameters that define the system by controlling behavior of the machine learning system.

17. The computer-program product of claim 1 , the instructions are operable to cause a computing device to output the indication of the selected design case by indicating each of design options for each of the multiple factors of the selected design case.

18. A computer-implemented method comprising:

receiving factor information indicating multiple factors, the multiple factors for designing a system by assigning one of respective possible options to each of the multiple factors;

receiving range information indicating initial ranges with a respective range for each of the respective possible options,

wherein at least one of the multiple factors is a continuous factor that defines continuous values within a given range of the initial ranges;

wherein the initial ranges define a design space; and

wherein the range information indicates a first range of the initial ranges for an experimental factor of the multiple factors;

obtaining a space-filling design for the design space,

wherein the space-filing design indicates selected design points in the design space,

wherein each of the selected design points represents assigned options assigned to the multiple factors, the assigned options assigned from the initial ranges, and

wherein the selected design points are separated from one another in the design space based on a criterion for separating the selected design points in the design space;

generating, based on the space-filling design, an initial design suite that provides initial design cases corresponding to one or more of the selected design points, wherein each element of a respective design case of the initial design suite is one of the assigned options represented by a selected design point of the one or more of the selected design points;

generating evaluations of the initial design cases by modeling respective initial responses of the system for each of the initial design cases, wherein each of the respective initial responses corresponds to an operation of the system defined by each element of a given respective initial design case of the initial design suite; and

outputting, based on the evaluations of the initial design cases, an indication of a selected design case by:

determining, based on the evaluations of the initial design cases, a first design case with a first design option for the experimental factor;

determining an updated range for the experimental factor, wherein the updated range comprises the first design option and the updated range is a subset of the first range;

generating an updated design suite that provides updated design cases for the system according to the updated range for the experimental factor, wherein each element of an updated design case corresponds to one of the multiple factors, and wherein one element of each of the updated design cases corresponds to the experimental factor and is selected from the updated range for the experimental factor;

generating evaluations of the updated design cases by modeling respective updated responses of the system for one or more of the updated design cases, wherein each of the respective updated responses corresponds to an operation of the system defined by each element of a given updated design case of the updated design suite; and

outputting, based on the evaluations of the one or more of the updated design cases, the indication of the selected design case.

19. The computer-implemented method of claim 18 , wherein the determining the first design case comprises generating a generated design case different than all of the initial design cases of the initial design suite based on a correlation of individual design options of the assigned options for factors of the multiple factors and the evaluations of the initial design cases.

20. The computer-implemented method of claim 18 , wherein the determining the updated range for the experimental factor comprises:

determining a respective updated range for each factor of the multiple factors, wherein a given respective updated range is a subset of an initial range of the initial ranges corresponding to a same factor of the multiple factors; and

generating an updated design suite that provides design cases for the system according to the respective updated range for each factor of the multiple factors.

21. The computer-implemented method of claim 18 ,

wherein the design space is an initial design space and the space-filling design is an initial space-filling design for the initial design space; and

wherein the generating the updated design suite comprises:

determining an updated design space defined by the updated range for the experimental factor;

obtaining an updated space-filling design based on a criterion for separating selected design points in the updated design space; and

generating the updated design suite based on the updated space-filling design.

22. The computer-implemented method of claim 18 ,

wherein the receiving the range information comprises obtaining a predefined percentage for each respective initial range; and

wherein the determining the updated range for the experimental factor by using the predefined percentage to determine the subset of the first range around the first design option.

23. The computer-implemented method of claim 18 ,

wherein the method further comprises for N iterations, where N is a numeral greater than 0:

setting a refined range that is a subset of a range of a previous updated design suite;

generating a further updated design suite that provides design cases for the system according to the refined range;

generating evaluations of the further updated design cases by modeling respective responses of the system for each of the further updated design cases, wherein each respective response corresponds to an operation of the system defined by each element of a given further updated design case of the further updated suite;

selecting, based on the evaluations of the further updated design cases, a design case; and

determining whether to update the refined range based on the selected design case; and

wherein the outputting the indication of a selected design case comprises outputting an indication of a selected design from the further updated design cases.

24. The computer-implemented method of claim 23 , wherein the method further comprises setting the N iterations by:

obtaining an indication of a number of iterations for updating the initial design suite, or obtaining a threshold for evaluating a response of the system defined by updated options of each of the design cases of the further updated design suite; and

setting N based on the indication or threshold.

25. The computer-implemented method of claim 18 , wherein the

computer-implemented method further comprises receiving restriction information indicating disallowed options within a given respective range of the range information; and

wherein the obtaining the space-filling design comprises restricting potential design points of the design space that represent disallowed options in the design space.

26. The computer-implemented method of claim 18 ,

wherein the generating the evaluations of the initial design cases comprises computing a coefficient of determination for each of the respective initial responses of the system; and

wherein the outputting the indication of the selected design cases comprises selecting the selected design case based on an optimality criterion for the computed coefficient of determination for each of the respective initial responses of the system.

27. The computer-implemented method of claim 18 , wherein the method further comprises:

displaying, on a display device, a graphical user interface for user entry or modification of user information in the graphical user interface, the user information comprising one or more of the factor information, the range information; and a number of design cases for a given design suite; and

receiving, from a user of the graphical user interface, via one or more input devices, user input indicating the user information.

28. A computing device comprising processor and memory, the memory containing instructions executable by the processor wherein the computing device is configured to:

receive factor information indicating multiple factors, the multiple factors for designing a system by assigning one of respective possible options to each of the multiple factors;

receive range information indicating initial ranges with a respective range for each of the respective possible options,

wherein at least one of the multiple factors is a continuous factor that defines continuous values within a given range of the initial ranges;

wherein the initial ranges define a design space; and

wherein the range information indicates a first range of the initial ranges for an experimental factor of the multiple factors;

obtain a space-filling design for the design space,

wherein the space-filing design indicates selected design points in the design space,

wherein each of the selected design points represents assigned options assigned to the multiple factors, the assigned options assigned from the initial ranges, and

wherein the selected design points are separated from one another in the design space based on a criterion for separating the selected design points in the design space;

generate, based on the space-filling design, an initial design suite that provides initial design cases corresponding to one or more of the selected design points, wherein each element of a respective design case of the initial design suite is one of the assigned options represented by a selected design point of the one or more of the selected design points;

generate evaluations of the initial design cases by modeling respective initial responses of the system for each of the initial design cases, wherein each of the respective initial responses corresponds to an operation of the system defined by each element of a given respective initial design case of the initial design suite; and

output, based on the evaluations of the initial design cases, an indication of a selected design case by:

determining, based on the evaluations of the initial design cases, a first design case with a first design option for the experimental factor;

determining an updated range for the experimental factor, wherein the updated range comprises the first design option and the updated range is a subset of the first range;

generating an updated design suite that provides updated design cases for the system according to the updated range for the experimental factor, wherein each element of an updated design case corresponds to one of the multiple factors, and wherein one element of each of the updated design cases corresponds to the experimental factor and is selected from the updated range for the experimental factor;

generating evaluations of the updated design cases by modeling respective updated responses of the system for one or more of the updated design cases, wherein each of the respective updated responses corresponds to an operation of the system defined by each element of a given updated design case of the updated design suite; and

outputting, based on the evaluations of the one or more of the updated design cases, the indication of the selected design case.

29. The computer-implemented method of claim 18 , wherein the outputting the indication of the selected design case comprises:

determining, based on the evaluations of the initial design cases, the first design case with the first design option for the experimental factor of the multiple factors by selecting a subset of the initial design cases;

determining a respective updated range for each factor of each design case of the subset of the initial design cases, wherein a given respective updated range corresponding a same factor in each design case is different;

generating respective updated design suites for each design case of the subset;

evaluating respective responses of the system defined by updated options of each of the design cases of the respective updated design suites; and

selecting, based on the evaluation of the updated design suite, a design case from or generated based on one of the respective updated design suites.

30. The computer-implemented method of claim 18 ,

wherein the multiple factors comprise a categorical factor defining discrete design options for the system or a partitioned factor that is a continuous factor partitioned into partitions using equivalence partitioning;

wherein the computer-implemented method comprises assigning level values to each of the discrete design options or partitions; and

wherein the level values are within a respective initial range for the categorical factor or partitioned factor.