Methods and apparatus for machine learning engine optimization
Methods, apparatus, systems and articles of manufacture are disclosed for machine learning engine optimization. An example apparatus includes a selection metric analyzer to compute a first selection metric based on a first set of ordered output values from a first candidate machine learning engine and a set of reference data values; compute a second selection metric based on a second set of ordered output values from a second candidate machine learning engine and the set of reference data values; and a machine learning engine replacer to determine whether to replace an active machine learning engine with at least one of the first candidate machine learning engine or the second candidate machine learning engine based on the first selection metric and the second selection metric.
1. An apparatus comprising:
a selection metric analyzer to:
compute a first selection metric based on a first set of ordered output values from a first candidate machine learning engine and a set of reference data values, the first selection metric computed based on (i) a difference between a first number of the reference data values in the set of reference data values and a second number of the reference data values included in beginning positions of the first set of ordered output values from the first candidate machine learning engine, and (ii) division of the difference by the first number of the reference data values in the set of reference data values; and
compute a second selection metric based on a second set of ordered output values from a second candidate machine learning engine and the set of reference data values; and
a machine learning engine replacer to determine whether to replace an active machine learning engine with at least one of the first candidate machine learning engine or the second candidate machine learning engine based on the first selection metric and the second selection metric.
2. The apparatus of claim 1 , wherein the first set of ordered output values are ranked by order of importance based on a configuration of the first candidate machine learning engine, and the second set of ordered output values are ranked by order of importance based on a configuration of the second candidate machine learning engine.
3. The apparatus of claim 2 , wherein the reference data values in the set of reference data values are unordered reference data values that are utilized to determine consecutive highest rankings in the first set of ordered output values from the first candidate machine learning engine and the second set of ordered output values from second candidate machine learning engine.
4. The apparatus of claim 3 , wherein the division of the difference by the first number of the reference data values corresponds to a first evaluation metric.
5. The apparatus of claim 4 , wherein the difference is a first difference, and the selection metric analyzer is to determine a second evaluation metric based on:
a second difference between 1) a number corresponding to how many of the reference data values from the set of reference data values are present in the first set of ordered output values and 2) the first number of the reference data values from the set of reference data values;
addition of a penalty to the second difference for each of the reference data values from the set of reference data values that is not present in the first set of ordered output values; and
division of the second difference by an addition of 1) a number of values in the first set of ordered output values and 2) the first number of the reference data values from the set of reference data values.
6. The apparatus of claim 5 , wherein the penalty has a value of one-fourth.
7. The apparatus of claim 5 , wherein to determine the first selection metric further, the selection metric analyzer is to:
add the first evaluation metric and the second evaluation metric to determine a result; and
divide the result by two to determine the first selection metric.
8. The apparatus of claim 1 , wherein to determine whether to replace the active machine learning engine with the at least one of the first candidate machine learning engine or the second candidate machine learning engine, the selection metric analyzer is to:
compare the first selection metric to the second selection metric to identify a lowest selection metric;
compare the lowest selection metric to a threshold to determine if the at least one of the first candidate machine learning engine or the second candidate machine learning engine should replace the active machine learning engine; and
the machine learning engine replacer is to replace the active machine learning engine with the at least one of the first candidate machine learning engine or the second candidate machine learning engine when the lowest selection metric satisfies the threshold.
9. A non-transitory computer readable storage medium comprising instructions which, when executed, cause one or more processors to at least:
compute a first selection metric based on a first set of ordered output values from a first candidate machine learning engine and a set of reference data values, the first selection metric computed based on (i) a difference between a first number of the reference data values in the set of reference data values and a second number of the reference data values included in beginning positions of the first set of ordered output values from the first candidate machine learning engine, and (ii) division of the difference by the first number of the reference data values in the set of reference data values;
compute a second selection metric based on a second set of ordered output values from a second candidate machine learning engine and the set of reference data values; and
determine whether to replace an active machine learning engine with at least one of the first candidate machine learning engine or the second candidate machine learning engine based on the first selection metric and the second selection metric.
10. The computer readable storage medium of claim 9 , wherein the first set of ordered output values are ranked by order of importance based on a configuration of the first candidate machine learning engine, and the second set of ordered output values are ranked by order of importance based on a configuration of the second candidate machine learning engine.
11. The computer readable storage medium of claim 10 , wherein the reference data values in the set of reference data values are unordered reference data values that are utilized to determine consecutive highest rankings in the first set of ordered output values from the first candidate machine learning engine and the second set of ordered output values from second candidate machine learning engine.
12. The computer readable storage medium of claim 11 , wherein the division of the difference by the first number of the reference data values corresponds to a first evaluation metric.
13. The computer readable storage medium of claim 12 , wherein the difference is a first difference and the instructions, when executed, cause the one or more processors to determine a second evaluation metric based on:
a second difference between 1) a number corresponding to how many of the unordered reference data values from the set of reference data values are present in the first set of ordered output values and 2) the first number of the reference data values from the set of reference data values;
addition of a penalty to the second difference for each of the reference data values from the set of reference data values that is not present in the first set of ordered output values; and
division of the second difference by an addition of 1) a number of values in the first set of ordered output values and 2) the first number of the reference data values from the set of reference data values.
14. The computer readable storage medium of claim 13 , wherein the instructions, when executed, cause the one or more processors to:
add the first evaluation metric and the second evaluation metric to determine a result; and
divide the result by two to determine the first selection metric.
15. A method comprising:
computing, by executing an instruction with a processor, a first selection metric based on a first set of ordered output values from a first candidate machine learning engine and a set of reference data values, the first selection metric computed based on (i) a difference between a first number of the reference data values in the set of reference data values and a second number of the reference data values included in beginning positions of the first set of ordered output values from the first candidate machine learning engine, and (ii) division of the difference by the first number of the reference data values in the set of reference data values;
computing, by executing an instruction with the processor, a second selection metric based on a second set of ordered output values from a second candidate machine learning engine and the set of reference data values; and
determining, by executing an instruction with the processor, whether to replace an active machine learning engine with at least one of the first candidate machine learning engine or the second candidate machine learning engine based on the first selection metric and the second selection metric.
16. The method of claim 15 , wherein the first set of ordered output values are ranked by order of importance based on a configuration of the first candidate machine learning engine, and the second set of ordered output values are ranked by order of importance based on a configuration of the second candidate machine learning engine.
17. The method of claim 16 , wherein the reference data values in the set of reference data values are unordered reference data values that are utilized to determine consecutive highest rankings in the first set of ordered output values from the first candidate machine learning engine and the second set of ordered output values from second candidate machine learning engine.
18. The method of claim 17 , wherein the division of the difference by the first number of the reference data values corresponds to a first evaluation metric.
19. The method of claim 17 , wherein the difference is a first difference, and further including determining a second evaluation metric based on:
a second difference between 1) a number corresponding to how many of the reference data values from the set of reference data values are present in the first set of ordered output values and 2) the first number of the reference data values from the set of reference data values;
addition of a penalty to the second difference for each of the reference data values from the set of reference data values that is not present in the first set of ordered output values; and
division of the second difference by an addition of 1) a number of values in the first set of ordered output values and 2) the first number of the unordered reference data values from the set of reference data values.
20. The method of claim 19 , wherein to determine the first selection metric further includes:
adding the first evaluation metric and the second evaluation metric to determine a result; and
dividing the result by two to determine the first selection metric.