Multi-armed bandit improvement
In one embodiment, a system includes a processor to control a resource according to policies selected by a multi-armed bandit machine learning agent in exploration phases and in exploitation phases, and execute the multi-armed bandit machine learning agent to select from the policies to control the resource in the exploration phases according to probabilities to explore corresponding one of the policies, wherein the probabilities include different probabilities, perform measurements on the system during execution of the multi-armed bandit machine learning agent, and execute the multi-armed bandit machine learning agent to select from the policies to maximize potential rewards from controlling the resource in exploitation phases based on the performed measurements, and a memory to store data used by the processor.
1 . A system, comprising:
a processor to:
control a resource according to policies selected by a multi-armed bandit machine learning agent in exploration phases and in exploitation phases; and
execute the multi-armed bandit machine learning agent to select from the policies to control the resource in the exploration phases according to probabilities to explore corresponding one of the policies, wherein the probabilities include different probabilities;
perform measurements on the system during execution of the multi-armed bandit machine learning agent; and
execute the multi-armed bandit machine learning agent to select from the policies to maximize potential rewards from controlling the resource in exploitation phases based on the performed measurements; and
a memory to store data used by the processor.
2 . The system according to claim 1 , wherein:
the resource includes prefetcher engines to predict next memory access addresses of the memory, each of the prefetcher engines being configurable to provide a level of aggressiveness;
the policies correspond to different configurations of the prefetcher engines; and
the measurements include executed instructions per cycle.
3 . The system according to claim 1 , further comprising processing circuitry to compute the probabilities to explore with the corresponding policies based on prior knowledge gained from controlling the resource according to the policies while executing benchmark applications.
4 . The system according to claim 3 , wherein the processing circuitry is comprised in the processor.
5 . The system according to claim 3 , wherein the processing circuitry is to compute the probability to explore with a given policy of the policies based on prior knowledge of an extent of success and failure of the given policy while executing the benchmark applications.
6 . The system according to claim 5 , wherein the processing circuitry is to compute the probability to explore with the given policy such that the probability to explore with the given policy optimizes a loss function representing a system parameter to be optimized.
7 . The system according to claim 6 , wherein the loss function includes a parameter which compares: (a) a first value of a quality metric when the given policy is applied during execution of a given benchmark application; with (b) a second value of the quality metric of the best policy, which is one of the policies providing a highest value of the quality metric among the policies applied during execution of the given benchmark application.
8 . The system according to claim 7 , wherein the loss function includes any one or more of the following parameters:
an empiric parameter which measures the average or distribution of a number of times ones of the policies have to be explored in order to identify their relative quality compared with other ones of the policies for the given benchmark application;
a weight of the given benchmark application with respect to other ones of the benchmark applications; and
a number of checkpoints in the given benchmark application.
9 . The system according to claim 1 , wherein the processing circuitry is to compute the different probabilities to explore with the corresponding policies such that the probabilities to explore optimize a loss function representing a system parameter to be optimized.
10 . The system according to claim 9 , wherein the loss function includes:
a first term that represents the loss of benefit due to the time taken during exploration for the multi-armed bandit machine learning agent to find ones of the policies that are better than other ones of the policies; and
a second term that represents the loss of benefit due to exploration with sub-optimal policies.
11 . The system according to claim 10 , wherein the processing circuitry is to compute the different probabilities to explore with the corresponding policies based on an expression formed by comparing a derivative of the loss function to zero.
12 . A method, comprising:
controlling a resource according to policies selected by a multi-armed bandit machine learning agent in exploration phases and in exploitation phases; and
executing the multi-armed bandit machine learning agent to select from the policies to control the resource in the exploration phases according to probabilities to explore corresponding one of the policies, wherein the probabilities include different probabilities;
performing measurements on the system during execution of the multi-armed bandit machine learning agent; and
executing the multi-armed bandit machine learning agent to select from the policies to maximize potential rewards from controlling the resource in exploitation phases based on the performed measurements; and
storing data used by the processor.
13 . The method according to claim 12 , wherein:
the resource includes prefetcher engines;
each of the prefetcher engines is configurable to provide a level of aggressiveness;
the policies correspond to different configurations of the prefetcher engines; and
the measurements include executed instructions per cycle.
14 . The method according to claim 12 , further comprising computing the probabilities to explore with the corresponding policies based on prior knowledge gained from controlling the resource according to the policies while executing benchmark applications.
15 . The method according to claim 14 , wherein the computing includes computing the probability to explore with a given policy of the policies based on prior knowledge of an extent of success and failure of the given policy while executing the benchmark applications.
16 . The method according to claim 15 , wherein the computing includes computing the probability to explore with the given policy such that the probability to explore with the given policy optimizes a loss function representing a system parameter to be optimized.
17 . The method according to claim 16 , wherein the loss function includes a parameter which compares: (a) a first value of a quality metric when the given policy is applied during execution of a given benchmark application; with (b) a second value of the quality metric of the best policy, which is one of the policies providing a highest value of the quality metric among the policies applied during execution of the given benchmark application.
18 . The method according to claim 17 , wherein the loss function includes any one or more of the following parameters:
an empiric parameter which measures the average or distribution of a number of times ones of the policies have to be explored in order to identify their relative quality compared with other ones of the policies for the given benchmark application;
a weight of the given benchmark application with respect to other ones of the benchmark applications; and
a number of checkpoints in the given benchmark application.
19 . The method according to claim 12 , further comprising computing the different probabilities to explore with the corresponding policies such that the probabilities to explore optimize a loss function representing a system parameter to be optimized.
20 . The method according to claim 19 , wherein the loss function includes:
a first term that represents the loss of benefit due to the time taken during exploration for the multi-armed bandit machine learning agent to find ones of the policies that are better than other ones of the policies; and
a second term that represents the loss of benefit due to exploration with sub-optimal policies.
21 . The method according to claim 20 , wherein the computing includes computing the different probabilities to explore with the corresponding policies based on an expression formed by comparing a derivative of the loss function to zero.