Managing data stored in a cache using a reinforcement learning agent
Managing data stored in a cache using a reinforcement learning agent may include: determining a set of current state observations with respect to a cache, wherein the set of current state observations is determined based on historical cache accesses to the cache; inputting the set of current state observations into an actor network of a reinforcement learning (RL) agent to obtain an action output by the actor network, wherein the RL agent is configured to manage data stored at the cache; inputting the set of current state observations and the action into a critic network of the RL agent to obtain a score corresponding to the action from the critic network; causing the RL agent to perform the action with respect to managing the data stored at the cache; using the score to update the actor network; and using a reward corresponding to the action to update the critic network.
1 . A system, comprising:
one or more processors; and
one or more memories coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
receiving a cache access requesting a current data item identified by a data item identifier from a cache;
determining, based at least in part on a plurality of historical cache accesses to the cache, a set of current states with respect to the cache;
inputting the set of current states into an actor network of a reinforcement learning (RL) agent to obtain an action output by the actor network, wherein the RL agent is configured to manage data stored at the cache, and the action specifies an eviction time for the current data item;
inputting the set of current states and the action into a critic network of the RL agent to obtain a score corresponding to the action from the critic network, for the RL agent to perform the action with respect to managing the data stored at the cache; and
servicing the cache access according to the action by performing a cache-management operation that includes evicting the current data item from the cache when a system time reaches the eviction time and storing a replacement data item retrieved from a second-level auxiliary storage at a physical address in the cache determined by the action, thereby transforming a physical state of the cache;
determining a reward corresponding to the action based at least in part on whether the cache access resulted in a cache hit or a cache miss and whether an eviction occurred in connection with the cache access;
using the score corresponding to the action to update the actor network; and
using the reward corresponding to the action to update the critic network,
wherein the cache-management operation performed according to the action reduces accesses to the second-level auxiliary storage for a workload of cache accesses relative to a replacement policy that does not use the actor network and the critic network.
2 . The system of claim 1 , wherein the set of current states comprises values corresponding to respective ones of a set of state definitions, wherein each of the set of state definitions comprises an attribute associated with cache accesses within a sliding window of recent cache accesses at the cache.
3 . The system of claim 1 , wherein the set of current states is determined in response to an occurrence of a triggering event with respect to the cache.
4 . The system of claim 1 , wherein the action output by the actor network comprises outputting an instruction of when a data item is to be evicted from the cache.
5 . The system of claim 1 , wherein the action output by the actor network comprises outputting an instruction of a location in the cache to store a newly added data item.
6 . The system of claim 1 , wherein using the score corresponding to the action to update the actor network comprises performing backpropagation at the actor network based at least in part on the score.
7 . The system of claim 1 , wherein the reward corresponding to the action comprises a first reward value for a cache hit associated with the action performed at the cache or a second reward value for a cache miss associated with the action performed at the cache, wherein the first reward value is different from the second reward value.
8 . The system of claim 1 , wherein the operations further comprise:
determining a set of next states with respect to the cache;
generating a sample comprising the set of current states, the set of next states, the action, and the reward; and
storing the sample in a storage that is configured to store a plurality of samples.
9 . The system of claim 8 , wherein the operations further comprise:
selecting a subset of samples from the plurality of samples; and
using the subset of samples to train the RL agent.
10 . The system of claim 1 , wherein the RL agent comprises a deep deterministic policy gradient (DDPG) RL agent.
11 . A method, comprising:
receiving, by one or more processors, a cache access requesting a current data item identified by a data item identifier from a cache;
determining, by the one or more processors based at least in part on a plurality of historical cache accesses to the cache, a set of current states with respect to the cache;
inputting, by the one or more processors, the set of current states into an actor network of a reinforcement learning (RL) agent to obtain an action output by the actor network, wherein the RL agent is configured to manage data stored at the cache, and the action specifies an eviction time for the current data item;
inputting, by the one or more processors, the set of current states and the action into a critic network of the RL agent to obtain a score corresponding to the action from the critic network, for the RL agent to perform the action with respect to managing the data stored at the cache; and
servicing the cache access according to the action by performing a cache-management operation that includes evicting the current data item from the cache when a system time reaches the eviction time and storing a replacement data item retrieved from a second-level auxiliary storage at a physical address in the cache determined by the action, thereby transforming a physical state of the cache;
determining a reward corresponding to the action based at least in part on whether the cache access resulted in a cache hit or a cache miss and whether an eviction occurred in connection with the cache access;
using, by the one or more processors, the score corresponding to the action to update the actor network; and
using, by the one or more processors, the reward corresponding to the action to update the critic network.
12 . The method of claim 11 , wherein the set of current states comprises values corresponding to respective ones of a set of state definitions, wherein each of the set of state definitions comprises an attribute associated with cache accesses within a sliding window of recent cache accesses at the cache.
13 . The method of claim 11 , wherein the set of current states is determined in response to an occurrence of a triggering event with respect to the cache.
14 . The method of claim 11 , wherein the action output by the actor network comprises outputting an instruction of when a data item is to be evicted from the cache.
15 . The method of claim 11 , wherein the action output by the actor network comprises outputting an instruction of a location in the cache to store a newly added data item.
16 . The method of claim 11 , wherein using the score corresponding to the action to update the actor network comprises performing backpropagation at the actor network based at least in part on the score.
17 . The method of claim 11 , wherein the reward corresponding to the action comprises a first reward value for a cache hit associated with the action performed at the cache or a second reward value for a cache miss associated with the action performed at the cache, wherein the first reward value is different from the second reward value.
18 . A non-transitory computer readable storage medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving a cache access requesting a current data item identified by a data item identifier from a cache;
determining, based at least in part on a plurality of historical cache accesses to the cache, a set of current states with respect to the cache;
inputting the set of current states into an actor network of a reinforcement learning (RL) agent to obtain an action output by the actor network, wherein the RL agent is configured to manage data stored at the cache, and the action specifies an eviction time for the current data item;
inputting the set of current states and the action into a critic network of the RL agent to obtain a score corresponding to the action from the critic network, for the RL agent to perform the action with respect to managing the data stored at the cache; and
servicing the cache access according to the action by performing a cache-management operation that includes evicting the current data item from the cache when a system time reaches the eviction time and storing a replacement data item retrieved from a second-level auxiliary storage at a physical address in the cache determined by the action, thereby transforming a physical state of the cache;
determining a reward corresponding to the action based at least in part on whether the cache access resulted in a cache hit or a cache miss and whether an eviction occurred in connection with the cache access;
using the score corresponding to the action to update the actor network; and
using the reward corresponding to the action to update the critic network.