Distributed cache or replay service for massively scalable distributed reinforcement learning
A computing system for performing distributed large scale reinforcement learning with improved efficiency can include a plurality of actor devices, wherein each actor device locally stores a local version of a machine-learned model, wherein each actor device is configured to implement the local version of the machine-learned model at the actor device to determine an action to take in an environment to generate an experience, a server computing system configured to perform one or more learning algorithms to learn an updated version of the machine-learned model based on the experiences generated by the plurality of actor devices, and a hierarchical and distributed data caching system including a plurality of layers of data caches that propagate data descriptive of the updated version of the machine-learned model from the server computing system to the plurality of actor devices to enable each actor device to update its respective local version of the model.
1 . A computing system for performing distributed large scale reinforcement learning with improved efficiency, the computing system comprising:
a plurality of actor devices, wherein each actor device locally stores a local version of a machine-learned model, and wherein each actor device is configured to implement the local version of the machine-learned model at the actor device to determine an action to take in an environment to generate an experience;
a server computing system configured to perform one or more learning algorithms to learn an updated version of the machine-learned model based on the experiences generated by the plurality of actor devices; and
a hierarchical and distributed data caching system comprising a plurality of layers of data caches that propagate data descriptive of the updated version of the machine-learned model from the server computing system to the plurality of actor devices to enable each actor device to update its respective local version of the machine-learned model, wherein the plurality of layers of data caches comprise at least a first layer that receives the data descriptive of the updated version of the machine-learned model from the server computing system and a final layer that provides the data descriptive of the updated version of the machine-learned model to the plurality of actor devices.
2 . The computing system of claim 1 , wherein the plurality of layers of data caches further comprise one or more intermediate layers hierarchically positioned between the first layer and the final layer.
3 . The computing system of claim 2 , wherein one or more data caches included in each of the one or more intermediate layers communicate the data descriptive of the updated version of the machine-learned model to one or more data caches in a subsequent layer using a push messaging mechanism.
4 . The computing system of claim 1 , wherein each of the plurality of layers of data caches from the first layer to the final layer comprises an increasing number of data caches.
5 . The computing system of claim 1 , wherein the first layer comprises a single data cache.
6 . The computing system of claim 1 , wherein, for each of the plurality of layers of data caches except the final layer, each data cache propagates the data descriptive of the updated version of the machine-learned model to two or more data caches in a next sequential layer.
7 . The computing system of claim 1 , wherein:
the plurality of actor devices are organized into a plurality of cells;
the respective actor devices in each cell are managed as a single unit; and
for each of the plurality of cells, the final layer comprises one or more cell-specific data caches that service only the respective actor devices in the cell.
8 . The computing system of claim 1 , wherein the server computing system is organized into a server cell, and wherein the first layer is organized into the server cell.
9 . The computing system of claim 1 , wherein one or more data caches included in the first layer communicate the data descriptive of the updated version of the machine-learned model to one or more data caches in a subsequent layer using a push messaging mechanism.
10 . The computing system of claim 1 , wherein the plurality of actor devices obtain the data descriptive of the updated version of the machine- learned model from one or more data caches included in the final layer using a pull messaging mechanism.
11 . The computing system of claim 1 , wherein the data caches communicate the data descriptive of the updated version of the machine-learned model using remote procedure calls.
12 . The computing system of claim 1 , wherein the data descriptive of the updated version of the machine-learned model comprises a current model version index.
13 . The computing system of claim 12 , wherein the data caches communicate data descriptive of the updated version of the machine-learned model to an actor device of the plurality of actor devices only if the data descriptive of the updated version of the machine-learned model comprises a new model version index than a model version index at the actor device.
14 . The computing system of claim 1 , wherein the data caches comprise a backup system.
15 . The computing system of claim 1 , wherein the plurality of actor devices are configured to periodically request updated data from the data caching system.
16 . The computing system of claim 1 , wherein the plurality of actor devices are configured to request updated data from the data caching system in response to a notification from the server computing system.
17 . The computing system of claim 1 , wherein the data caches comprise a key and value data structure.
18 . A computing system for performing distributed large scale reinforcement learning with improved efficiency, the computing system comprising:
a plurality of actor devices, wherein each actor device locally stores a local version of a machine-learned model, and wherein each actor device is configured to implement the local version of the machine-learned model at the actor device to determine an action to take in an environment to generate an experience;
a server computing system configured to perform one or more learning algorithms to learn an updated version of the machine-learned model based on the experiences generated by the plurality of actor devices; and
a distributed replay memory including a plurality of replay buffers, a first replay buffer of the plurality of replay buffers configured to store first experiences generated by a first plurality of actor devices, a second replay buffer of the plurality of replay buffers configured to store second experiences generated by a second plurality of actor devices, wherein the plurality of replay buffers provide the first and second experiences to the server computing system for use in learning the updated version of the machine-learned model, the server computing system comprising:
a first compute engine configured to receive the first experiences cached by the first replay buffer; and
a second compute engine configured to receive the second experiences cached by the second replay buffer.
19 . The computing system of claim 18 , wherein the first replay buffer is connected to the first compute engine and the second replay buffer is connected to the second compute engine.
20 . The computing system of claim 18 , wherein the plurality of compute engines comprise a plurality of compute cores in a processing unit.
21 . The computing system of claim 18 , wherein the processing unit comprises a tensor processing unit.
22 . A computing system for performing distributed large scale reinforcement learning with improved efficiency, the computing system comprising:
a server computing system configured to perform one or more learning algorithms to learn an updated version of a machine-learned model based on a plurality of experiences generated by a plurality of actor devices, wherein each actor device locally stores a local version of the machine-learned model, and wherein each actor device is configured to implement the local version of the machine-learned model at the actor device to determine an action to take in an environment to generate one of the plurality of experiences; and
a hierarchical and distributed data caching system comprising a plurality of layers of data caches that propagate data descriptive of the updated version of the machine-learned model from the server computing system to the plurality of actor devices to enable each actor device to update its respective local version of the machine-learned model, wherein the plurality of layers of data caches comprise at least a first layer that receives the data descriptive of the updated version of the machine-learned model from the server computing system and a final layer that provides the data descriptive of the updated version of the machine-learned model to the plurality of actor devices.
23 . A computing system for performing distributed large scale reinforcement learning with improved efficiency, the computing system comprising:
a server computing system configured to perform one or more learning algorithms to learn an updated version of a machine-learned model based on a plurality of experiences generated by the plurality of actor devices, wherein each actor device locally stores a local version of the machine-learned model, and wherein each actor device is configured to implement the local version of the machine-learned model at the actor device to determine an action to take in an environment to generate one of the plurality of experiences; and
a distributed replay memory including a plurality of replay buffers, a first replay buffer of the plurality of replay buffers configured to store first experiences generated by a first plurality of actor devices, a second replay buffer of the plurality of replay buffers configured to store second experiences generated by a second plurality of actor devices, wherein the plurality of replay buffers provide the first and second experiences to the server computing system for use in learning the updated version of the machine-learned model, the server computing system comprising:
a first compute engine configured to receive the first experiences cached by the first replay buffer; and
a second compute engine configured to receive the second experiences cached by the second replay buffer.