Data tiering for edge computers, hubs and central systems
Systems and methods for tiering data in distributed data networks. A global model is developed based on federated learning where edge servers are able to train a model. The learning from the edge servers are collectively applied to the global model. This process can be repeated until the global model is ready for deployment. The global model allows data to be tiered. This may include pushing data from a datacenter to edge servers or cleaning edge servers of data that does not satisfy the global model. The model can be retrained and can be used to proactively push new content out to the edge servers.
1. A method for tiering data in a distributed data network, the method comprising:
(a) pushing a global model from a central system to each of selected edge servers in the distributed data network, wherein the selected edge servers are a subset of edge servers operating in the distributed data network;
(b) training the global model at each of the selected edge servers, wherein each of the selected edge servers uses their own local data to train the global model;
(c) receiving updates to the global model from the selected edge servers at the global model at the central system;
(d) updating the global model at the central system based on the updates;
(e) identifying data, by the global model, to distribute to the edge servers;
(f) distributing the data from a datacenter to the edge servers, the data including first data and second data, wherein the first data is at least partially different from the second data, wherein the first data is delivered to a first edge server included in the edge servers and the second data is delivered to a second edge server included in the edge servers;
(g) retraining the global model when a request miss rate exceeds a threshold rate; and
(h) avoiding cache misses by pushing new data received at the datacenter to the edge using the global model.
2. The method of claim 1 , wherein updating the global model includes averaging the updates and applying the average update to the global model.
3. The method of claim 1 , further comprising repeating steps (a)-(d) until the updates are less than a threshold.
4. The method of claim 1 , further comprising distributing the global model to the edge servers.
5. The method of claim 4 , further comprising validating the global model by each of the edge servers not included in the selected edge servers using corresponding local data.
6. The method of claim 4 , wherein the updated global model allows each of the edge servers to benefit from learning of the selected edge servers.
7. The method of claim 1 , wherein distributing the data includes tiering data in the distributed data network.
8. The method of claim 7 , further comprising applying the global model to data at a datacenter and pushing data to at least some of the edge servers using the global model.
9. The method of claim 7 , further comprising applying the global model to data at the edge servers and cleaning data from the edge servers that does not meet a threshold.
10. The method of claim 1 , further comprising retraining the global model when a request miss rate exceeds a threshold rate.
11. The method of claim 1 , further comprising avoiding the cache misses by pushing the new data received at the datacenter to the edge servers before receiving requests for the new data using the global model.
12. The method of claim 1 , further comprising initiating the global model, wherein the global model is configured to predict data that end users are expected to access in a stipulated time.
13. A non-transitory computer readable medium comprising computer executable instructions that, when executed by a processor and memory, perform a method for tiering data in a distributed data network, the method comprising:
initiating, at a datacenter, a global model for tiering data in a distributed data network that includes the datacenter, hub servers, and edge servers;
repeating until updates to a global model are less than a threshold:
pushing the global model from a central system associated with the datacenter to each of selected edge servers in the distributed data network, wherein the selected edge servers are a subset of edge servers operating in the distributed data network and wherein the edge servers are closest to end user devices;
training the global model at each of the selected edge servers, wherein each of the selected edge servers uses their own local data to train the global model;
sending updates to the global model at the central system from the selected edge servers to the central system; and
updating the global model by the central system based on the updates;
identifying data, by the global model, to distribute to the edge servers;
distributing the data from a datacenter to the edge servers, the data including first data and second data, wherein the first data is at least partially different from the second data, wherein the first data is delivered to a first edge server included in the edge servers and the second data is delivered to a second edge server included in the edge servers;
retraining the global model when a request miss rate exceeds a threshold rate; and
avoiding cache misses by pushing new data received at the datacenter to the edge servers using the global model.
14. The non-transitory computer readable medium of claim 13 , wherein updating the global model includes averaging the updates received from the selected edge servers and applying the average update to the global model at the central system.
15. The non-transitory computer readable medium of claim 13 , further comprising distributing the global model to the edge servers once the updates are less than the threshold.
16. The non-transitory computer readable medium of claim 15 , further comprising validating the global model by each of the edge servers not included in the selected edge servers using corresponding local data.
17. The non-transitory computer readable medium of claim 13 , wherein the updated global model allows each of the edge servers to benefit from learning of the first edge servers.
18. The non-transitory computer readable medium of claim 13 , further comprising tiering data in the distributed data network.
19. The non-transitory computer readable medium of claim 13 , further comprising:
applying the global model to data at a datacenter and pushing data that meets a threshold to at least some of the edge servers; and
applying the global model to data at the edge servers and cleaning data from the edge servers that does not meet a threshold.
20. The method of claim 1 , further comprising:
avoiding the cache misses by pushing the new data received at the datacenter to the edge servers in anticipation of requests for the new data using the global model.