Data object delivery for distributed cluster computing
Methods and systems for delivering data for cluster computing are described herein. A worker device may receive a dataset and store the dataset in a local storage media. This may prevent the need for the dataset to be sent over a network each time the applications are used to perform a task. Each application may be able to access the dataset in the local storage area. This may prevent the need to copy the dataset to memory associated with each application. A worker device may store a dataset, for example, if it determines that the frequency of updates to the dataset satisfy a threshold. The worker device may receive updates to the dataset via a messaging system and may store the updated data in the local storage media.
1. A method comprising:
subscribing, by a worker device of a plurality of worker devices, to a dataset;
receiving, from a master device of the plurality of worker devices the dataset;
determining, based on metadata associated with the dataset, that a quantity of updates to the dataset within a time period satisfies a data update frequency threshold;
based on determining that the quantity of updates within the time period satisfies the data update frequency threshold, storing the dataset in a local storage media; and
executing, by the worker device, a plurality of applications to use the dataset, wherein each application of the plurality of applications is configured to share, with each other application of the plurality of applications, the dataset from the local storage media.
2. The method of claim 1 , further comprising:
ceasing, based on a determination that a no-op message has not been received within a threshold time period, execution of the plurality of applications.
3. The method of claim 2 , wherein the no-op message indicates that there have been no updates to the dataset.
4. The method of claim 1 , wherein the storing the dataset comprises:
determining, based on metadata associated with the dataset, that a size of the dataset satisfies a threshold; and
based on the determining that the size of the dataset satisfies the threshold, storing the dataset in the local storage media.
5. The method of claim 1 , further comprising:
receiving, during execution of the plurality of applications, a message comprising an update to the dataset; and
updating the dataset in the local storage media.
6. The method of claim 1 , wherein the executing the plurality of applications comprises joining the dataset from the local storage media with a second dataset received from the master device.
7. The method of claim 1 , wherein each of the plurality of applications comprise:
a virtual processor.
8. The method of claim 1 , wherein the receiving the dataset comprises:
receiving, from the master device, an update message indicating change to the dataset; and
updating the dataset on the local storage media.
9. The method of claim 1 , wherein the determination that the quantity of updates to the dataset satisfies the data update frequency threshold further comprises:
determining, based on a snapshot of the dataset, that the quantity of updates to the dataset satisfies the data update frequency threshold.
10. A computing system comprising:
a master device of a plurality of worker devices comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors cause the master device to:
monitor a dataset on the master device for changes to the dataset; and
a worker device comprising:
one or more second processors; and
second memory storing instructions that, when executed by the one or more second processors, cause the worker device to:
subscribe to the dataset on the master device;
receive, from the master device, the dataset;
determine, based on metadata associated with the dataset, that a quantity of updates to the dataset within a time period satisfies a data update frequency threshold;
based on the determination that the quantity of updates within the time period satisfies the data update frequency threshold, store the dataset on a local storage media; and
execute a plurality of applications to use the dataset, wherein each application of the plurality of applications is configured to share, with each other application of the plurality of applications, the dataset from the local storage media.
11. A method comprising:
monitoring, by a master device of a plurality of worker devices, a dataset for changes to the dataset;
receiving, by the master device and from a worker device of the plurality of worker devices, a request to subscribe to the dataset;
sending, by the master device and to the worker device, the dataset;
determining, by the master device, based on metadata associated with the dataset, that a quantity of updates to the dataset within a time period satisfies a data update frequency threshold; and
based on determining that the quantity of updates to the dataset within the time period satisfies the data update frequency threshold, sending, by the master device, information to the worker device to update the dataset.
12. The method of claim 11 , wherein the information to update the dataset comprises:
a snapshot of the dataset.
13. The method of claim 12 , wherein the snapshot of the dataset comprises:
at least one of:
a set of reference markers for data at a point in time; or
an indication of a collection of data objects at a point in time.
14. The method of claim 11 , wherein the information to update the dataset comprises:
metadata associated with the dataset.
15. The method of claim 11 , further comprising, sending by the master device and to the worker device a data update message indicating a change to the dataset.
16. The method of claim 15 , wherein the data update message is sent based on the request to subscribe to the dataset.
17. The method of claim 11 , further comprising:
receiving, by the master device and from the worker device, a request for an updated dataset based on a determination that the quantity of updates within the time period satisfies a data update frequency threshold; and
sending, by the master device and to the worker device, the updated dataset.
18. The method of claim 17 , wherein the receiving the request for the updated dataset is further based on a determination that a size of the dataset satisfies a threshold.