Data object delivery for distributed cluster computing
Methods and systems for delivering data for cluster computing are described herein. A worker device may receive a dataset and store the dataset in a local storage media. This may prevent the need for the dataset to be sent over a network each time the applications are used to perform a task. Each application may be able to access the dataset in the local storage area. This may prevent the need to copy the dataset to memory associated with each application. A worker device may store a dataset, for example, if it determines that the frequency of updates to the dataset satisfy a threshold. The worker device may receive updates to the dataset via a messaging system and may store the updated data in the local storage media.
1. A method comprising:
subscribing, by a worker device of a cluster, to a dataset;
receiving, via a first process executing on the worker device and from a master device of the cluster, the dataset;
storing the dataset on a local storage media of the worker device comprising:
determining, based on metadata associated with the dataset, that a quantity of updates to the dataset within a time period satisfies a data update frequency threshold; and
based on determining that the quantity of updates within the time period satisfies the data update frequency threshold, storing the dataset in the local storage media;
executing, via one or more processes different from the first process, a plurality of applications to perform a task on the dataset, wherein each application of the plurality of applications is configured to share, with each other application of the plurality of applications, the dataset from the local storage media;
receiving, based on the subscribing and from the master device, a data update message indicating a change to the dataset; and
updating, based on the data update message, the dataset on the local storage media.
2. The method of claim 1 , further comprising:
ceasing, based on a determination that a no-op message has not been received within a threshold time period, execution of the plurality of applications.
3. The method of claim 2 , wherein the no-op message indicates that there have been no updates to the dataset.
4. The method of claim 1 , wherein the storing the dataset comprises:
determining, based on metadata associated with the dataset, that a size of the dataset satisfies a threshold; and
based on the determining that the size of the dataset satisfies the threshold, storing the dataset in the local storage media.
5. The method of claim 1 , further comprising:
receiving, via the first process and during execution of the plurality of applications, a message comprising an update to the dataset; and
updating, via the first process, the dataset in the local storage media.
6. The method of claim 1 , wherein the local storage media comprises a solid-state drive.
7. The method of claim 1 , wherein the executing the plurality of applications comprises joining the dataset from the local storage media with a second dataset received from the master device.
8. A cluster computing system comprising:
a worker device in communication with a master device;
wherein the worker device is configured to:
subscribe to a dataset;
receive, via a first process executing on the worker device and from the master device, the dataset;
store the dataset on a local storage media of the worker device;
wherein the storing the dataset comprises:
determining, based on metadata associated with the dataset, that a quantity of updates to the dataset with a time period satisfies a data update frequency threshold; and
based on the determining that the quantity of updates within the time period satisfies the data update frequency threshold, storing the dataset in the local storage media;
execute, via one or more processes different from the first process, a plurality of applications to perform a task on the dataset, wherein each application of the plurality of applications is configured to share, with each other application of the plurality of applications, the dataset from the local storage media;
receive, based on the subscribing and from the master device, a data update message indicating a change to the dataset; and
update, based on the data update message, the dataset on the local storage media.
9. The system of claim 8 , wherein the worker device is further configured to:
cease, based on a determination that a no-op message has not been received within a threshold time period, execution of the plurality of applications.
10. The system of claim 9 , wherein the no-op message indicates that there have been no updates to the dataset.
11. The system of claim 8 , wherein the storing the dataset comprises:
determining, based on metadata associated with the dataset, that a size of the dataset satisfies a threshold; and
based on the determining that the size of the dataset satisfies the threshold, storing the dataset in the local storage media.
12. The system of claim 8 , wherein the worker device is further configured to:
receive, via the first process and during execution of the plurality of applications, a message comprising an update to the dataset; and
update, via the first process, the dataset in the local storage media.
13. The system of claim 8 , wherein the local storage media comprises a solid-state drive.
14. The system of claim 8 , wherein the executing the plurality of applications comprises joining the dataset from the local storage media with a second dataset received from the master device.
15. A non-transitory machine-readable medium storing instructions, that when executed by one or more processors, cause the one or more processors to:
subscribe, by a worker device, to a dataset;
receive, via a first process executing on the worker device and from a master device, the dataset;
store the dataset on a local storage media of the worker device;
wherein the storing the dataset comprises;
determining based on metadata associated with the dataset, that a quantity of updates to the dataset within a time period satisfies a data update frequency threshold; and
based on the determining that the quantity of updates within the time period satisfies the data update frequency threshold, store the dataset on a local storage media;
execute, via one or more processes different from the first process, a plurality of applications to perform a task on the dataset, wherein each application of the plurality of applications is configured to share, with each other application of the plurality of applications, the dataset from the local storage media;
receive, based on the subscribing and from the master device, a data update message indicating a change to the dataset; and
update, based on the data update message, the dataset on the local storage media.
16. The non-transitory machine-readable medium of claim 15 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
cease, based on a determination that a no-op message has not been received within a threshold time period, execution of the plurality of applications.
17. The non-transitory machine-readable medium of claim 16 , wherein the no-op message indicates that there have been no updates to the dataset.