Managing metadata for a distributed processing system with manager agents and worker agents
A manager agent access a grain hierarchy that represents a state for a computation that is to performed by a distributed processing system, wherein the grain hierarchy includes manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system, and worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task. A manager agent performs processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent, and worker agents perform tasks assigned to the worker agents based on an assignment of a respective worker agent grain to the worker agent.
1. A data processing apparatus that manages metadata, comprising:
a memory for storing instructions, and
one or more computing devices coupled to the memory, the instructions causing the one or computing devices to:
receive a specification that includes a set of grains that specify an initial state for a computation;
communicate with one or more manager agents and worker agents, at least one manager agent and at least one worker agent, of the one or more manager agents and worker agents, associated with performing the computation;
represent the set of grains in a grain hierarchy, the grain hierarchy representing a computation state and comprising subscription grains for one or more subscriptions to the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy and each subscription specifying one or more actions to be performed by the data processing apparatus;
perform, during performance of the computation, at least one of the one or more actions specified by the one or more subscriptions;
receive, during performance of the computation, updates associated with the set of grains; and
update the grain hierarchy based on the received updates.
2. The data processing apparatus of claim 1 , wherein the grain hierarchy comprises one or more manager agent grains specifying metadata for one or more manager agent processes that manage performance of the computation.
3. The data processing apparatus of claim 1 , wherein the grain hierarchy comprises one or more worker agent grains specifying metadata for one or more worker agent processes that carry out data processing for the computation.
4. The data processing apparatus of claim 1 , wherein one of the one or more subscriptions is an observe subscription that instructs the one or more computing devices to notify a subscribing agent of the observe subscription of a specified change in the a portion of the state specified by the one or more grains of the grain hierarchy to which the grain subscription corresponds.
5. The data processing apparatus of claim 1 , wherein one of the one or more subscriptions is a query subscription that instructs the one or more computing devices to, in response to receiving the query subscription, provide a subscribing agent of the query subscription with specific information associated with a portion of the state specified by the one or more grains of the grain hierarchy to which the query Subscription corresponds.
6. The data processing apparatus of claim 1 , wherein the instructions cause the one or more computing devices to store an updated hierarchy updated based on the received updates.
7. The data processing apparatus of claim 1 , wherein the grain hierarchy includes one or more of a manager agents grain that is a first root of a manager agent sub-tree, a datasets grain that is a second root of a dataset sub-tree and a worker agents grain that is a third root of a worker agent sub-tree.
8. The data processing apparatus of claim 7 , wherein the subscription grains correspond to a subscription grain sub-tree in the grain hierarchy.
9. A data processing method, comprising:
receiving a specification that includes a set of grains that specify an initial state for a computation;
communicating with one or more manager agents and worker agents, at least one manager agent and at least one worker agent, of the one or more manager agents and worker agents, associated with performing the computation;
representing the set of grains in a grain hierarchy, the grain hierarchy representing a computation state and comprising subscription grains for one or more subscriptions to the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy and each subscription specifying one or more actions to be performed by a data processing apparatus;
performing, during performance of the computation, at least one of the one or more actions specified by the one or more subscriptions;
receiving, during performance of the computation, updates associated with the set of grains; and
updating the grain hierarchy based on the received updates.
10. The data processing method of claim 9 , wherein the grain hierarchy comprises one or more manager agent grains specifying metadata for one or more manager agent processes that manage performance of the computation.
11. The data processing method of claim 9 , wherein the grain hierarchy comprises one or more worker agent grains specifying metadata for one or more worker agent processes that carry out data processing for the computation.
12. The data processing method of claim 9 , wherein one of the one or more subscriptions is an observe subscription that instructs the one or more computing devices to notify a subscribing agent of an observe subscription of a specified change in a portion of the state specified by the one or more grains of the grain hierarchy to which the grain subscription corresponds.
13. The data processing method of claim 9 , wherein one of the one or more subscriptions is a query subscription that instructs the one or more computing devices to, in response to receiving the query subscription, provide a subscribing agent of the query subscription with specific information associated with the portion of the state specified by the one or more grains of the grain hierarchy to which the query Subscription corresponds.
14. The data processing method of claim 9 , further comprising storing an updated hierarchy updated based on the received updates.
15. The data processing method of claim 9 , wherein the grain hierarchy includes one or more of a manager agents grain that is a first root of a manager agent sub-tree, a datasets grain that is a second root of a dataset sub-tree and a worker agents grain that is a third root of a worker agent sub-tree.
16. The data processing method of claim 15 , wherein the subscription grains correspond to a subscription grain sub-tree in the grain hierarchy.