Managing metadata for a distributed processing system with manager agents and worker agents

1. A method implemented by data processing apparatus, the method comprising:

accessing, by a manager agent executing in data processor, a grain hierarchy that represents a state for a computation that is to be performed by a distributed processing system that includes a plurality of data processors, the grain hierarchy organizing a set of grains according to a tree structure, and wherein each grain included in the set of grains comprises metadata that specifies a portion of the state for the computation, wherein the grain hierarchy comprises:

manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system;

worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task; and

subscription grains defining subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy and specifying an interest in the grain by a subscribing agent, and wherein each subscribing agent is a i) manager agent having a corresponding grain included in the grain hierarchy, or ii) a worker agent having a corresponding grain included in the grain hierarchy;

performing, by the manager agent, manager agent processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent; and

performing, by each of a plurality of worker agents, tasks assigned to the worker agents, each task assigned to a worker agent based on an assignment of a respective worker agent grain to the worker agent;

wherein one of the one or more subscriptions is an observe subscription that results in a notification to a subscribing agent of the observe subscription of a specified change in the portion of the state specified by the one or more grains of the grain hierarchy to which the observe subscription corresponds.

2. The method of claim 1 , wherein each grain specifies one or more of:

a name that describes a location of the grain within the grain hierarchy;

a current state for the grain;

an owner that owns the grain;

an expiration time at which ownership of the grain will expire;

statistics that include measurements for monitoring progress of the computation;

restrictions on ownership of the grain;

dependencies on other grains included in the grain hierarchy; or

a timestamp indicating the most recent update to the grain.

3. The method of claim 1 , wherein the manager agent comprises one of:

a launcher agent process that observes a worker agent grain sub-tree and launches processes;

a matcher agent process that matches data to be processed by the distributed processing system to one or more of the worker agents; or

a task shaper agent process that manages a task grain sub-tree.

4. A method implemented by data processing apparatus, the method comprising:

accessing, by a manager agent executing in data processor, a grain hierarchy that represents a state for a computation that is to be performed by a distributed processing system that includes a plurality of data processors, the grain hierarchy organizing a set of grains according to a tree structure, and wherein each grain included in the set of grains comprises metadata that specifies a portion of the state for the computation, wherein the grain hierarchy comprises:

manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system;

worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task; and

subscription grains defining subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy and specifying an interest in the grain by a subscribing agent, and wherein each subscribing agent is a i) manager agent having a corresponding grain included in the grain hierarchy, or ii) a worker agent having a corresponding grain included in the grain hierarchy;

performing, by the manager agent, manager agent processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent; and

performing, by each of a plurality of worker agents, tasks assigned to the worker agents, each task assigned to a worker agent based on an assignment of a respective worker agent grain to the worker agent;

wherein one of the one or more subscriptions is a query subscription that results in a subscribing agent of the query subscription to be provided with specific information associated with the portion of the state specified by the one or more grains of the grain hierarchy to which the query subscription corresponds.

5. A method implemented by data processing apparatus, the method comprising:

accessing, by a manager agent executing in data processor, a grain hierarchy that represents a state for a computation that is to be performed by a distributed processing system that includes a plurality of data processors, the grain hierarchy organizing a set of grains according to a tree structure, and wherein each grain included in the set of grains comprises metadata that specifies a portion of the state for the computation, wherein the grain hierarchy comprises:

manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system;

worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task; and

subscription grains defining subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy and specifying an interest in the grain by a subscribing agent, and wherein each subscribing agent is a i) manager agent having a corresponding grain included in the grain hierarchy, or ii) a worker agent having a corresponding grain included in the grain hierarchy;

performing, by the manager agent, manager agent processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent; and

performing, by each of a plurality of worker agents, tasks assigned to the worker agents, each task assigned to a worker agent based on an assignment of a respective worker agent grain to the worker agent;

wherein one of the one or more subscriptions is a claim subscription that results in an assignment of ownership of a grain included in the one or more grains of the grain hierarchy to which the claim subscription corresponds.

6. A method implemented by data processing apparatus, the method comprising:

accessing, by a manager agent executing in data processor, a grain hierarchy that represents a state for a computation that is to be performed by a distributed processing system that includes a plurality of data processors, the grain hierarchy organizing a set of grains according to a tree structure, and wherein each grain included in the set of grains comprises metadata that specifies a portion of the state for the computation, wherein the grain hierarchy comprises:

manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system;

worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task; and

subscription grains defining subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy and specifying an interest in the grain by a subscribing agent, and wherein each subscribing agent is a i) manager agent having a corresponding grain included in the grain hierarchy, or ii) a worker agent having a corresponding grain included in the grain hierarchy;

performing, by the manager agent, manager agent processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent; and

performing, by each of a plurality of worker agents, tasks assigned to the worker agents, each task assigned to a worker agent based on an assignment of a respective worker agent grain to the worker agent; further comprising:

monitoring one or more changes to one or more of the grains included in the grain hierarchy during performance of the computation; and

providing a subscribing agent a notification in response to detecting a particular change to a particular grain having a corresponding subscription subscribed to by the subscribing agent.

7. A method implemented by data processing apparatus, the method comprising:

accessing, by a manager agent executing in data processor, a grain hierarchy that represents a state for a computation that is to be performed by a distributed processing system that includes a plurality of data processors, the grain hierarchy organizing a set of grains according to a tree structure, and wherein each grain included in the set of grains comprises metadata that specifies a portion of the state for the computation, wherein the grain hierarchy comprises:

manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system;

worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task; and

subscription grains defining subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy and specifying an interest in the grain by a subscribing agent, and wherein each subscribing agent is a i) manager agent having a corresponding grain included in the grain hierarchy, or ii) a worker agent having a corresponding grain included in the grain hierarchy;

performing, by the manager agent, manager agent processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent; and

performing, by each of a plurality of worker agents, tasks assigned to the worker agents, each task assigned to a worker agent based on an assignment of a respective worker agent grain to the worker agent;

wherein the subscription grains specify actions that include one or more of:

providing a subscribing agent with a notification that a pre-specified event occurred in the one or more grains to which an observe subscription corresponds;

assigning, to a subscribing agent, ownership of the one or more grains to which a claim subscription corresponds;

updating a current state for one or more grains; or

providing a subscribing agent with particular metadata included in the one or more grains to which a query subscription corresponds.

8. A system comprising:

a plurality of data processing apparatus; and

a data store storing instructions that, when executed by plurality of data processing apparatus, cause the data processing apparatus to perform operations comprising:

accessing, by a manager agent executing in data processor, a grain hierarchy that represents a state for a computation that is to be performed by a distributed processing system that includes a plurality of data processors, the grain hierarchy organizing a set of grains according to a tree structure, and wherein each grain included in the set of grains comprises metadata that specifies a portion of the state for the computation, wherein the grain hierarchy comprises:

manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system;

worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task; and

subscription grains defining subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy and specifying an interest in the grain by a subscribing agent, and wherein each subscribing agent is a i) manager agent having a corresponding grain included in the grain hierarchy, or ii) a worker agent having a corresponding grain included in the grain hierarchy;

performing, by the manager agent, manager agent processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent; and

performing, by each of a plurality of worker agents, tasks assigned to the worker agents, each task assigned to a worker agent based on an assignment of a respective worker agent grain to the worker agent.

9. A non-transitory computer readable medium storing instructions that, when executed by a plurality of data processing apparatus, cause the data processing apparatus to perform operations comprising:

accessing, by a manager agent executing in data processor, a grain hierarchy that represents a state for a computation that is to be performed by a distributed processing system that includes a plurality of data processors, the grain hierarchy organizing a set of grains according to a tree structure, and wherein each grain included in the set of grains comprises metadata that specifies a portion of the state for the computation, wherein the grain hierarchy comprises:

manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system;

worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task; and

subscription grains defining subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy and specifying an interest in the grain by a subscribing agent, and wherein each subscribing agent is a i) manager agent having a corresponding grain included in the grain hierarchy, or ii) a worker agent having a corresponding grain included in the grain hierarchy;

performing, by the manager agent, manager agent processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent; and

performing, by each of a plurality of worker agents, tasks assigned to the worker agents, each task assigned to a worker agent based on an assignment of a respective worker agent grain to the worker agent.