Distributed real-time partitioned MapReduce for a data fabric

1. A system for a distributed real-time partitioned map reduce, comprising

a plurality of nodes;

a service node, wherein the service node is selected by a client application system to manage execution of a task to obtain a service node result, wherein the service node is selected to balance a load on the service node and the plurality of nodes; and

a plurality of partitions,

wherein one or more partitions of the plurality of partitions are stored on a first node of the plurality of nodes,

wherein a partition stored on the first node comprises partition data and is configured to:

receive a request for a partition result for the partition data from the service node;

determine the partition result by querying a set of predetermined partition results stored by the partition to select as the partition result a predetermined partition result of the set of predetermined partition results that corresponds to a previous execution of the request for the partition result;

provide the partition result to the service node; and

wherein the service node is configured to:

receive a set of partition results from the plurality of partitions, wherein the set of partition results comprises the partition result; and

perform a service node reduce on the set of partition results to obtain the service node result.

2. The system of claim 1 , wherein the service node is configured to receive the task from the client application system.

3. The system of claim 1 , wherein the service node is configured to provide a request for a partition result to a plurality of partitions.

4. The system of claim 1 , wherein the partitions are part of a data fabric.

5. The system of claim 1 , wherein the partition receives partition task logic from the service node, wherein the partition task logic comprises the request for the partition task result.

6. The system of claim 1 , wherein the predetermined partition result is a first predetermined partition result, wherein the partition is further configured to:

receive an indication that the partition data has changed;

determine whether the change to the partition data causes a change to a second predetermined partition result of the set of predetermined partition results stored by the partition; and

in response to a determination that the change to the partition data affects the second predetermined partition result of the set of predetermined partition results stored by the partition:

determine a new value for the second predetermined partition result; and

store the new value.

7. The system of claim 1 , wherein the service node is further configured to determine the service node result by querying a set of predetermined service node results stored by the service node.

8. The system of claim 5 , wherein the set of predetermined service node results is stored by each node of the plurality of nodes.

9. The system of claim 5 , wherein the predetermined partition result is a first predetermined partition result, wherein the partition is further configured to:

detect a change to a second predetermined partition result of the set of predetermined partition results; and

provide the changed predetermined partition result to the service node for updating a set of predetermined service node results.

10. The system of claim 1 , wherein the first node is configured to store a set of predetermined node results.

11. The system of claim 10 , wherein the predetermined partition result is a first predetermined partition result, wherein the partition is further configured to:

detect a change to a second predetermined partition result of the set of predetermined partition results; and

provide the changed predetermined partition result to the first node for updating a set of predetermined node results.

12. The system of claim 10 , wherein the first node is configured to receive a changed partition result from the partition and update a predetermined node result of a set of predetermined node results.

13. The system of claim 10 , wherein each node of the set of nodes is configured to store a set of predetermined node results.

14. The system of claim 13 , wherein the service node is further configured to query each node for a predetermined node result.

15. The system of claim 14 , wherein the first node is further configured to:

receive a request from the service node for a predetermined node result;

determine the predetermined node result from the set of predetermined node results; and

provide the predetermined node result in response to the request.

16. The system of claim 1 , wherein the service node is selected from the plurality of nodes.

17. The system of claim 16 , wherein a first service node for a first task and a second service node for a second task comprise different nodes.

18. A computer program product for a distributed partitioned map reduce, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

receiving at a partition stored on a first node of a plurality of nodes a request for a partition result for partition data from a service node, wherein the service node is selected by a client application system to manage execution of a task to obtain a service node result, wherein the service node is selected to balance a load on the plurality of loads and the service node, and wherein the partition comprises the partition data and wherein the partition comprises the partition data and is one of a plurality of partitions, and wherein one or more partitions of the plurality of partitions are stored on the first node;

determining, using a processor, the partition result by querying a set of predetermined partition results stored by the partition to select as the partition result a predetermined partition result of the set of predetermined partition results that corresponds to a previous execution of the request for the partition result;

providing the partition result to the service node;

receiving at the service node a set of partition results from the plurality of partitions, wherein the set of partition results comprises the partition result; and

performing at the service node a service node reduce on the set of partition results to obtain the service node result.

19. A method for a distributed real-time partitioned map reduce, comprising

receiving at a partition stored on a first node of a plurality of nodes a request for a partition result for partition data from a service node, wherein the service node is selected by a client application system to manage execution of a task to obtain a service node result, wherein the service node is selected to balance a load on the plurality of loads and the service node, and wherein the partition comprises the partition data and is one of a plurality of partitions, and wherein one or more partitions of the plurality of partitions are stored on the first node;

determining, using a processor, the partition result by querying a set of predetermined partition results stored by the partition to select as the partition result a predetermined partition result of the set of predetermined partition results that corresponds to a previous execution of the request for the partition result;

providing the partition result to the service node;

receiving at the service node a set of partition results from the plurality of partitions, wherein the set of partition results comprises the partition result; and

performing at the service node a service node reduce on the set of partition results to obtain the service node result.