Method of executing a tuple graph program across a network

1. A method for executing a program in a distributed architecture, comprising:

executing, by one or more first shards of the distributed architecture, one or more operations;

sending, from the one or more first shards, tuples to at least one second shard, the tuples being part of a stream and being based on the one or more operations;

sending, from each of the one or more first shards, a token value to the at least one second shard when the sending of the tuples in the stream is complete;

determining, by the second shard, whether a total of the token values matches a number of the one or more first shards; and

taking a first action in response to determining that the total of the token values matches the number of the one or more first shards.

2. The method of claim 1 , wherein the at least one second shard is a receive shard of the one of the one or more first shards, the method further comprising:

generating, by the one of the one or more first shards, a list of the receive shards with which the one or more first shards communicate; and

transmitting, by the one of the one or more first shards, the list to a controller.

3. The method of claim 2 , further comprising:

tracking, by the controller, all receive shards that have started processing;

determining, by the controller, whether one or more of the receive shards that have started processing are not present in the list; and

for each receive shard that has started processing and is not present in the list, sending, by the controller, a token value on behalf of the one of the one or more first shards to the receive shard.

4. The method of claim 1 , further comprising:

determining, by a controller, whether any shards have not begun processing;

determining, by the controller, whether the shards that have not begun processing are intentionally skipped by design of the program; and

sending, by the controller on behalf of any intentionally skipped shards that have not begun processing, a token value to the second shard.

5. The method of claim 1 , wherein taking the first action comprises at least one of marking the stream as being complete or generating a message indicating that the stream is complete.

6. The method of claim 1 , further comprising:

constructing a graph, wherein each node of the graph represents a shard; and

verifying, based on the graph, whether the program will be accurately executed across the distributed architecture.

7. The method of claim 6 , further comprising dynamically building activations of the graph as the program is executed.

8. The method of claim 7 , further comprising:

sending, by a dynamic send operation, a data input stream to all activations of a destination graph;

receiving, at the dynamic send operation from the controller, new tuples received as additional activations of the destination graph are detected.

9. The method of claim 6 , wherein the graph is non-unique.

10. The method of claim 1 , wherein executing the one or more operations is part of a pipelined data processing flow.

11. A system, comprising:

one or more first shards operable as one or more first computing devices in communication with memory in a distributed computing environment; and

at least one second shard operable as at least a second computing device in communication with memory in the distributed computing environment, the at least one second shard being remote from the one or more first shards;

wherein the one or more first shards are configured to:

execute one or more operations;

send tuples to at least one second shard, the tuples being part of a stream and being based on the one or more operations;

send a token value to the at least one second shard when the sending of the tuples in the stream is complete; and

wherein the at least one second shard is configured to:

determine whether a total of the token values matches a number of the one or more first shards; and

take a first action in response to determining that the total of the token values matches the number of the one or more first shards.

12. The system of claim 11 , further comprising a controller, wherein the at least one second shard is a receive shard of the one of the one or more first shards, and wherein the one or more first shards are further configured to:

generate a list of the receive shards with which the one or more first shards communicate; and

transmit the list to the controller.

13. The system of claim 12 , wherein the controller is configured to:

track all receive shards that have started processing;

determine whether one or more of the receive shards that have started processing are not present in the list; and

for each receive shard that has started processing and is not present in the list, send a token value on behalf of the one of the one or more first shards to the receive shard.

14. The system of claim 11 , wherein the controller is configured to:

determine whether any shards have not begun processing;

determine whether the shards that have not begun processing are intentionally skipped by design of the program; and

send, on behalf of any intentionally skipped shards that have not begun processing, a token value to the second shard.

15. The system of claim 11 , wherein taking the first action comprises at least one of marking the stream as being complete or generating a message indicating that the stream is complete.

16. The system of claim 11 , further comprising a client device in communication with at least one of the one or more first shards, the at least one second shard, or the controller, the client device configured to:

construct a graph, wherein each node of the graph represents a shard; and

verify, based on the graph, whether the program will be accurately executed across the distributed architecture.

17. The system of claim 16 , wherein the client device is further configured to dynamically building activations of the graph as the program is executed.

18. The system of claim 17 , further comprising a dynamic send operation executed on a computing device in the distributed architecture, wherein the dynamic send operation:

sends a data input stream to all activations of a destination graph;

receives, from the controller, new tuples received as additional activations of the destination graph are detected.

19. The system of claim 16 , wherein the graph is non-unique.

20. The system of claim 11 , wherein executing the one or more operations is part of a pipelined data processing flow.