Distributed data stream programming and processing
Techniques are described herein for distributed data stream programming and processing. The techniques include sending a request indicating one or more regions of a program code to access a stream in a stream pool and to execute on a processing node in a processing nodes pool. The techniques also include accessing the stream defined in the one or more regions of the program code to service the request. Thereafter, the processing node is selected to use for execution of the one or more regions of the program code and the processing node executes one or more instances of the one or more regions of the program code.
1. One or more non-transitory computer-readable media storing computer-executable instructions that upon execution cause one or more processors to perform acts comprising:
sending a request indicating one or more regions of program code to access a stream in a stream pool and to execute on a processing node in a processing nodes pool, the processing node corresponding to a stream rank of the stream;
accessing the stream defined in the one or more regions of the program code to service the request;
selecting the processing node to use for execution of the one or more regions of the program code based at least on the stream rank of the stream; and
commanding the processing node to execute one or more instances of the one or more regions of the program code.
2. The one or more non-transitory computer-readable media of claim 1 , wherein the one or more regions of the program code comprises a global descriptor corresponding to rank information of the processing node.
3. The one or more non-transitory computer-readable media of claim 1 , wherein the one or more regions of the program code defines a global descriptor corresponding to the stream rank of the stream.
4. The one or more non-transitory computer-readable media of claim 1 , wherein the stream comprises one or more partitions and the one or more regions of the program code define a global descriptor corresponding to the one or more partitions.
5. The one or more non-transitory computer-readable media of claim 4 , wherein the acts further comprise:
accessing the one or more partitions corresponding to the global descriptor to service the request.
6. The one or more non-transitory computer-readable media of claim 1 , wherein the one or more regions of the program code is marked up with language-specific keywords mapping to an application programming interface (API).
7. The one or more non-transitory computer-readable media of claim 1 , wherein the stream rank maps to a topic rank that is associated with a topic; and
selecting the processing node to use for execution of the one or more regions of the program code based at least on the topic specified by the stream rank.
8. A computer-implemented method, comprising:
sending a request indicating one or more regions of a program code to access a stream in a stream pool and to execute on a processing node in a processing nodes pool, the processing node corresponding to a stream rank of the stream;
accessing the stream defined in the one or more regions of the program code to service the request;
selecting the processing node to use for execution of the one or more regions of the program code based at least on the stream rank of the stream; and
commanding the processing node to execute one or more instances of the one or more regions of the program code.
9. The computer-implemented method of claim 8 , wherein the one or more regions of the program code comprises a global descriptor corresponding to a name of the processing node.
10. The computer-implemented method of claim 8 , wherein the processing node is selected to load balance the request.
11. The computer-implemented method of claim 8 , wherein the one or more regions of the program code defines a global descriptor corresponding to a name of the stream.
12. The computer-implemented method of claim 8 , wherein the stream comprises one or more partitions and the one or more regions of the program code define a global descriptor corresponding to the one or more partitions.
13. The computer-implemented method of claim 12 , further comprising:
accessing the one or more partitions corresponding to the global descriptor to service the request.
14. The computer-implemented method of claim 8 , wherein the one or more regions of the program code is marked up with a language-generic pragma code words.
15. A system, comprising:
one or more non-transitory storage mediums configured to provide stored computer-readable instructions, the one or more non-transitory storage mediums coupled to one or more processors, the one or more processors configured to execute the computer-readable instructions to cause the one or more processors to:
send a request indicating one or more regions of a program code to access a stream in a stream pool and to execute on a processing node in a processing nodes pool, the processing node corresponding to a stream rank of the stream;
access the stream defined in the one or more regions of the program code to service the request;
select the processing node to use for execution of the one or more regions of the program code based at least on the stream rank of the stream; and
command the processing node to execute one or more instances of the one or more regions of the program code.
16. The system of claim 15 , wherein the one or more regions of the program code comprises a global descriptor corresponding to a group of processing nodes comprising the processing node.
17. The system of claim 15 , wherein the one or more regions of the program code defines a global descriptor corresponding to a range of stream ranks of the stream.
18. The system of claim 15 , wherein the stream comprises one or more partitions and the one or more regions of the program code define a global descriptor corresponding to the one or more partitions.
19. The system of claim 18 , wherein the instructions further cause the one or more processors to:
access the one or more partitions corresponding to the global descriptor to service the request.
20. The system of claim 15 , wherein the one or more regions of the program code comprises a global descriptor corresponding to a total number of processing nodes in the processing nodes pool.