Distributed stream processing in the Cloud
A low-latency cloud-scale computation environment includes a query language, optimization, scheduling, fault tolerance and fault recovery. An event model can be used to extend a declarative query language so that temporal analysis of event of an event stream can be performed. Extractors and outputters can be used to define and implement functions that extend the capabilities of the event-based query language. A script written in the extended query language can be translated into an optimal parallel continuous execution plan. Execution of the plan can be orchestrated by a streaming job manager which schedules vertices on available computing machines. The streaming job manager can monitor overall job execution. Fault tolerance can be provided by tracking execution progress and data dependencies in each vertex. In the event of a failure, another instance of the failed vertex can be scheduled. An optimal recovery point can be determined based on checkpoints and data dependencies.
1. A computing device in a network-based computing environment, comprising:
at least one processor:
at least one memory connected to the at least one processor; and
a distributed stream processing system stored in the at least one memory and executed by the at least one processor, comprising
a streaming job manager that monitors execution information about streaming jobs executed by a plurality of vertices executing on a plurality of computing devices, the streaming job manager receiving execution progress information and data dependencies for the plurality of vertices, each vertex of the plurality of vertices configured to process events associated with one or more streaming jobs,
wherein the plurality of vertices includes at least one stream extractor vertex configured to consume events of one or more event streams, including continually waiting for and performing computations on data received in the one or more event streams; and
the streaming job manager is configured to
detect, based on the monitored execution information, a failed vertex of the plurality of vertices, and
restart the failed vertex.
2. The computing device of claim 1 , wherein the stream extractor vertex is invoked from at least one of a query or script written in a query language that includes one or more temporal operators.
3. The computing device of claim 1 , wherein the stream extractor is configured to indicate temporal information for an event.
4. The computing device of claim 3 , wherein the temporal information includes at least one of a time the event began, a time the event ended, a time period during which the event was active, or sequence information for the event.
5. The computing device of claim 1 , wherein the stream extractor vertex is user-defined.
6. The computing device of claim 1 , further comprising:
a garbage collection process configured to garbage collect the event based on a sequence number of the event.
7. The computing device of claim 1 , wherein the plurality of vertices includes a stream outputter vertex configured to perform one or more user-defined actions processing one or more streaming output events.
8. A method in a distributed stream processing system implemented in at least one computing device, comprising
monitoring execution information about streaming jobs executed by a plurality of vertices executing on a plurality of computing devices, said monitoring including
receiving execution progress information and data dependencies for the plurality of vertices, each vertex of the plurality of vertices configured to process events associated with one or more streaming jobs, the plurality of vertices including a stream extractor vertex;
consuming events of one or more event streams at the stream extractor vertex, said consuming including
continually waiting for data received in the one or more event streams, and
performing computations on the data;
detecting, based on the monitored execution information, a failed vertex of the plurality of vertices, and
restarting the failed vertex.
9. The method of claim 8 , further comprising:
invoking the stream extractor vertex from at least one of a query or script written in a query language that includes one or more temporal operators.
10. The method of claim 8 , further comprising:
configuring the stream extractor to indicate temporal information for an event.
11. The method of claim 10 , wherein the temporal information includes at least one of a time the event began, a time the event ended, a time period during which the event was active, or sequence information for the event.
12. The method of claim 8 , further comprising:
enabling a user to define the stream extractor vertex.
13. The method of claim 8 , further comprising:
garbage collecting the event based on a sequence number of the event.
14. The method of claim 8 , wherein the plurality of vertices includes a stream outputter vertex, the method further comprising:
performing one or more user-defined actions with the stream outputter vertex to process one or more streaming output events.
15. A computer-readable storage medium having program instructions recorded thereon that, when executed by at least one processing circuit, perform a method for distributed stream processing, the method comprising:
monitoring execution information about streaming jobs executed by a plurality of vertices executing on a plurality of computing devices, said monitoring including
receiving execution progress information and data dependencies for the plurality of vertices, each vertex of the plurality of vertices configured to process events associated with one or more streaming jobs, the plurality of vertices including a stream extractor vertex; and
consuming events of one or more event streams at the stream extractor vertex, said consuming including
continually waiting for data received in the one or more event streams, and
performing computations on the data;
detecting, based on the monitored execution information, a failed vertex of the plurality of vertices, and
restarting the failed vertex.
16. The computer-readable storage medium of claim 15 , wherein the method further comprises:
invoking the stream extractor vertex from at least one of a query or script written in a query language that includes one or more temporal operators.
17. The computer-readable storage medium of claim 15 , wherein the method further comprises:
configuring the stream extractor to indicate temporal information for an event.
18. The computer-readable storage medium of claim 17 , wherein the temporal information includes at least one of a time the event began, a time the event ended, a time period during which the event was active, or sequence information for the event.
19. The computer-readable storage medium of claim 15 , wherein the method further comprises:
enabling a user to define the stream extractor vertex.
20. The computer-readable storage medium of claim 15 , wherein the method further comprises:
garbage collecting the event based on a sequence number of the event.