Dynamically scheduled checkpoints in distributed data streaming system
One or more dynamically occurring events in a distributed data streaming system are monitored. The occurrence of at least one of the one or more dynamically occurring events is evaluated. A checkpoint operation is initiated in the distributed data streaming system based on the evaluation of the occurrence of the at least one dynamically occurring event.
1. A method comprising:
monitoring one or more dynamically occurring events in a distributed data streaming system;
dynamically injecting a checkpoint request into the distributed data streaming system based on the occurrence of at least one of the one or more dynamically occurring events;
evaluating the checkpoint request;
dynamically scheduling a checkpoint operation based on the results of evaluating the checkpoint request; and
initiating the checkpoint operation in the distributed data streaming system based on the dynamic scheduling;
wherein the dynamically scheduled checkpoint operation is integrated with a predetermined checkpoint operation;
wherein the distributed data streaming system is implemented via one or more processing devices each comprising a processor coupled to a memory.
2. The method of claim 1 , where in the one or more dynamically occurring events comprise a system-based event.
3. The method of claim 2 , wherein the system-based event comprises at least one of a pending shutdown and a pending reboot.
4. The method of claim 1 , wherein the evaluating step further comprises prioritizing the one or more dynamically occurring events such that the checkpoint operation is initiated in accordance with the prioritization.
5. The method of claim 4 , wherein the prioritization is a function of the level of the one or more dynamically occurring events.
6. The method of claim 5 , wherein the level of the one or more dynamically occurring events comprises one of a stream-level event, a system-level event, and a processing node-level event, wherein the distributed data streaming system comprises a plurality of processing nodes.
7. The method of claim 1 , wherein the evaluating step further comprises determining an impact that a checkpoint operation may have on the distributed data streaming system if initiated.
8. The method of claim 1 , further comprising registering the one or more dynamically occurring events to be monitored.
9. The method of claim 8 , wherein the one or more dynamically occurring events to be monitored are registered via a user interface of the distributed data streaming system.
10. The method of claim 9 , wherein the interface is configured to enable setting of one or more parameters that dictate initiation of the checkpoint operation.
11. The method of claim 1 , wherein the one or more dynamically occurring events comprise a stream-based event.
12. The method of claim 11 , wherein the stream-based event comprises at least one of a stream modification, a stream redeployment, and a stream partitioning.
13. The method of claim 1 , wherein the one or more dynamically occurring events comprise a system resource-based event.
14. The method of claim 13 , wherein a system resource comprises one or more of a system resource capacity maintaining a predetermined capacity level, a compute resource, a memory resource, an input-output resource, and a network resource of the distributed data streaming system.
15. The method of claim 1 , wherein the one or more dynamically occurring events comprise a service level agreement-based event.
16. The method of claim 15 , wherein the service level agreement-based event comprises a service level agreement remaining compliant.
17. A system comprising:
at least one processor, coupled to a memory, and configured to:
monitor one or more dynamically occurring events in a distributed data streaming system;
dynamically inject a checkpoint request into the distributed data streaming system based on the occurrence of at least one of the one or more dynamically occurring events;
evaluate the checkpoint request;
dynamically schedule a checkpoint operation based on the results of evaluating the checkpoint request; and
initiate the checkpoint operation in the distributed data streaming system based on the dynamic scheduling;
wherein the dynamically scheduled checkpoint operation is integrated with a predetermined checkpoint operation.
18. The system of claim 17 , where in the one or more dynamically occurring events comprise a system-based event.
19. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device to:
monitoring one or more dynamically occurring events in a distributed data streaming system;
dynamically injecting a checkpoint request into the distributed data streaming system based on the occurrence of at least one of the one or more dynamically occurring events;
evaluating the checkpoint request;
dynamically scheduling a checkpoint operation based on the results of evaluating the checkpoint request; and
initiating the checkpoint operation in the distributed data streaming system based on the dynamic scheduling;
wherein the dynamically scheduled checkpoint operation is integrated with a predetermined checkpoint operation.
20. The computer program product of claim 19 , where in the one or more dynamically occurring events comprise a system-based event.