Fault-tolerant stream processing

1. A method, comprising:

processing, at a data processing system, input events to generate primary output events and secondary output events, wherein the primary output events are generated by a primary server of the data processing system and the secondary output events are generated by one or more secondary servers of the data processing system;

writing, by the data processing system, the primary output events to a primary target and the secondary output events to one or more secondary targets, wherein the primary output events are written by the primary server and the secondary output events are written by the one or more secondary servers, and wherein the processing the input events and the writing the primary output events to the primary target and the secondary output events to the one or more secondary targets are performed at a same time by the primary server and the one or more secondary servers;

receiving, at the data processing system, an election of a new primary server from a synchronization system upon a failure of the primary server, wherein the new primary server is elected from the one or more secondary servers;

reading, by the new primary server of the data processing system, the secondary output events from the one or more secondary targets;

determining, by the new primary server of the data processing system, the primary output events that failed to be written to the primary target because of the failure of the primary server; and

writing, by the new primary server of the data processing system, the failed primary output events to the primary target using the secondary output events read from the one or more secondary targets.

2. The method of claim 1 , further comprising:

reading, at the primary server of a data processing system, the input events from an input source;

reading, at the one or more secondary servers of the data processing system, the input events from the input source;

pausing, by the new primary server of the data processing system, the processing the input events and the writing of the secondary output events for at least the new primary server upon the election of the new primary server; and

unpausing, by the new primary server of the data processing system, the processing the input events and the writing of the secondary output events for at least the new primary server upon the writing the failed primary output events to the primary target.

3. The method of claim 2 , further comprising:

reading, at the new primary server of a data processing system, the input events from the input source upon the unpausing;

processing, at the new primary server of the data processing system, the input events to generate the primary output events; and

writing, by the new primary server of the data processing system, the primary output events to the primary target.

4. The method of claim 1 , further comprising:

writing, by the primary server of the data processing system, primary output information to the synchronization system upon writing the primary output events to the primary target; and

writing, by the one or more secondary servers of the data processing system, secondary output information to the synchronization system upon writing the secondary output events to the one or more secondary targets.

5. The method of claim 4 , wherein the primary output events that failed to be written to the primary target are determined based on the primary output information and the secondary information written to the synchronization system.

6. The method of claim 5 , wherein the primary output information and the secondary output information written include an identifier of the primary target and identifiers of the one or more secondary targets where the primary output events and the secondary output events are being written, respectively, and one or more indices to identify various levels of the primary output events and the secondary output events.

7. The method of claim 6 , further comprising:

reading, at the primary server of a data processing system, the input events from an input source; and

reading, at the one or more secondary servers of the data processing system, the input events from the input source,

wherein:

the input source is a topic of a messaging system;

the primary target is a primary output topic of the messaging system located on a broker or server of a primary datacenter; and

the one or more secondary targets are one or more secondary topics of the messaging system located on one or more brokers or servers of one or more secondary datacenters different from the primary datacenter.

8. A system comprising:

a data processing system that includes one or more processors and non-transitory machine readable storage medium having instructions stored thereon that when executed by the one or more processors cause the one or more processors to perform the process comprising:

processing, at the data processing system, input events to generate primary output events and secondary output events, wherein the primary output events are generated by a primary server of the data processing system and the secondary output events are generated by one or more secondary servers of the data processing system;

writing, by the data processing system, the primary output events to a primary target and the secondary output events to one or more secondary targets, wherein the primary output events are written by the primary server and the secondary output events are written by the one or more secondary servers, and wherein the processing the input events and the writing the primary output events to the primary target and the secondary output events to the one or more secondary targets are performed at a same time by the primary server and the one or more secondary servers;

receiving, at the data processing system, an election of a new primary server from a synchronization system upon a failure of the primary server, wherein the new primary server is elected from the one or more secondary servers;

reading, by the new primary server of the data processing system, the secondary output events from the one or more secondary targets;

determining, by the new primary server of the data processing system, the primary output events that failed to be written to the primary target because of the failure of the primary server; and

writing, by the new primary server of the data processing system, the failed primary output events to the primary target using the secondary output events read from the one or more secondary targets.

9. The system of claim 8 , wherein the process further comprises:

reading, at the primary server of a data processing system, the input events from an input source;

reading, at the one or more secondary servers of the data processing system, the input events from the input source;

pausing, by the new primary server of the data processing system, the processing the input events and the writing of the secondary output events for at least the new primary server upon the election of the new primary server; and

unpausing, by the new primary server of the data processing system, the processing the input events and the writing of the secondary output events for at least the new primary server upon the writing the failed primary output events to the primary target.

10. The system of claim 9 , wherein the process further comprises:

reading, at the new primary server of a data processing system, the input events from the input source upon the unpausing;

processing, at the new primary server of the data processing system, the input events to generate the primary output events; and

writing, by the new primary server of the data processing system, the primary output events to the primary target.

11. The system of claim 8 , wherein the process further comprises:

writing, by the primary server of the data processing system, primary output information to the synchronization system upon writing the primary output events to the primary target; and

writing, by the one or more secondary servers of the data processing system, secondary output information to the synchronization system upon writing the secondary output events to the one or more secondary targets.

12. The system of claim 11 , wherein the primary output events that failed to be written to the primary target are determined based on the primary output information and the secondary information written to the synchronization system.

13. The system of claim 12 , wherein the primary output information and the secondary output information written include an identifier of the primary target and identifiers of the one or more secondary targets where the primary output events and the secondary output events are being written, respectively, and one or more indices to identify various levels of the primary output events and the secondary output events.

14. The system of claim 13 , wherein the process further comprises:

reading, at the primary server of a data processing system, the input events from an input source; and

reading, at the one or more secondary servers of the data processing system, the input events from the input source,

wherein:

the input source is a topic of a messaging system;

the primary target is a primary output topic of the messaging system located on a broker or server of a primary datacenter; and

the one or more secondary targets are one or more secondary topics of the messaging system located on one or more brokers or servers of one or more secondary datacenters different from the primary datacenter.

15. A non-transitory machine readable storage medium having instructions stored thereon that when executed by one or more processors cause the one or more processors to perform a method comprising:

processing input events to generate primary output events and secondary output events, wherein the primary output events are generated by a primary server of a data processing system and the secondary output events are generated by one or more secondary servers of the data processing system;

writing the primary output events to a primary target and the secondary output events to one or more secondary targets, wherein the primary output events are written by the primary server and the secondary output events are written by the one or more secondary servers, and wherein the processing the input events and the writing the primary output events to the primary target and the secondary output events to the one or more secondary targets are performed at a same time by the primary server and the one or more secondary servers;

receiving an election of a new primary server from a synchronization system upon a failure of the primary server, wherein the new primary server is elected from the one or more secondary servers;

reading, by the new primary server, the secondary output events from the one or more secondary targets;

determining, by the new primary server, the primary output events that failed to be written to the primary target because of the failure of the primary server; and

writing, by the new primary server, the failed primary output events to the primary target using the secondary output events read from the one or more secondary targets.

16. The non-transitory machine readable storage medium of claim 15 , wherein the method further comprises:

reading, at the primary server, the input events from an input source;

reading, at the one or more secondary servers, the input events from the input source;

pausing, by the new primary server, the processing the input events and the writing of the secondary output events for at least the new primary server upon the election of the new primary server; and

unpausing, by the new primary server, the processing the input events and the writing of the secondary output events for at least the new primary server upon the writing the failed primary output events to the primary target.

17. The non-transitory machine readable storage medium of claim 16 , wherein the method further comprises:

reading, at the new primary server, the input events from the input source upon the unpausing;

processing, at the new primary server, the input events to generate the primary output events; and

writing, by the new primary server, the primary output events to the primary target.

18. The non-transitory machine readable storage medium of claim 15 , wherein the method further comprises:

writing, by the primary server of the data processing system, primary output information to the synchronization system upon writing the primary output events to the primary target; and

writing, by the one or more secondary servers of the data processing system, secondary output information to the synchronization system upon writing the secondary output events to the one or more secondary targets.

19. The non-transitory machine readable storage medium of claim 18 , wherein the primary output events that failed to be written to the primary target are determined based on the primary output information and the secondary information written to the synchronization system, and wherein the primary output information and the secondary output information written include an identifier of the primary target and identifiers of the one or more secondary targets where the primary output events and the secondary output events are being written, respectively, and one or more indices to identify various levels of the primary output events and the secondary output events.

20. The non-transitory machine readable storage medium of claim 19 , wherein the method further comprises:

reading, at the primary server of a data processing system, the input events from an input source; and

reading, at the one or more secondary servers of the data processing system, the input events from the input source,

wherein:

the input source is a topic of a messaging system;

the primary target is a primary output topic of the messaging system located on a broker or server of a primary datacenter; and

the one or more secondary targets are one or more secondary topics of the messaging system located on one or more brokers or servers of one or more secondary datacenters different from the primary datacenter.