STREAMING EVENT DATA TO DIFFERENT CONSUMER INSTANCES OF ONE EVENT CONSUMER DURING FAILURE OF A CONSUMER INSTANCE’S ABILITY TO PROCESS THE EVENT DATA
Some embodiments provide a novel method for providing data regarding events that is stored in a data store to a set of consumers. An event server provides, to a first instance of a particular consumer that includes several instances, a stream of data regarding a first event that is stored in a set of one or more partitions of the data store assigned to the first instance. The event server receives, from the first instance, notification that the first instance failed to process the stream of data. The event server unassigns the set of partitions to the first instance. The event server assigns the set of partitions to a second instance of the particular consumer. The event server provides, to the second instance, the stream of data regarding the first event.
1 . A method for providing data regarding events that is stored in a data store to a set of consumers, the method comprising:
at an event server:
providing, to a first instance of a particular consumer that comprises a plurality of instances, a stream of data regarding a first event that is stored in a set of one or more partitions of the data store assigned to the first instance;
receiving, from the first instance, notification that the first instance failed to process the stream of data;
unassigning the set of partitions to the first instance;
assigning the set of partitions to a second instance of the particular consumer; and
providing, to the second instance, the stream of data regarding the first event.
2 . The method of claim 1 further comprising:
before providing to the first instance the stream of data regarding the first event:
receiving, from the first instance, a registration to receive data for the first event;
using an identifier of the first instance to identify the set of partitions of the data store assigned to the first instance; and
retrieving the stream of data from the set of partitions.
3 . The method of claim 2 , wherein:
the event server maintains a cache comprising mappings between each of the plurality of instances of the particular consumer and different sets of one or more partitions of the data store, and
unassigning the set of partitions to the first instance comprises updating a first entry stored in the cache mapping the first instance to the set of partitions into a second entry that does not map the first instance to any partitions in the data store.
4 . The method of claim 3 , wherein the stream if data is a first stream of data and the set of partitions is a first set of partitions, and assigning the set of partitions to the second instance comprises:
after unassigning the first set of partitions to the first instance:
receiving, from the second instance, notification that the second instance has processed a second stream of data regarding the first event that is stored in a second set of one or more partitions of the data store; and
updating a third entry stored in the cache mapping the second instance to the second set of partitions into a fourth entry mapping the second instance to the first set of partitions.
5 . The method of claim 1 further comprising:
receiving, from the second instance, notification that the second instance failed to process the stream of data; and
resending the stream of data to the second instance.
6 . The method of claim 5 further comprising:
after receiving the notification that the second instance failed to process the stream of data and before resending the stream of data to the second instance:
unassigning the set of partitions to the second instance;
receiving, from the second instance, a registration to receive data for the first event; and
reassigning the set of partitions to the second instance.
7 . The method of claim 1 , wherein the set of partitions is a first set of partitions and the stream of data is a first stream of data, the method further comprising:
after unassigning the first set of partitions to the first instance:
receiving, from the first instance, a registration to receive data for the first event;
assigning a second set of partitions to the first instance that stores data regarding the first event; and
providing a second stream of data regarding the first event to the first instance.
8 . The method of claim 1 , wherein providing, to the second instance, the stream of data regarding the first event comprises iteratively providing data tuples that are stored continuously in the set of partitions regarding the first event.
9 . The method of claim 1 , wherein the data storage is a distributed data storage.
10 . The method of claim 9 , wherein the distributed data storage is a distributed database.
11 . The method of claim 1 , wherein at least two partitions of the data store are in two different geographic sites.
12 . The method of claim 1 , wherein the particular consumer resides in a first datacenter and the data store resides in a second datacenter.
13 . The method of claim 12 , wherein the first and second datacenters belong to a same cloud provider.
14 . The method of claim 12 , wherein the first and second datacenters belong to different cloud providers.
15 . The method of claim 1 , wherein the first instance resides in a first datacenter and the second instance resides in a second datacenter.
16 . The method of claim 1 , wherein the set of consumers has no knowledge of how the event server stores the data in the data store.
17 . A non-transitory machine readable medium storing a program for execution by at least one processing unit for providing data regarding events that is stored in a data store to a set of consumers, the program comprising sets of instructions for:
at the event server:
providing, to a first instance of a particular consumer that comprises a plurality of instances, a stream of data regarding a first event that is stored in a set of one or more partitions of the data store assigned to the first instance;
receiving, from the first instance, notification that the first instance failed to process the stream of data;
unassigning the set of partitions to the first instance;
assigning the set of partitions to a second instance of the particular consumer; and
providing, to the second instance, the stream of data regarding the first event.
18 . The non-transitory machine readable medium of claim 17 , wherein the program further comprises sets of instructions for:
before providing to the first instance the stream of data regarding the first event:
receiving, from the first instance, a registration to receive data for the first event;
using an identifier of the first instance to identify the set of partitions of the data store assigned to the first instance; and
retrieving the stream of data from the set of partitions.
19 . The non-transitory machine readable medium of claim 18 , wherein:
the event server maintains a cache comprising mappings between each of the plurality of instances of the particular consumer and different sets of one or more partitions of the data store, and
the set of instructions for unassigning the set of partitions to the first instance comprises a set of instructions for updating a first entry stored in the cache mapping the first instance to the set of partitions into a second entry that does not map the first instance to any partitions in the data store.
20 . The non-transitory machine readable medium of claim 19 , wherein the stream if data is a first stream of data and the set of partitions is a first set of partitions, and the set of instructions for assigning the set of partitions to the second instance comprises sets of instructions for:
after unassigning the first set of partitions to the first instance:
receiving, from the second instance, notification that the second instance has processed a second stream of data regarding the first event that is stored in a second set of one or more partitions of the data store; and
updating a third entry stored in the cache mapping the second instance to the second set of partitions into a fourth entry mapping the second instance to the first set of partitions.