Accurate global eventual counting
In various embodiments, a computer-implemented method comprises receiving, from a first endpoint device, a first event during a first time period, modifying an event log to include a record associated with the first event, causing a rollup queue to include a request to count a first count value associated with the first event, and generating, in a second time period subsequent to the first time period, a counter value associated with at least the first event based on the rollup queue and the event log.
1 . A computer-implemented method, comprising:
receiving a first event;
modifying an event log to include a record associated with the first event;
adding, based on the first event, a request to a queue; and
generating a counter value associated with at least the first event based on the queue and the event log.
2 . The computer-implemented method of claim 1 , wherein:
the event log includes a set of immutable events;
a first partition of the event log is not appended after a threshold time period; and
generating the counter value includes aggregating a set of count values in the first partition after the threshold time period.
3 . The computer-implemented method of claim 2 , wherein each record included in the event log includes a globally-unique idempotency token that includes (i) a unique identifier, and (ii) a timestamp.
4 . The computer-implemented method of claim 2 , further comprising:
determining that the threshold time period has elapsed;
deduplicating, based on a set of idempotency tokens, a set of records included in a first partition of the event log to generate a deduplicated set of records, wherein the set of records includes the record; and
generating an updated count value based on the deduplicated set of records, wherein the updated count value is included in the counter value.
5 . The computer-implemented method of claim 1 , further comprising:
receiving a second message that includes the first event;
appending a second record associated with the first event to the event log, wherein the record and the second record include a globally-unique token; and
deduplicating the record and the second record based on the globally-unique token.
6 . The computer-implemented method of claim 1 , further comprising:
receiving a request message;
determining that the request message includes a request for the first event;
retrieving, from the event log, the record; and
transmitting, to an endpoint device associated with the request message, a response that includes a first count value included in the queue and retrieved from the record.
7 . The computer-implemented method of claim 1 , further comprising:
receiving a request message;
determining that the request message includes a request for the counter value;
retrieving the counter value; and
transmitting, to an endpoint device associated with the request message, a response that includes the counter value.
8 . The computer-implemented method of claim 7 , wherein the counter value is stored in at least one of (i) an event cache, or (ii) a rollup table.
9 . The computer-implemented method of claim 7 , wherein retrieving the counter value overlaps with writing the record to the event log.
10 . The computer-implemented method of claim 1 , wherein a first count value included in the queue comprises one of (i) a negative value, or (ii) a value of 2 or higher.
11 . One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving, from a first endpoint device, a first event during a first time period;
modifying an event log to include a record associated with the first event;
causing a rollup queue to include a request to count a first count value associated with the first event;
generating, in a second time period subsequent to the first time period, a counter value associated with at least the first event based on the rollup queue and the event log;
determining that a threshold time period has elapsed;
upon determining that the threshold time period has elapsed, deduplicating a set of records included in a first partition of the event log to generate a deduplicated set of records, wherein:
the deduplicated set of records includes the record, and
the deduplicating is based on a set of idempotency tokens; and
generating an updated count value based on the deduplicated set of records, wherein the updated count vale is included in the counter value.
12 . The one or more non-transitory computer-readable storage media of claim 11 , wherein each record included in the event log includes an idempotency token that includes (i) a unique identifier, and (ii) a timestamp.
13 . The one or more non-transitory computer-readable storage media of claim 11 , further storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the steps of:
determining that the request to count a first count value associated with the first event is included in the rollup queue;
aggregating, based on the request included in the rollup queue, a set of records included in the event log to generate a count value update, wherein the set of records includes the first count value; and
updating the counter value based on the count value update, wherein the counter value is stored in an event cache.
14 . The one or more non-transitory computer-readable storage media of claim 13 , further storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the steps of:
generating a rollup count entry that includes the count value update and a rollup timestamp; and
updating a rollup table to include the rollup count entry and the rollup timestamp.
15 . The one or more non-transitory computer-readable storage media of claim 11 , wherein the first count value comprises one of (i) a negative value, or (ii) a value of 2 or higher.
16 . The one or more non-transitory computer-readable storage media of claim 11 , further comprising instructions that cause the one or more processors to further perform the steps of:
receiving, by an Internet exchange (IX) point in a first region from a set of endpoint devices, a set of events made by a plurality of users, wherein the set of events includes the first event;
appending a set of records associated with the set of events to a local event log; and
replicating the set of events included in the local event log by appending the set of events to the event log.
17 . A system comprising:
a memory storing a global event counter application; and
a processor coupled to the memory that executes the global event counter application by performing the steps of:
receiving, from a first endpoint device, a first event during a first time period;
modifying an event log to include a record associated with the first event;
causing a rollup queue to include a request to count a first count value associated with the first event;
generating, in a second time period subsequent to the first time period, a counter value associated with at least the first event based on the rollup queue and the event log;
determining that the request to count a first count value associated with the first event is included in the rollup queue;
aggregating, based on the request included in the rollup queue, a set of records included in the event log to generate a count value update, wherein the set of records includes the first count value; and
updating the counter value based on the count value update, wherein the counter value is stored in an event cache.
18 . The system of claim 17 , wherein the processor further executes the global event counter application to perform the steps of:
determining that a threshold time period has elapsed;
upon determining that the threshold time period has elapsed, deduplicating a set of records included in a first partition of the event log to generate a deduplicated set of records, wherein:
the deduplicated set of records includes the record,
each record included in the event log includes an idempotency token that includes (i) a unique identifier, and (ii) a timestamp, and
the deduplicating is based on a set of idempotency tokens; and
generating an updated count value based on the deduplicated set of records, wherein the updated count vale is included in the counter value.
19 . The system of claim 17 , wherein each record included in the event log includes an idempotency token that includes (i) a unique identifier, and (ii) a timestamp.
20 . The system of claim 17 , wherein the processor further executes the global event counter application to perform the steps of:
receiving, from the first endpoint device, a request message;
determining that the request message includes a request for the counter value;
retrieving the counter value; and
transmitting, to the first endpoint device, a response that includes the counter value.