Reconciliation systems and methods for unbounded streams
A computing system is configured to (i) inject, into an unbounded stream of events produced by the computing system, a window open sentinel that indicates one or more reconciliation rules for the window, (ii) inject, into the unbounded stream of events, a shard open sentinel that indicates one or more reconciliation rules for a shard, wherein the shard is within the window, (iii) transmit a set of events produced within the shard, (iv) execute the reconciliation rules for the shard to thereby generate one or more reconciliation attributes for the shard, (v) inject, into the unbounded stream of events, a shard close sentinel that indicates the reconciliation attributes for the shard, (vi) execute the reconciliation rules for the window to thereby generate one or more reconciliation attributes for the window, and (vii) inject, into the unbounded stream of events, a window close sentinel that indicates the reconciliation attributes for the window.
1 . A computing system comprising:
a network interface for communicating over at least one data network;
at least one processor;
at least one non-transitory computer-readable medium; and
program instructions stored on the at least one non-transitory computer-readable medium that, when executed by the at least one processor, cause the computing system to:
determine a start of a window in an unbounded stream of events produced by the computing system;
based on determining the start of the window, inject, into the unbounded stream of events, a window open sentinel that indicates one or more reconciliation rules for the window;
transmit a set of events produced within the window;
execute the one or more reconciliation rules for the window for the set of events produced within the window to thereby generate one or more reconciliation attributes for the window;
determine a close of the window in the unbounded stream of events; and
based on determining the close of the window, inject, into the unbounded stream of events, a window close sentinel that indicates the one or more reconciliation attributes for the window.
2 . The computing system of claim 1 , further comprising program instructions stored on the at least one non-transitory computer-readable medium that, when executed by the at least one processor, cause the computing system to:
receive, from a consumer computing system, an indication that the window was unsuccessfully reconciled; and
based on receiving the indication, retransmit the set of events produced within the window.
3 . The computing system of claim 1 , wherein the one or more reconciliation rules for the window comprise instructions for determining (i) a given number of events produced within the window and (ii) a signature for each event produced within the window.
4 . The computing system of claim 1 , wherein the one or more reconciliation rules for the window comprise instructions for determining one or more of (i) a summation of a given event attribute across all events produced within the window, (ii) a number of a given type of event produced within the window, or (iii) a signature for each of a given type of event produced within the window.
5 . The computing system of claim 1 , wherein the program instructions that, when executed by the at least one processor, cause the computing system to execute the one or more reconciliation rules for the window for the set of events produced within the window comprise program instructions that, when executed by the at least one processor, cause the computing system to execute the one or more reconciliation rules for the window after all events in the set of events are produced within the window.
6 . The computing system of claim 1 , wherein the program instructions that, when executed by the at least one processor, cause the computing system to execute the one or more reconciliation rules for the window for the set of events produced within the window comprise program instructions that, when executed by the at least one processor, cause the computing system to execute the one or more reconciliation rules for the window after each event in the set of events is produced within the window.
7 . The computing system of claim 1 , further comprising program instructions stored on the at least one non-transitory computer-readable medium that, when executed by the at least one processor, cause the computing system to:
determine the one or more reconciliation rules for the window.
8 . The computing system of claim 1 , further comprising program instructions stored on the at least one non-transitory computer-readable medium that, when executed by the at least one processor, cause the computing system to:
post the one or more reconciliation attributes to a secure registry; and
wherein the window close sentinel indicates the one or more reconciliation attributes by including, in the window close sentinel, information for accessing the one or more reconciliation attributes from the secure registry.
9 . The computing system of claim 1 , wherein the program instructions that, when executed by the at least one processor, cause the computing system to determine the close of the window in the unbounded stream of events comprise program instructions that, when executed by the at least one processor, cause the computing system to determine that a prescribed number of events has been produced within the window.
10 . A non-transitory computer-readable medium, wherein the non-transitory computer-readable medium is provisioned with program instructions that, when executed by at least one processor, cause a computing system to:
determine a start of a window in an unbounded stream of events produced by the computing system;
based on determining the start of the window, inject, into the unbounded stream of events, a window open sentinel that indicates one or more reconciliation rules for the window;
transmit a set of events produced within the window;
execute the one or more reconciliation rules for the window for the set of events produced within the window to thereby generate one or more reconciliation attributes for the window;
determine a close of the window in the unbounded stream of events; and
based on determining the close of the window, inject, into the unbounded stream of events, a window close sentinel that indicates the one or more reconciliation attributes for the window.
11 . The non-transitory computer-readable medium of claim 10 , wherein the non-transitory computer-readable medium is further provisioned with program instructions that, when executed by the at least one processor, cause the computing system to:
receive, from a consumer computing system, an indication that the window was unsuccessfully reconciled; and
based on receiving the indication, retransmit the set of events produced within the window.
12 . The non-transitory computer-readable medium of claim 10 , wherein the one or more reconciliation rules for the window comprise instructions for determining (i) a given number of events produced within the window and (ii) a signature for each event produced within the window.
13 . The non-transitory computer-readable medium of claim 10 , wherein the one or more reconciliation rules for the window comprise instructions for determining one or more of (i) a summation of a given event attribute across all events produced within the window, (ii) a number of a given type of event produced within the window, or (iii) a signature for each of a given type of event produced within the window.
14 . The non-transitory computer-readable medium of claim 10 , wherein the program instructions that, when executed by the at least one processor, cause the computing system to execute the one or more reconciliation rules for the window for the set of events produced within the window comprise program instructions that, when executed by the at least one processor, cause the computing system to execute the one or more reconciliation rules for the window after all events in the set of events are produced within the window.
15 . The non-transitory computer-readable medium of claim 10 , wherein the program instructions that, when executed by the at least one processor, cause the computing system to execute the one or more reconciliation rules for the window for the set of events produced within the window comprise program instructions that, when executed by the at least one processor, cause the computing system to execute the one or more reconciliation rules for the window after each event in the set of events is produced within the window.
16 . The non-transitory computer-readable medium of claim 10 , wherein the non-transitory computer-readable medium is further provisioned with program instructions that, when executed by the at least one processor, cause the computing system to:
determine the one or more reconciliation rules for the window.
17 . The non-transitory computer-readable medium of claim 10 , wherein the non-transitory computer-readable medium is further provisioned with program instructions that, when executed by the at least one processor, cause the computing system to:
post the one or more reconciliation attributes to a secure registry; and
wherein the window close sentinel indicates the one or more reconciliation attributes by including, in the window close sentinel, information for accessing the one or more reconciliation attributes from the secure registry.
18 . The non-transitory computer-readable medium of claim 10 , wherein the program instructions that, when executed by the at least one processor, cause the computing system to determine the close of the window in the unbounded stream of events comprise program instructions that, when executed by the at least one processor, cause the computing system to determine that a prescribed number of events has been produced within the window.
19 . A method carried out by a computing system, the method comprising:
determining a start of a window in an unbounded stream of events produced by the computing system;
based on determining the start of the window, injecting, into the unbounded stream of events, a window open sentinel that indicates one or more reconciliation rules for the window;
transmitting a set of events produced within the window;
executing the one or more reconciliation rules for the window for the set of events produced within the window to thereby generate one or more reconciliation attributes for the window;
determining a close of the window in the unbounded stream of events; and
based on determining the close of the window, injecting, into the unbounded stream of events, a window close sentinel that indicates the one or more reconciliation attributes for the window.
20 . The method of claim 19 , further comprising:
receiving, from a consumer computing system, an indication that the window was unsuccessfully reconciled; and
based on receiving the indication, retransmitting the set of events produced within the window.