Methods, systems, and products for maintaining data consistency in a stream warehouse
Methods, systems, and products characterize consistency of data in a stream warehouse. A warehouse table is derived from a continuously received a stream of data. The warehouse table is stored in memory as a plurality of temporal partitions, with each temporal partition storing data within a contiguous range of time. A level of consistency is assigned to each temporal partition in the warehouse table.
1. A method, comprising:
receiving, by a server, a stream of data associated with a data warehouse;
partitioning, by the server, the stream of data into a plurality of temporal partitions according to a partitioning function, with each temporal partition of the plurality of temporal partitions assigned an integer, and the each temporal partition capable of containing a data record obtained from the stream of data;
assigning, by the server, an open consistency to a single partition of the plurality of temporal partitions, the open consistency assigned in response to determining that a later-received partition of the plurality of temporal partitions contains the data record;
assigning, by the server, a closed consistency to the single partition in response to determining no more data records are expected within the single partition;
assigning, by the server, a complete consistency to the single partition in response to determining all expected data records have arrived within the single partition;
assigning, by the server, a query consistency to the single partition based on at least one of the open consistency, the closed consistency, and the complete consistency; and
determining, by the server, an answer to a query based on the query consistency.
2. The method of claim 1 , further comprising receiving the query.
3. The method of claim 1 , further comprising marking the single partition with the query consistency.
4. A system, comprising:
a hardware processor; and
a memory device, the memory device storing code, the code when executed causing the hardware processor to perform operations, the operations comprising:
receiving a stream of data associated with a data warehouse;
partitioning the stream of data into a plurality of temporal partitions according to a partitioning function, with each temporal partition of the plurality of temporal partitions identified by an integer, and the each temporal partition capable of containing a data record obtained from the stream of data;
assigning an open consistency to a single partition of the plurality of temporal partitions, the open consistency assigned in response to determining that a later-received partition of the plurality of temporal partitions contains the data record;
assigning a closed consistency to the single partition in response to determining no more data records are expected within the single partition;
assigning a complete consistency to the single partition in response to determining all expected data records have arrived within the single partition;
assigning a query consistency to the single partition based on at least one of the open consistency, the closed consistency, and the complete consistency; and
determining an answer to a query based on the query consistency.
5. The system of claim 4 , wherein the operations further comprise receiving the query.
6. The system of claim 4 , wherein the operations further comprise marking the single partition with the query consistency.
7. A memory device storing instructions that when executed cause a hardware processor to perform operations, the operations comprising:
partitioning the stream of data into a plurality of temporal partitions according to a partitioning function, with each temporal partition of the plurality of temporal partitions identified by a partition integer value, and the each temporal partition capable of containing a data record obtained from the stream of data;
assigning an open consistency to a single partition of the plurality of temporal partitions, the open consistency assigned in response to determining that a later-received partition of the plurality of temporal partitions contains the data record;
assigning a closed consistency to the single partition in response to determining no more data records are expected within the single partition;
assigning a complete consistency to the single partition in response to determining all expected data records have arrived within the single partition;
assigning a query consistency to the single partition based on at least one of the open consistency, the closed consistency, and the complete consistency; and
determining an answer to a query based on the query consistency.
8. The memory device of claim 7 , wherein the operations further comprise receiving the query.