Processing streaming data in a transaction-based distributed database system
Techniques for implementing a streaming transaction in a transaction based data storage system is disclosed. In an embodiment, a server computer system stores a dataset comprising a plurality of files where the dataset comprises a corresponding version number. The server computer system receives a command to modify the dataset and, in response, starts a streaming transaction for the dataset. During the streaming transaction, the server computer system receives a plurality of updates to the dataset, wherein each of the plurality of updates modifies one or more files of the plurality of files. The server computer system executes the plurality of updates in the dataset without modifying the corresponding version number of the dataset. When the server computer system commits the streaming transaction to the dataset, the server computer system increments the corresponding version number of the dataset.
1. A computer-implemented method comprising:
storing a dataset comprising a plurality of files, wherein the dataset comprises a corresponding version number;
receiving a command to modify the dataset;
in response to receiving the command to modify the dataset, starting a streaming transaction for the dataset;
during the streaming transaction, receiving a plurality of updates to the dataset, wherein each update of the plurality of updates modifies one or more files of the plurality of files, wherein the plurality of updates comprises a first update to the dataset and a second update to the dataset;
during the streaming transaction, executing, in the dataset, each update of the plurality of updates as it is received without modifying the corresponding version number of the dataset;
during the streaming transaction, receiving a first update that adds a portion of a particular file;
storing the portion of the particular file separately from the dataset;
after receiving the first update, but before receiving the second update, receiving a query to the dataset at a query time;
identifying a state of the dataset at the query time, wherein the state of the dataset includes one or more updates to the dataset of the plurality of updates to the dataset, but does not include updates based on the portion of the particular file;
responding to the query using the state of the dataset at the query time
receiving the second update that adds a remaining portion of the particular file;
in response to receiving the remaining portion of the particular file, updating the dataset using the particular file;
committing the streaming transaction to the dataset, wherein committing the streaming transaction comprises incrementing the corresponding version number of the dataset;
wherein the method is performed using one or more processors.
2. The method of claim 1 , further comprising:
in response to committing the streaming transaction to the dataset, aggregating the plurality of updates to the dataset into one or more update files;
storing the one or more update files with the dataset.
3. The method of claim 1 , further comprising:
after receiving the second update, but before committing the streaming transaction to the dataset, receiving a second query to the dataset at a second query time;
identifying a second state of the dataset at the second query time, wherein the second state of the dataset includes updates based on the particular file;
responding to the second query using the second state of the dataset at the second query time.
4. The method of claim 1 , further comprising:
in response to receiving the command to modify the dataset, identifying a second dataset that is dependent on the dataset;
starting a second streaming transaction for the second dataset;
using the plurality of updates to the dataset, generating a second plurality of updates for the second dataset;
executing, in the second dataset, each update of the second plurality of updates without modifying a second corresponding version number of the second dataset;
committing the second streaming transaction to the second dataset, wherein committing the second streaming transaction comprises incrementing the second corresponding version number of the second dataset.
5. A system comprising:
one or more processors;
a memory storing instructions which, when executed by the one or more processors, cause performing:
storing a dataset comprising a plurality of files, wherein the dataset comprises a corresponding version number;
receiving a command to modify the dataset;
in response to receiving the command to modify the dataset, starting a streaming transaction for the dataset;
during the streaming transaction, receiving a plurality of updates to the dataset, wherein each update of the plurality of updates modifies one or more files of the plurality of files, wherein the plurality of updates comprises a first update to the dataset and a second update to the dataset;
during the streaming transaction, executing, in the dataset, each update of the plurality of updates as it is received without modifying the corresponding version number of the dataset;
during the streaming transaction, receiving a first update that adds a portion of a particular file;
storing the portion of the particular file separately from the dataset;
after receiving the first update, but before receiving the second update, receiving a query to the dataset at a query time;
identifying a state of the dataset at the query time, wherein the state of the dataset includes one or more updates to the dataset of the plurality of updates to the dataset, but does not include updates based on the portion of the particular file;
responding to the query using the state of the dataset at the query time
receiving the second update that adds a remaining portion of the particular file;
in response to receiving the remaining portion of the particular file, updating the dataset using the particular file;
committing the streaming transaction to the dataset, wherein committing the streaming transaction comprises incrementing the corresponding version number of the dataset.
6. The system of claim 5 , wherein the instructions, when executed by the one or more processors, further cause performance of:
in response to committing the streaming transaction to the dataset, aggregating the plurality of updates to the dataset into one or more update files;
storing the one or more update files with the dataset.
7. The system of claim 5 , wherein the instructions, when executed by the one or more processors, further cause performance of:
after receiving the second update, but before committing the streaming transaction to the dataset, receiving a second query to the dataset at a second query time;
identifying a second state of the dataset at the second query time, wherein the second state of the dataset includes updates based on the particular file;
responding to the second query using the second state of the dataset at the second query time.
8. The system of claim 5 , wherein the instructions, when executed by the one or more processors, further cause performance of:
in response to receiving the command to modify the dataset, identifying a second dataset that is dependent on the dataset;
starting a second streaming transaction for the second dataset;
using the plurality of updates to the dataset, generating a second plurality of updates for the second dataset;
executing, in the second dataset, each update of the second plurality of updates without modifying a second corresponding version number of the second dataset;
committing the second streaming transaction to the second dataset, wherein committing the second streaming transaction comprises incrementing the second corresponding version number of the second dataset.
9. One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause performing:
storing a dataset comprising a plurality of files, wherein the dataset comprises a corresponding version number;
receiving a command to modify the dataset;
in response to receiving the command to modify the dataset, starting a streaming transaction for the dataset;
during the streaming transaction, receiving a plurality of updates to the dataset, wherein each update of the plurality of updates modifies one or more files of the plurality of files, wherein the plurality of updates comprises a first update to the dataset and a second update to the dataset;
during the streaming transaction, executing, in the dataset, each update of the plurality of updates as it is received without modifying the corresponding version number of the dataset;
during the streaming transaction, receiving a first update that adds a portion of a particular file;
storing the portion of the particular file separately from the dataset;
after receiving the first update, but before receiving the second update, receiving a query to the dataset at a query time;
identifying a state of the dataset at the query time, wherein the state of the dataset includes one or more updates to the dataset of the plurality of updates to the dataset, but does not include updates based on the portion of the particular file;
responding to the query using the state of the dataset at the query time
receiving the second update that adds a remaining portion of the particular file;
in response to receiving the remaining portion of the particular file, updating the dataset using the particular file;
committing the streaming transaction to the dataset, wherein committing the streaming transaction comprises incrementing the corresponding version number of the dataset.
10. The one or more non-transitory computer-readable media of claim 9 , wherein the instructions, when executed by the one or more processors, further cause performance of:
in response to committing the streaming transaction to the dataset, aggregating the plurality of updates to the dataset into one or more update files;
storing the one or more update files with the dataset.
11. The one or more non-transitory computer-readable media of claim 9 , wherein the instructions, when executed by the one or more processors, further cause performance of:
after receiving the second update, but before committing the streaming transaction to the dataset, receiving a second query to the dataset at a second query time;
identifying a second state of the dataset at the second query time, wherein the second state of the dataset includes updates based on the particular file;
responding to the second query using the second state of the dataset at the second query time.