Mutable data ingestion and storage
Systems and techniques are provided for mutable data ingestion and storage. Data for a database may be received at a computing device. New data from the data for the database may be stored as a mutable data chunk. The data for the database may be determined to include updated data for a previously stored mutable data chunk. The updated data may be applied to the previously stored mutable data chunk to update the previously stored mutable chunk. A second previously stored mutable data chunk may be determined to have been initially stored a specified amount of time ago. In response to determining that the second previously stored mutable data chunk was initially stored the specified amount of time ago, the second previously stored mutable data chunk may be caused to become an immutable data chunk.
1. A computer-implemented method performed by a data processing apparatus, the method comprising:
performing a merge operation on two or more mutable data chunks of a database to generate merged mutable data;
performing a union operation on two or more immutable data chunks of the database to generate unioned immutable data;
performing a union operation on the merged mutable data and the unioned immutable data to generate a materialized database; and
making the materialized database available to be queried.
2. The method of claim 1 , wherein the two or more immutable data chunks are not updatable with any additional data for the database received at the computing device.
3. The method of claim 1 , wherein the two or more mutable data chunks are updateable with additional data for the database received at the computing device.
4. The method of claim 1 , wherein the materialized database comprises a database table comprising records from one of the two or more mutable data chunks and one of the two or more immutable data chunks.
5. The method of claim 1 , wherein the materialized database comprises records of the database that are current as of the time the merge operation is performed on the two or more mutable data chunks.
6. The method of claim 1 , wherein the materialized database comprises schema for database tables formed from data in the two or mutable data chunks and the two or more immutable data chunks.
7. The method of claim 1 , wherein the two or more immutable data chunks were previously stored as mutable data chunks that were caused to become immutable data chunks based on a determination that the mutable data chunks had been stored as mutable data chunks for at least a specified amount of time.
8. A computer-implemented system comprising:
a storage, and
a processor that performs a merge operation on two or more mutable data chunks of a database to generate merged mutable data,
performs a union operation on two or more immutable data chunks of the database to generate unioned immutable data,
performs a union operation on the merged mutable data and the unioned immutable data to generate a materialized database, and
makes the materialized database available to be queried.
9. The system of claim 8 , wherein the two or more immutable data chunks are not updatable with any additional data for the database received at the computing device.
10. The system of claim 8 , wherein the two or more mutable data chunks are updateable with additional data for the database received at the computing device.
11. The system of claim 8 , wherein the materialized database comprises a database table comprising records from one of the two or more mutable data chunks and one of the two or more immutable data chunks.
12. The system of claim 8 , wherein the materialized database comprises records of the database that are current as of the time the merge operation is performed on the two or more mutable data chunks.
13. The system of claim 8 , wherein the materialized database comprises schema for database tables formed from data in the two or mutable data chunks and the two or more immutable data chunks.
14. The system of claim 8 , wherein the two or more immutable data chunks were previously stored as mutable data chunks that were caused to become immutable data chunks based on a determination that the mutable data chunks had been stored as mutable data chunks for at least a specified amount of time.
15. A system comprising: one or more computers and one or more storage devices storing instructions which are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
comprising:
performing a merge operation on two or more mutable data chunks of a database to generate merged mutable data;
performing a union operation on two or more immutable data chunks of the database to generate unioned immutable data;
performing a union operation on the merged mutable data and the unioned immutable data to generate a materialized database; and
making the materialized database available to be queried.
16. The system of claim 15 , wherein the two or more immutable data chunks are not updatable with any additional data for the database received at the computing device.
17. The system of claim 15 , wherein the two or more mutable data chunks are updateable with additional data for the database received at the computing device.
18. The system of claim 15 , wherein the materialized database comprises a database table comprising records from one of the two or more mutable data chunks and one of the two or more immutable data chunks.
19. The system of claim 15 , wherein the materialized database comprises records of the database that are current as of the time the merge operation is performed on the two or more mutable data chunks.
20. The system of claim 15 , wherein the materialized database comprises schema for database tables formed from data in the two or mutable data chunks and the two or more immutable data chunks.