System for transforming data into table format
Systems and methods are directed to transforming data into a distributed table format. The system accesses new data for a topic. One or more materializers generate data files from the new data, whereby each of the materializers can process a specific range of offsets within a partition of the topic. The materializers also generate materializer committables corresponding to the data files. A maintainer performs one or more maintenance operations on previously committed files and generates a maintainer committable for each maintenance operation performed. The maintenance operations can include merging previously committed smaller files into a larger file and deleting obsolete files. Subsequently, a committer scheduler schedules the maintainer and materializer committables for a commit process. Scheduling includes prioritizing which committables are applied first and ordering materializer committables based on their offsets. The commit process is performed based on the scheduling resulting in a single snapshot of the table.
1 . A method for transforming data into a distributed table format for a table, the method comprising:
accessing new data for a topic;
generating, by one or more materializers, data files based on the new data;
generating, by the one or more materializers, materializer committables corresponding to the data files, each materializer committable comprising a metadata object describing one or more corresponding data files;
performing, by a maintainer, one or more maintenance operations on previously committed files;
generating, by the maintainer, a maintainer committable for each maintenance operation performed, each maintainer committable comprising a metadata object describing the corresponding maintenance operation;
scheduling, by a committer scheduler, the maintainer and materializer committables for a commit process; and
performing the commit process based on the scheduling, the commit process committing the maintainer and materializer committables to the distributed table format.
2 . The method of claim 1 , wherein the generating the data files comprises processing a specific offset range within a partition of the topic.
3 . The method of claim 1 , wherein the performing the one or more maintenance operations comprises merging smaller previously committed files into a larger file or deleting obsolete previously committed files.
4 . The method of claim 1 , further comprising:
performing a pre-commit merge operation that merges a plurality of contiguous materializer committables into a single materializer committable.
5 . The method of claim 1 , further comprising:
performing a pre-commit merge operation that squashing updates for a same key and retains only a latest value for the key.
6 . The method of claim 1 , wherein the scheduling the maintainer and materializer committables comprises prioritizing the maintainer committables over the materializer committables.
7 . The method of claim 1 , wherein the scheduling the maintainer and materializer committables comprises ordering the materializer committables based on offsets associated with partitions of the topic.
8 . The method of claim 1 , wherein the committer scheduler processes a subset of topics in a given cloud region and priorities topics based on configurable properties.
9 . The method of claim 8 , wherein the configurable properties comprise a freshness property that indicates how soon published data should appear in the table, the committer scheduler prioritizing commits to topics with exceeded or soon to be exceeded target freshness.
10 . The method of claim 1 , wherein the performing the commit process comprises generating a snapshot of a state of the table.
11 . The method of claim 1 , wherein the data files are Parquet™ files and the table is an Iceberg™ table.
12 . A system for transforming data into a distributed table format for a table, the system comprising:
one or more hardware processors; and
a memory storing instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations comprising:
accessing new data for a topic;
generating data files based on the new data;
generating materializer committables corresponding to the data files, each materializer committable comprising a metadata object describing one or more corresponding data files;
performing one or more maintenance operations on previously committed files;
generating a maintainer committable for each maintenance operation performed, each maintainer committable comprising a metadata object describing the corresponding maintenance operation;
scheduling the maintainer and materializer committables for a commit process; and
performing the commit process based on the scheduling, the commit process committing the maintainer and materializer committables to the distributed table format.
13 . The system of claim 12 , wherein the generating the data files comprises processing a specific offset range within a partition of the topic.
14 . The system of claim 12 , wherein the performing the one or more maintenance operations comprises merging smaller previously committed files into a larger file or deleting obsolete previously committed files.
15 . The system of claim 12 , wherein the operations further comprise:
performing a pre-commit merge operation that merges a plurality of contiguous materializer committables into a single materializer committable.
16 . The system of claim 12 , wherein the operations further comprise:
performing a pre-commit merge operation that squashing updates for a same key and retains only a latest value for the key.
17 . The system of claim 12 , wherein the scheduling the maintainer and materializer committables comprises prioritizing the maintainer committables over the materializer committables.
18 . The system of claim 12 , wherein the scheduling the maintainer and materializer committables comprises ordering the materializer committables based on offsets associated with partitions of the topic.
19 . The system of claim 12 , wherein the system processes a subset of topics in a given cloud region and priorities topics based on a freshness property that indicates how soon published data should appear in the table.
20 . A machine-storage medium comprising instructions which, when executed by one or more processors of a machine, cause the machine to perform operations for transforming data into a distributed table format for a table, the operations comprising,
accessing new data for a topic;
generating data files based on the new data;
generating materializer committables corresponding to the data files, each materializer committable comprising a metadata object describing one or more corresponding data files;
performing one or more maintenance operations on previously committed files;
generating a maintainer committable for each maintenance operation performed, each maintainer committable comprising a metadata object describing the corresponding maintenance operation;
scheduling the maintainer and materializer committables for a commit process; and
performing the commit process based on the scheduling, the commit process committing the maintainer and materializer committables to the distributed table format.