Efficient merging of tabular data with post-processing compaction
A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, obtaining one or more other resulting files based at least in part on unmatched rows, and obtaining a set of processed files based at least in part on performing a post-processing operation with respect to the set of resulting files. The set of processed files has less files than the set of resulting files. Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and obtaining the second job resulting file(s).
1 . A system comprising:
one or more computer processors; and
one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the system to perform operations comprising:
merging a target table and a source table, wherein the source table has one or more records that are different from the target table, wherein the merging yields a set of resulting files;
determining that a subset of the set of resulting files satisfy a compaction criteria defined by a compaction policy by determining that a number of files in the subset of the set of resulting files is greater than a threshold number of files;
compacting the subset of the set of resulting files to obtain a set of compacted files, wherein the compacting reduces a number of files in the subset of the set of resulting files;
determining that a given file from the set of resulting files should not be compacted based on determining that a size of the given file is larger than a threshold size defined by the compaction policy; and
obtaining a resulting table based at least in part on the set of compacted files and the given file that is not compacted, the resulting table comprising a result of merging the target table and the source table.
2 . The system of claim 1 , wherein the resulting table is obtained based at least in part on the set of compacted files and one or more remaining files of the set of resulting files that were not subject to compaction.
3 . The system of claim 2 , wherein determining that the subset of the set of resulting files satisfy the compaction criteria comprises:
determining that an expected amount of time for combining the subset of the set of resulting files does not exceed a time threshold defined by the compaction policy.
4 . The system of claim 1 , wherein determining that the subset of the set of resulting files satisfy the compaction criteria comprises:
determining that a size of each file in the subset of the set of resulting files is less than the threshold size defined by the compaction policy.
5 . The system of claim 4 , wherein determining that the subset of the set of resulting files satisfy the compaction criteria further comprises:
determining that the subset of the set of resulting files are ordered adjacent to each other in the set of resulting files.
6 . The system of claim 4 , wherein determining that the subset of the set of resulting files satisfy the compaction criteria further comprises:
determining that a number of files in the set of resulting files exceeds a threshold number of files defined by the compaction policy.
7 . The system of claim 4 , wherein determining that the subset of the set of resulting files satisfy the compaction criteria further comprises:
determining that a merged size of the subset of resulting files does not exceed a threshold merged size defined by the compaction policy.
8 . The system of claim 7 , wherein determining that the subset of the set of resulting files satisfy the compaction criteria further comprises:
determining that a merged size of a first file and a second file from the set of resulting files is larger than the threshold merged size defined by the compaction policy; and
determining that the first file and the second file should not be compacted based on determining that the merged size of the first file and the second file from the set of resulting files is larger than the threshold merged size defined by the compaction policy.
9 . The system of claim 1 , wherein determining that the subset of the set of resulting files satisfy the compaction criteria comprises:
determining that a time of creation of each file within the subset of the set of resulting files falls within a threshold time window defined by the compaction policy.
10 . The system of claim 1 , wherein determining that the subset of the set of resulting files satisfy the compaction criteria comprises:
determining that each file within the subset of the set of resulting files belongs to a same partition of the set of resulting files.
11 . The system of claim 1 , wherein obtaining the resulting table comprises:
identifying a set of unmatched rows among the target table and the source table; and
obtaining the set of resulting files based at least in part on the unmatched rows among the target table and the source table.
12 . A method comprising:
merging a target table and a source table, wherein the source table has one or more records that are different from the target table, wherein the merging yields a set of resulting files;
determining that a subset of the set of resulting files satisfy a compaction criteria defined by a compaction policy by determining that a number of files in the subset of the set of resulting files is greater than a threshold number of files;
compacting the subset of the set of resulting files to obtain a set of compacted files, wherein the compacting reduces a number of files in the subset of the set of resulting files;
determining that a given file from the set of resulting files should not be compacted based on determining that a size of the given file is larger than a threshold size defined by the compaction policy; and
obtaining a resulting table based at least in part on the set of compacted files and the given file that is not compacted, the resulting table comprising a result of merging the target table and the source table.
13 . The method of claim 12 , wherein the resulting table is obtained based at least in part on the set of compacted files and one or more remaining files of the set of resulting files that were not subject to compaction.
14 . The method of claim 12 , wherein determining that the subset of the set of resulting files satisfy the compaction criteria comprises:
determining that a size of each file in the subset of the set of resulting files is less than the threshold size defined by the compaction policy.
15 . The method of claim 14 , wherein determining that the subset of the set of resulting files satisfy the compaction criteria further comprises:
determining that the subset of the set of resulting files are ordered adjacent to each other in the set of resulting files.
16 . A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of a computing system, cause the computing system to perform operations comprising:
merging a target table and a source table, wherein the source table has one or more records that are different from the target table, wherein the merging yields a set of resulting files;
determining that a subset of the set of resulting files satisfy a compaction criteria defined by a compaction policy by determining that a number of files in the subset of the set of resulting files is greater than a threshold number of files;
compacting the subset of the set of resulting files to obtain a set of compacted files, wherein the compacting reduces a number of files in the subset of the set of resulting files;
determining that a given file from the set of resulting files should not be compacted based on determining that a size of the given file is larger than a threshold size defined by the compaction policy; and
obtaining a resulting table based at least in part on the set of compacted files and the given file that is not compacted, the resulting table comprising a result of merging the target table and the source table.