Scaling delta table optimize command
The interface is to receive an indication to execute an optimize command. The processor is to receive a file name; determine whether adding a file of the file name to a current bin causes the current bin to exceed a threshold; associate the file with the current bin in response to determining that adding the file does not cause the current bin to exceed the bin threshold; in response to determining that adding the file to the current bin causes the current bin to exceed the bin threshold: associate the file with a next bin, indicate that the current bin is closed, and add the current bin to a batch of bins; determine whether a measure of the batch of bins exceeds a batch threshold; and in response to determining that the measure exceeds the batch threshold, provide the batch of bins for processing.
1 . A method comprising:
allocating files from a data store to a first bin of files until a total size of files included in the first bin of files remains under a bin threshold value;
determining that addition of an unallocated file from the data store to the first bin of files would cause the total size of files included in the first bin of files to exceed the bin threshold value; and
based on the determination:
allocating the unallocated file to a second bin of files based on further determining that addition of the unallocated file to the second bin of files would not cause a total size of files included in the second bin of files to exceed the bin threshold value,
determining a number of files included in the first bin of files,
comparing the number of files to a threshold number of files, and
in response to determining that the number of files meets or exceeds the threshold number of files, compacting the files included in the first bin into a first aggregated file.
2 . The method of claim 1 , further comprising:
determining that addition of a subsequent unallocated file from the data store to the second bin of files would cause the total size of the files included in the second bin of files to exceed the bin threshold value; and
based on determining that addition of the subsequent unallocated file from the data store to the second bin of files would cause the total size of the files included in the second bin of files to exceed the bin threshold value:
determining a number of files included in the second bin of files;
comparing the number of files included in the second bin of files to the threshold number of files; and
allocating the subsequent unallocated file to a third bin of files based on further determining that addition of the subsequent unallocated file to the third bin of files would not cause a total size of files included in the third bin of files to exceed the bin threshold value.
3 . The method of claim 2 , further comprising:
compacting the files included in the second bin of files into a second aggregated file based on determining that the number of files included in the second bin of files meets or exceeds the threshold number of files.
4 . The method of claim 2 , further comprising:
determining that the files in the second bin of files should not be compacted based on determining that the number of files included in the second bin of files does not meet or exceed the threshold number of files; and
generating a data table based at least in part on the first aggregated file and the files in the second bin of files.
5 . The method of claim 3 , further comprising:
generating a data table based at least in part on the first aggregated file and the second aggregated file.
6 . A system comprising:
one or more computer processors; and
one or more non-transitory computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the system to perform operations comprising:
allocating files from a data store to a first bin of files until a total size of files included in the first bin of files remains under a bin threshold value;
determining that addition of an unallocated file from the data store to the first bin of files would cause the total size of files included in the first bin of files to exceed the bin threshold value; and
based on the determination:
allocating the unallocated file to a second bin of files based on further determining that addition of the unallocated file to the second bin of files would not cause a total size of files included in the second bin of files to exceed the bin threshold value,
determining a number of files included in the first bin of files,
comparing the number of files to a threshold number of files, and
in response to determining that the number of files meets or exceeds the threshold number of files, compacting the files included in the first bin into a first aggregated file.
7 . The system of claim 6 , wherein the operations further comprise:
determining that addition of a subsequent unallocated file from the data store to the second bin of files would cause the total size of the files included in the second bin of files to exceed the bin threshold value; and
based on determining that addition of the subsequent unallocated file from the data store to the second bin of files would cause the total size of the files included in the second bin of files to exceed the bin threshold value:
determining a number of files included in the second bin of files;
comparing the number of files included in the second bin of files to the threshold number of files; and
allocating the subsequent unallocated file to a third bin of files based on further determining that addition of the subsequent unallocated file to the third bin of files would not cause a total size of files included in the third bin of files to exceed the bin threshold value.
8 . The system of claim 7 , wherein the operations further comprise:
compacting the files included in the second bin of files into a second aggregated file based on determining that the number of files included in the second bin of files meets or exceeds the threshold number of files.
9 . The system of claim 7 , wherein the operations further comprise:
determining that the files in the second bin of files should not be compacted based on determining that the number of files included in the second bin of files does not meet or exceed the threshold number of files; and
generating a data table based at least in part on the first aggregated file and the files in the second bin of files.
10 . The system of claim 8 , wherein the operations further comprise:
generating a data table based at least in part on the first aggregated file and the second aggregated file.
11 . A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of a computing system, cause the computing system to perform operations comprising:
allocating files from a data store to a first bin of files until a total size of files included in the first bin of files remains under a bin threshold value;
determining that addition of an unallocated file from the data store to the first bin of files would cause the total size of files included in the first bin of files to exceed the bin threshold value; and
based on the determination:
allocating the unallocated file to a second bin of files based on further determining that addition of the unallocated file to the second bin of files would not cause a total size of files included in the second bin of files to exceed the bin threshold value,
determining a number of files included in the first bin of files,
comparing the number of files to a threshold number of files, and
in response to determining that the number of files meets or exceeds the threshold number of files, compacting the files included in the first bin into a first aggregated file.
12 . The non-transitory computer-readable medium of claim 11 , wherein the operations further comprise:
determining that addition of a subsequent unallocated file from the data store to the second bin of files would cause the total size of the files included in the second bin of files to exceed the bin threshold value; and
based on determining that addition of the subsequent unallocated file from the data store to the second bin of files would cause the total size of the files included in the second bin of files to exceed the bin threshold value:
determining a number of files included in the second bin of files;
comparing the number of files included in the second bin of files to the threshold number of files; and
allocating the subsequent unallocated file to a third bin of files based on further determining that addition of the subsequent unallocated file to the third bin of files would not cause a total size of files included in the third bin of files to exceed the bin threshold value.
13 . The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise:
compacting the files included in the second bin of files into a second aggregated file based on determining that the number of files included in the second bin of files meets or exceeds the threshold number of files; and
generating a data table based at least in part on the first aggregated file and the second aggregated file.
14 . The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise:
determining that the files in the second bin of files should not be compacted based on determining that the number of files included in the second bin of files does not meet or exceed the threshold number of files; and
generating a data table based at least in part on the first aggregated file and the files in the second bin of files.