Scalable data import into managed lakehouses
Aspects of the disclosure are directed to managing files in data lakehouses using rewrite-free loading. Rewrite-free loading includes keeping track of information from data files that could be missing when imported to the data lakehouses without having to perform full-copy loading. Rewrite-free loading can store this information in table metadata or augment headers and/or footers of the data files with this information when importing to a data lakehouse. Rewrite-free loading allows for more accurate management of data lakehouses with lower computational costs.
1 . A method for managing files in a data lakehouse comprising:
receiving, by one or more processors, a request from a query engine to load a data file to the data lakehouse;
retrieving, by the one or more processors, the data file and information associated with the data file;
generating, by the one or more processors, a header/footer file containing the information associated with the data file;
generating, by the one or more processors, a composite file based on the data file and the header/footer file by concatenating the data file with the header/footer file, wherein at least one of a header or footer of the data file becomes available space in memory; and
storing, by the one or more processors, the composite file in the data lakehouse.
2 . The method of claim 1 , wherein the information associated with the data file comprises at least one of a stable column identifier, a partition key, or an integrity signature.
3 . The method of claim 1 , wherein the header or footer of the data file becomes available space in memory in response to being concatenated with the header/footer file.
4 . The method of claim 1 , further comprising storing, by the one or more processors, an extension in the available space in memory between the data file and the header/footer file of the composite file.
5 . The method of claim 4 , wherein the extension comprises an integrity signature.
6 . The method of claim 4 , wherein the extension is a non-standard extension in a data format only understandable by the data lakehouse.
7 . The method of claim 1 , further comprising:
receiving, by the one or more processors, a second request from the query engine or a second query engine to load a second data file to the data lakehouse;
retrieving, by the one or more processors, the second data file and information associated with the second data file;
generating, by the one or more processors, a second header/footer file containing the information associated with the second data file;
copying, by the one or more processors, the second data file to generate a copied data file; and
storing, by the one or more processors, the copied data file and the second header/footer file in the data lakehouse.
8 . The method of claim 1 , wherein generating the composite file and storing the composite file is performed automatically in response to a trigger mechanism.
9 . The method of claim 8 , wherein the trigger mechanism comprises at least one of a notification of a file addition or an instruction to list new files since a predetermined date.
10 . A system comprising:
one or more processors; and
one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for managing files in a data lakehouse, the operations comprising:
receiving a request from a query engine to load a data file to the data lakehouse;
retrieving the data file and information associated with the data file;
generating a header/footer file containing the information associated with the data file;
generating a composite file based on the data file and the header/footer file by concatenating the data file with the header/footer file, wherein at least one of a header or footer of the data file becomes available space in memory; and
storing the composite file in the data lakehouse.
11 . The system of claim 10 , wherein the information associated with the data file comprises at least one of a stable column identifier, a partition key, or an integrity signature.
12 . The system of claim 10 , wherein the header or footer of the data file becomes available space in memory in response to being concatenated with the header/footer file.
13 . The system of claim 10 , wherein the operations further comprise storing an extension in the available space in memory between the data file and the header/footer file of the composite file.
14 . The system of claim 13 , wherein the extension comprises an integrity signature.
15 . The system of claim 13 , wherein the extension is a non-standard extension in a data format only understandable by the data lakehouse.
16 . The system of claim 10 , wherein the operations further comprise:
receiving a second request from the query engine or a second query engine to load a second data file to the data lakehouse;
retrieving the second data file and information associated with the second data file;
generating a second header/footer file containing the information associated with the second data file;
copying the second data file to generate a copied data file; and
storing the copied data file and the second header/footer file in the data lakehouse.
17 . The system of claim 10 , wherein generating the composite file and storing the composite file is performed automatically in response to a trigger mechanism.
18 . A non-transitory computer readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for managing files in a data lakehouse, the operations comprising:
receiving a request from a query engine to load a data file to the data lakehouse;
retrieving the data file and information associated with the data file;
generating a header/footer file containing the information associated with the data file;
generating a composite file based on the data file and the header/footer file by concatenating the data file with the header/footer file, wherein at least one of a header or footer of the data file becomes available space in memory; and
storing the composite file in the data lakehouse.