File-catalog table for file stage
Disclosed herein are systems and methods for implementing a file-catalog table for a file stage. In an embodiment, a data platform receives, from a client, a request that the data platform create a file-catalog table for a file stage of a user account associated with the client, where the file stage contains one or more files at a storage location. The user account includes a file-stage object that corresponds to the file stage and that includes an indication of the storage location. In response to receiving the request, the data platform creates the file-catalog table for the file stage. The file-catalog table lists the one or more files in the file stage, and includes at least two different metadata properties of the files listed in the file-catalog table.
1. A method performed by a data platform executing instructions on at least one hardware processor, the method comprising:
receiving, from a client, a request that the data platform create a file-catalog table for a file stage of a user account associated with the client, the user account being associated with one or more databases that collectively include one or more tables, the file stage comprising data storage having a particular storage location, the file stage containing one or more files at the particular storage location, the user account comprising a file-stage object that corresponds to the file stage and that comprises an indication of the particular storage location; and
creating, in response to receiving the request, the file-catalog table for the file stage, the file-catalog table being an addition to the one or more tables in the one or more databases with which the user account is associated, the file-catalog table comprising a row for each of the one or more files in the file stage, the file-catalog table comprising a column for each of at least two different metadata properties of the one or more files in the file stage.
2. The method of claim 1 , wherein the file stage comprises an internal stage with respect to the data platform.
3. The method of claim 1 , wherein the file stage comprises an external stage with respect to the data platform.
4. The method of claim 1 , further comprising:
receiving, from the client, a stage-creation instruction, the stage-creation instruction requesting creation by the data platform of the file stage for the user account; and
creating, in response to receiving the stage-creation instruction, the file stage for the user account, the creating of the file stage for the user account comprising creating the file-stage object in the user account.
5. The method of claim 4 , wherein the receiving of the request that the data platform create the file-catalog table occurs subsequent to the receiving of the stage-creation instruction.
6. The method of claim 5 , further comprising:
receiving a list-files-table-function call from the client with respect to the file stage; and
executing, in response to receiving the list-files-function call, a list-files table function to populate the file-catalog table.
7. The method of claim 4 , wherein:
the request that the data platform create the file-catalog table requests creation by the data platform of the file-catalog table for a plurality of file stages, the plurality of file stages including the file stage, the plurality of file stages containing a multiple-stage set of one or more files, the multiple-stage set of one or more files including the one or more files in the file stage; and
the file-catalog table comprises a row for each of the one or more files in the multiple-stage set of one or more files.
8. The method of claim 1 , further comprising receiving and executing commands to create and share a view over the file-catalog table, the view comprising presigned Uniform Resource Locators (URLs) for providing access to files referenced in the view, the files referenced in the view being less than all of the files in the file stage.
9. The method of claim 1 , further comprising:
implementing a stream on the file stage; and
updating the file-catalog table based on the stream on the file stage.
10. The method of claim 1 , further comprising:
implementing a stream on the file-catalog table; and
updating a second table based on the stream on the file-catalog table.
11. A data platform comprising:
at least one hardware processor; and
one or more computer-storage media containing instructions that, when executed by the at least one hardware processor, cause the data platform to perform operations including:
receiving, from a client, a request that the data platform create a file-catalog table for a file stage of a user account associated with the client, the user account being associated with one or more databases that collectively include one or more tables, the file stage comprising data storage having a particular storage location, the file stage containing one or more files at the particular storage location, the user account comprising a file-stage object that corresponds to the file stage and that comprises an indication of the particular storage location; and
creating, in response to receiving the request, the file-catalog table for the file stage, the file-catalog table being an addition to the one or more tables in the one or more databases with which the user account is associated, the file-catalog table comprising a row for each of the one or more files in the file stage, the file-catalog table comprising a column for each of at least two different metadata properties of the one or more files in the file stage.
12. The data platform of claim 11 , wherein the file stage comprises an internal stage with respect to the data platform.
13. The data platform of claim 11 , wherein the file stage comprises an external stage with respect to the data platform.
14. The data platform of claim 11 , the operations further comprising:
receiving, from the client, a stage-creation instruction, the stage-creation instruction requesting creation by the data platform of the file stage for the user account; and
creating, in response to receiving the stage-creation instruction, the file stage for the user account, the creating of the file stage for the user account comprising creating the file-stage object in the user account.
15. The data platform of claim 14 , wherein the receiving of the request that the data platform create the file-catalog table occurs subsequent to the receiving of the stage-creation instruction.
16. The data platform of claim 15 , the operations further comprising:
receiving a list-files-table-function call from the client with respect to the file stage; and
executing, in response to receiving the list-files-function call, a list-files table function to populate the file-catalog table.
17. The data platform of claim 14 , wherein:
the request that the data platform create the file-catalog table requests creation by the data platform of the file-catalog table for a plurality of file stages, the plurality of file stages including the file stage, the plurality of file stages containing a multiple-stage set of one or more files, the multiple-stage set of one or more files including the one or more files in the file stage; and
the file-catalog table comprises a row for each of the one or more files in the multiple-stage set of one or more files.
18. The data platform of claim 11 , the operations further comprising receiving and executing commands to create and share a view over the file-catalog table, the view comprising presigned Uniform Resource Locators (URLs) for providing access to files referenced in the view, the files referenced in the view being less than all of the files in the file stage.
19. The data platform of claim 11 , the operations further comprising:
implementing a stream on the file stage; and
updating the file-catalog table based on the stream on the file stage.
20. The data platform of claim 11 , the operations further comprising:
implementing a stream on the file-catalog table; and
updating a second table based on the stream on the file-catalog table.
21. One or more computer-storage media containing instructions that, when executed by at least one hardware processor of a data platform, cause the at least one hardware processor to perform operations including:
receiving, from a client, a request that the data platform create a file-catalog table for a file stage of a user account associated with the client, the user account being associated with one or more databases that collectively include one or more tables, the file stage comprising data storage having a particular storage location, the file stage containing one or more files at the particular storage location, the user account comprising a file-stage object that corresponds to the file stage and that comprises an indication of the particular storage location; and
creating, in response to receiving the request, the file-catalog table for the file stage, the file-catalog table being an addition to the one or more tables in the one or more databases with which the user account is associated, the file-catalog table comprising a row for each of the one or more files in the file stage, the file-catalog table comprising a column for each of at least two different metadata properties of the one or more files in the file stage.
22. The one or more computer-storage media of claim 21 , wherein the file stage comprises an internal stage with respect to the data platform.
23. The one or more computer-storage media of claim 21 , wherein the file stage comprises an external stage with respect to the data platform.
24. The one or more computer-storage media of claim 21 , the operations further comprising:
receiving, from the client, a stage-creation instruction, the stage-creation instruction requesting creation by the data platform of the file stage for the user account; and
creating, in response to receiving the stage-creation instruction, the file stage for the user account, the creating of the file stage for the user account comprising creating the file-stage object in the user account.
25. The one or more computer-storage media of claim 24 , wherein:
wherein the receiving of the request that the data platform create the file-catalog table occurs subsequent to the receiving of the stage-creation instruction; and
the operations further comprise:
receiving a list-files-table-function call from the client with respect to the file stage; and
executing, in response to receiving the list-files-function call, a list-files table function to populate the file-catalog table.
26. The method of claim 4 , wherein:
the stage-creation instruction comprises the request that the data platform create the file-catalog table; and
the file-catalog table comprises a built-in file-catalog table that is managed by the data platform.
27. The data platform of claim 14 , wherein:
the stage-creation instruction comprises the request that the data platform create the file-catalog table; and
the file-catalog table comprises a built-in file-catalog table that is managed by the data platform.
28. The one or more computer-storage media of claim 25 , wherein:
the stage-creation instruction comprises the request that the data platform create the file-catalog table; and
the file-catalog table comprises a built-in file-catalog table that is managed by the data platform.