Supporting unstructured, semi-structured, and structured files
A data platform provides support for unstructured files, semi-structured files, and structured files. In various different embodiments, the data platform provides functions for obtaining relative paths, absolute paths, and presigned URLs for files in internal and external stages; application programming interfaces (APIs) for integrating with external functions; built-in file-catalog tables; integration with cloud-storage platforms; customized views for flexible file-sharing options, stage-encryption options; notification integrations; table functions for generating file-catalog tables; streams on stages; streams on tables; and other features.
1. A method comprising:
receiving, at a data platform that comprises at least one hardware processor, a stage-creation instruction from a client, the stage-creation instruction requesting creation by the data platform of a file stage for a user account on the data platform, the user account being associated with the client;
creating, by the data platform in response to receiving the stage-creation instruction from the client, the file stage for the user account, the file stage comprising data storage having a particular storage location, the creating of the file stage for the user account comprising creating a file-stage object in the user account, the file-stage object representing the file stage in the user account, the file-stage object comprising an indication of the particular storage location of the file stage;
receiving, at the data platform, a file-catalog-table-creation instruction from the client, the file-catalog-table-creation instruction requesting creation by the data platform of a file-catalog table for the file stage; and
creating, by the data platform in response to receiving the file-catalog-table-creation instruction, the file-catalog table for the file stage, the file stage containing one or more files, the file-catalog table listing the one or more files in the file stage, each of the files listed in the file-catalog table corresponding to a row in the file-catalog table, the file-catalog table comprising at least two columns, each of the at least two columns corresponding to a different metadata property of each of the files listed in the file-catalog table.
2. The method of claim 1 , wherein the file stage comprises an internal stage with respect to the data platform.
3. The method of claim 1 , wherein the file stage comprises an external stage with respect to the data platform.
4. The method of claim 1 , wherein:
the stage-creation instruction comprises the file-catalog-table-creation instruction; and
the file-catalog table comprises a built-in file-catalog table that is managed by the data platform.
5. The method of claim 1 , wherein the receiving of the file-catalog-table-creation instruction occurs subsequent to the receiving of the stage-creation instruction.
6. The method of claim 5 , further comprising:
receiving a list-files-table-function call from the client with respect to the file stage; and
executing, in response to receiving the list-files-function call, a list-files table function to populate the file-catalog table.
7. The method of claim 1 , further comprising receiving and executing commands to create and share a view over the file-catalog table, the view comprising presigned Uniform Resource Locators (URLs) for providing access to files referenced in the view, the files referenced in the view being less than all of the files in the file stage.
8. The method of claim 1 , further comprising:
implementing a stream on the file stage; and
updating the file-catalog table based on the stream on the file stage.
9. The method of claim 1 , further comprising:
implementing a stream on the file-catalog table; and
updating a second table based on the stream on the file-catalog table.
10. The method of claim 1 , wherein:
the file-catalog-table-creation instruction requests creation by the data platform of a file-catalog table for a plurality of file stages, the plurality of file stages including the file stage, the plurality of file stages containing a multiple-stage set of one or more files, the multiple-stage set of one or more files including the one or more files in the file stage; and
the file-catalog table lists the one or more files in the multiple-stage set of one or more files.
11. A data platform comprising:
at least one hardware processor; and
one or more computer-storage media containing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations including:
receiving a stage-creation instruction from a client, the stage-creation instruction requesting creation by the data platform of a file stage for a user account on the data platform, the user account being associated with the client;
creating, in response to receiving the stage-creation instruction from the client, the file stage for the user account, the file stage comprising data storage having a particular storage location, the creating of the file stage for the user account comprising creating a file-stage object in the user account, the file-stage object representing the file stage in the user account, the file-stage object comprising an indication of the particular storage location of the file stage;
receiving a file-catalog-table-creation instruction from the client, the file-catalog-table-creation instruction requesting creation by the data platform of a file-catalog table for the file stage; and
creating, in response to receiving the file-catalog-table-creation instruction, the file-catalog table for the file stage, the file stage containing one or more files, the file-catalog table listing the one or more files in the file stage, each of the files listed in the file-catalog table corresponding to a row in the file-catalog table, the file-catalog table comprising at least two columns, each of the at least two columns corresponding to a different metadata property of each of the files listed in the file-catalog table.
12. The data platform of claim 11 , wherein the file stage comprises an internal stage with respect to the data platform.
13. The data platform of claim 11 , wherein the file stage comprises an external stage with respect to the data platform.
14. The data platform of claim 11 , wherein:
the stage-creation instruction comprises the file-catalog-table-creation instruction; and
the file-catalog table comprises a built-in file-catalog table that is managed by the data platform.
15. The data platform of claim 11 , wherein the receiving of the file-catalog-table-creation instruction occurs subsequent to the receiving of the stage-creation instruction.
16. The data platform of claim 15 , the operations further comprising:
receiving a list-files-table-function call from the client with respect to the file stage; and
executing, in response to receiving the list-files-function call, a list-files table function to populate the file-catalog table.
17. The data platform of claim 11 , the operations further comprising receiving and executing commands to create and share a view over the file-catalog table, the view comprising presigned Uniform Resource Locators (URLs) for providing access to files referenced in the view, the files referenced in the view being less than all of the files in the file stage.
18. The data platform of claim 11 , the operations further comprising:
implementing a stream on the file stage; and
updating the file-catalog table based on the stream on the file stage.
19. The data platform of claim 11 , the operations further comprising:
implementing a stream on the file-catalog table; and
updating a second table based on the stream on the file-catalog table.
20. The data platform of claim 11 , wherein:
the file-catalog-table-creation instruction requests creation by the data platform of a file-catalog table for a plurality of file stages, the plurality of file stages including the file stage, the plurality of file stages containing a multiple-stage set of one or more files, the multiple-stage set of one or more files including the one or more files in the file stage; and
the file-catalog table lists the one or more files in the multiple-stage set of one or more files.
21. One or more computer-storage media containing instructions that, when executed by at least one hardware processor of a data platform, cause the at least one hardware processor to perform operations including:
receiving a stage-creation instruction from a client, the stage-creation instruction requesting creation by the data platform of a file stage for a user account on the data platform, the user account being associated with the client;
creating, in response to receiving the stage-creation instruction from the client, the file stage for the user account, the file stage comprising data storage having a particular storage location, the creating of the file stage for the user account comprising creating a file-stage object in the user account, the file-stage object representing the file stage in the user account, the file-stage object comprising an indication of the particular storage location of the file stage;
receiving a file-catalog-table-creation instruction from the client, the file-catalog-table-creation instruction requesting creation by the data platform of a file-catalog table for the file stage; and
creating, in response to receiving the file-catalog-table-creation instruction, the file-catalog table for the file stage, the file stage containing one or more files, the file-catalog table listing the one or more files in the file stage, each of the files listed in the file-catalog table corresponding to a row in the file-catalog table, the file-catalog table comprising at least two columns, each of the at least two columns corresponding to a different metadata property of each of the files listed in the file-catalog table.
22. The one or more computer-storage media of claim 21 , wherein the file stage comprises an internal stage with respect to the data platform.
23. The one or more computer-storage media of claim 21 , wherein the file stage comprises an external stage with respect to the data platform.
24. The one or more computer-storage media of claim 21 , wherein:
the stage-creation instruction comprises the file-catalog-table-creation instruction; and
the file-catalog table comprises a built-in file-catalog table that is managed by the data platform.
25. The one or more computer-storage media of claim 21 , wherein the receiving of the file-catalog-table-creation instruction occurs subsequent to the receiving of the stage-creation instruction.
26. The one or more computer-storage media of claim 25 , the operations further comprising:
receiving a list-files-table-function call from the client with respect to the file stage; and
executing, in response to receiving the list-files-function call, a list-files table function to populate the file-catalog table.
27. The one or more computer-storage media of claim 21 , the operations further comprising receiving and executing commands to create and share a view over the file-catalog table, the view comprising presigned Uniform Resource Locators (URLs) for providing access to files referenced in the view, the files referenced in the view being less than all of the files in the file stage.
28. The one or more computer-storage media of claim 21 , the operations further comprising:
implementing a stream on the file stage; and
updating the file-catalog table based on the stream on the file stage.
29. The one or more computer-storage media of claim 21 , the operations further comprising:
implementing a stream on the file-catalog table; and
updating a second table based on the stream on the file-catalog table.
30. The one or more computer-storage media of claim 21 , wherein:
the file-catalog-table-creation instruction requests creation by the data platform of a file-catalog table for a plurality of file stages, the plurality of file stages including the file stage, the plurality of file stages containing a multiple-stage set of one or more files, the multiple-stage set of one or more files including the one or more files in the file stage, and
the file-catalog table lists the one or more files in the multiple-stage set of one or more files.