Auto refresh of directory tables for stages

1. A method comprising:

storing an internal stage of a database platform;

generating, at the database platform, a file stage of the internal stage based on file stage properties attributes indicating an auto-refresh setting enabled;

in response to the auto-refresh setting enabled in the file stage properties attributes, generating, at the database platform, a hidden stage directory table and a data pipeline for the file stage;

binding the data pipeline to the file stage of the internal stage; and

updating the hidden stage directory table of the data pipeline with metadata extracted from the database platform.

2. The method of claim 1 , wherein the metadata is extracted from files stored at an internal storage location that is internal to the database platform.

3. The method of claim 1 , further comprising:

detecting that the file stage properties attributes indicate the auto-refresh setting enabled;

wherein the updating is in response to detecting that the file stage properties attributes indicate the auto-refresh setting enabled.

4. The method of claim 1 , further comprising:

updating the hidden stage directory table of the data pipeline without receiving an internal stage directory refresh command at the database platform.

5. The method of claim 1 , wherein generating the file stage of the internal stage in a user account on the database platform.

6. The method of claim 5 , further comprising:

updating the hidden stage directory table of the data pipeline without receiving, from the user account, an internal stage directory refresh command at the database platform.

7. The method of claim 1 , further comprising:

receiving a notification of an event for the file stage of the internal stage in a user account on the database platform;

identifying the data pipeline corresponding to the file stage of the internal stage; and

in response to identifying the data pipeline, updating the hidden stage directory table of the data pipeline.

8. The method of claim 7 , further comprising:

polling a stage bucket for a put or delete event;

detecting a put or delete event; and

in response to detecting the put or delete event, routing the notification of the event to a corresponding distributed database queue partition corresponding to the user account.

9. The method of claim 8 , further comprising:

processing an Async task queue of the corresponding distributed database queue partition as a new ingest notification channel; and

directing the notification of the event to a corresponding data pipeline by matching the data pipeline to the event.

10. The method of claim 7 , wherein identifying the data pipeline includes:

identifying a distributed database queue partition corresponding to the user account associated with the event.

11. The method of claim 10 , wherein identifying the data pipeline corresponding to the file stage comprises:

splitting file updates from the distributed database queue partition to the data pipeline.

12. The method of claim 11 , further comprising:

accessing a metadata table and metadata files associated with table metadata of the data pipeline.

13. The method of claim 12 , further comprising:

rejecting out-of-order file notifications for updates.

14. The method of claim 7 , wherein the event includes a stage put or delete event.

15. The method of claim 7 , wherein properties of the file stage include a directory attribute and an auto-refresh attribute.

16. The method of claim 7 , wherein a first directory table of an external stage being automatically refreshed in response to a first file event at the external stage, a second directory table of the internal stage being automatically refreshed in response to a second file event at the internal stage, wherein the notification of the event for the file stage indicates the second file event in the internal stage, wherein the external stage is managed by a client device, wherein the internal stage is managed by internally by the database platform exclusive of the user account and the client device.

17. A computing apparatus comprising:

one or more processors; and

a memory storing instructions that, when executed by the one or more processors, configure the apparatus to perform operations comprising:

storing an internal stage of a database platform;

generating, at the database platform, a file stage of the internal stage based on file stage properties attributes indicating an auto-refresh setting enabled;

in response to the auto-refresh setting enabled in the file stage properties attributes, generating, at the database platform, a hidden stage directory table and a data pipeline for the file stage;

binding the data pipeline to the file stage of the internal stage; and

updating the hidden stage directory table of the data pipeline with metadata extracted from the database platform.

18. The computing apparatus of claim 17 , wherein the metadata is extracted from files stored at an internal storage location that is internal to the database platform.

19. The computing apparatus of claim 17 , wherein the operations further comprise:

detecting that the file stage properties attributes indicate the auto-refresh setting enabled,

wherein the updating is in response to detecting that the file stage properties attributes indicate the auto-refresh setting enabled.

20. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:

storing an internal stage of a database platform;

generating, at the database platform, a file stage of the internal stage based on file stage properties attributes indicating an auto-refresh setting enabled;

in response to the auto-refresh setting enabled in the file stage properties attributes, generating, at the database platform, a hidden stage directory table and a data pipeline for the file stage;

binding the data pipeline to the file stage of the internal stage; and

updating the hidden stage directory table of the data pipeline with metadata extracted from the database platform.