Systems and methods for selective scanning of external partitions

1. A method performed by a database platform executing instructions on at least one hardware processor, the method comprising:

receiving a query directed at least in part to an external table stored on a data storage platform external to the database platform, the external table being partitioned into a plurality of partitions corresponding to storage locations in the external data storage platform;

pruning, using external-table metadata that maps the partitions in the plurality of partitions of the external table to the storage locations in the external data storage platform, those partitions among the plurality of partitions that do not potentially contain data that satisfies the query, the external-table metadata being stored by the database platform;

identifying data that satisfies the query by scanning any one or more partitions in the plurality of partitions that were not pruned; and

responding to the query at least in part with the identified data that satisfies the query.

2. The method of claim 1 , further comprising:

receiving, from the external data storage platform, a notification of a modification having been made in a particular storage location in the source directory of the external data storage platform; and

updating the external-table metadata to reflect the modification having been made in the particular storage location.

3. The method of claim 2 , wherein the modification comprises one or more of a file having been added to the particular storage location, a file having been modified in the particular storage location, and a file having been deleted from the particular storage location.

4. The method of claim 1 , further comprising refreshing the external-table metadata at threshold time periods.

5. The method of claim 1 , further comprising refreshing the external-table metadata in response to a threshold number of modifications being made to the external data.

6. The method of claim 1 , further comprising refreshing the external-table metadata in response to receiving a request to refresh the external-table metadata.

7. The method of claim 1 , further comprising reading multiple different file formats in the external data.

8. The method of claim 1 , wherein:

the database platform stores data in a first format;

the external data includes data that is stored in the external data storage platform in at least one second format that is different from the first format; and

the method further comprises converting, into the first format for storage at the database platform, the data that is stored in the at least one second format.

9. The method of claim 1 , further comprising generating the external-table metadata based on a hierarchical structure of the storage locations in the source directory of the external data storage platform.

10. The method of claim 1 , further comprising generating the external-table metadata, the generating of the external-table metadata comprising one or more of:

defining cumulative table metadata for the external-data-mapping table;

defining a grouping expression property for a grouping of the partitions in the external-data-mapping table;

defining an expression property for at least one of the partitions of the external-data-mapping table;

defining partition statistics for at least one of the partitions of the external-data-mapping table; and

defining a column expression property for at least one column of at least one of the partitions of the external-data-mapping table.

11. A database platform comprising:

at least one processor; and

one or more non-transitory computer readable storage media containing instructions that, when executed by the at least one processor, cause the database platform to perform operations comprising:

receiving a query directed at least in part to an external table stored on a data storage platform external to the database platform, the external table being partitioned into a plurality of partitions corresponding to storage locations in the external data storage platform;

pruning, using external-table metadata that maps the partitions in the plurality of partitions of the external table to the storage locations in the external data storage platform, those partitions among the plurality of partitions that do not potentially contain data that satisfies the query, the external-table metadata being stored by the database platform;

identifying data that satisfies the query by scanning any one or more partitions in the plurality of partitions that were not pruned; and

responding to the query at least in part with the identified data that satisfies the query.

12. The database platform of claim 11 , the operations further comprising:

receiving, from the external data storage platform, a notification of a modification having been made in a particular storage location in the source directory of the external data storage platform; and

updating the external-table metadata to reflect the modification having been made in the particular storage location.

13. The database platform of claim 12 , wherein the modification comprises one or more of a file having been added to the particular storage location, a file having been modified in the particular storage location, and a file having been deleted from the particular storage location.

14. The database platform of claim 11 , the operations further comprising refreshing the external-table metadata at threshold time periods.

15. The database platform of claim 11 , the operations further comprising refreshing the external-table metadata in response to a threshold number of modifications being made to the external data.

16. The database platform of claim 11 , the operations further comprising refreshing the external-table metadata in response to receiving a request to refresh the external-table metadata.

17. The database platform of claim 11 , the operations further comprising reading multiple different file formats in the external data.

18. The database platform of claim 11 , wherein:

the database platform stores data in a first format;

the external data includes data that is stored in the external data storage platform in at least one second format that is different from the first format; and

the operations further comprise converting, into the first format for storage at the database platform, the data that is stored in the at least one second format.

19. The database platform of claim 11 , the operations further comprising generating the external-table metadata based on a hierarchical structure of the storage locations in the source directory of the external data storage platform.

20. The database platform of claim 11 , the operations further comprising generating the external-table metadata, the generating of the external-table metadata comprising one or more of:

defining cumulative table metadata for the external-data-mapping table;

defining a grouping expression property for a grouping of the partitions in the external-data-mapping table;

defining an expression property for at least one of the partitions of the external-data-mapping table;

defining partition statistics for at least one of the partitions of the external-data-mapping table; and

defining a column expression property for at least one column of at least one of the partitions of the external-data-mapping table.

21. One or more non-transitory computer readable storage media containing instructions that, when executed by at least one hardware processor of a database platform, cause the database platform to perform operations comprising:

receiving a query directed at least in part to an external table stored on a data storage platform external to the database platform, the external table being partitioned into a plurality of partitions corresponding to storage locations in the external data storage platform;

pruning, using external-table metadata that maps the partitions in the plurality of partitions of the external table to the storage locations in the external data storage platform, those partitions among the plurality of partitions that do not potentially contain data that satisfies the query, the external-table metadata being stored by the database platform;

identifying data that satisfies the query by scanning any one or more partitions in the plurality of partitions that were not pruned; and

responding to the query at least in part with the identified data that satisfies the query.

22. The one or more non-transitory computer readable storage media of claim 21 , the operations further comprising:

receiving, from the external data storage platform, a notification of a modification having been made in a particular storage location in the source directory of the external data storage platform; and

updating the external-table metadata to reflect the modification having been made in the particular storage location.

23. The one or more non-transitory computer readable storage media of claim 21 , the operations further comprising refreshing the external-table metadata in response to receiving a request to refresh the external-table metadata.

24. The one or more non-transitory computer readable storage media of claim 21 , the operations further comprising generating the external-table metadata based on a hierarchical structure of the storage locations in the source directory of the external data storage platform.

25. The one or more non-transitory computer readable storage media of claim 21 , the operations further comprising generating the external-table metadata, the generating of the external-table metadata comprising one or more of:

defining cumulative table metadata for the external-data-mapping table;

defining a grouping expression property for a grouping of the partitions in the external-data-mapping table;

defining an expression property for at least one of the partitions of the external-data-mapping table;

defining partition statistics for at least one of the partitions of the external-data-mapping table; and

defining a column expression property for at least one column of at least one of the partitions of the external-data-mapping table.