Retrieval and caching of object metadata across data sources and storage systems

1 . A method, comprising:

receiving, from a client device, a request to perform a query operation on a set of data objects stored in a data source, wherein the data source is remote from a computing system;

prior to accessing the set of data objects in the data source,

accessing a metadata cache storing metadata of one or more data objects, the metadata cache located at the computing system, and the metadata describing information of the one or more data objects stored in the data source;

identifying metadata corresponding to the set of data objects for the query operation in the metadata cache;

determining whether the identified metadata for the set of data objects meets an update condition;

executing, responsive to determining that the identified metadata does not meet the update condition, the query operation based on the identified metadata; and

providing one or more results of executing the query operation to the client device.

2 . The method of claim 1 , wherein the metadata in the metadata cache is fetched from the data source and the metadata cache is stored at a computing resource of the computing system, and the computing resource is different from a cluster that fetches the metadata.

3 . The method of claim 1 , wherein the identified metadata comprises a timestamp associated with the update condition.

4 . The method of claim 1 , further comprising:

determining that the identified metadata is not consistent with the set of data objects for the query operation;

fetching, from the data source, updated metadata for the set of data objects; and

executing the query operation based on the updated metadata.

5 . The method of claim 4 , wherein fetching the updated metadata comprising:

identifying a cluster identifier configured to fetch the updated metadata from the data source; and

sending a request to the identified cluster based on the cluster identifier, causing the identified cluster to fetch the updated metadata from the data source.

6 . The method of claim 1 , wherein the metadata cache stores the metadata according to a hierarchical namespace.

7 . The method of claim 1 , further comprising:

identifying second metadata corresponding to a second set of data objects in the metadata cache;

determining whether the identified second metadata for the second set of data objects meets a refresh deadline;

fetching, responsive to the identified second metadata meeting the update condition, updated second metadata from the data source; and

storing the updated second metadata in the metadata cache.

8 . A non-transitory computer-readable medium comprising stored instructions that when executed by one or more processors of one or more computing devices, cause the one or more computing devices to:

receive, from a client device, a request to perform a query operation on a set of data objects stored in a data source, wherein the data source is remote from the one or more computing devices;

prior to accessing the set of data objects in the data source,

access a metadata cache storing metadata of one or more data objects, the metadata cache located at the one or more computing devices, and the metadata describing information of the one or more data objects stored in the data source;

identify metadata corresponding to the set of data objects for the query operation in the metadata cache;

determine whether the identified metadata for the set of data objects meets an update condition;

execute, responsive to determining that the identified metadata does not meet the update condition, the query operation based on the identified metadata; and

provide one or more results of executing the query operation to the client device.

9 . The non-transitory computer-readable medium of claim 8 , wherein the metadata in the metadata cache is fetched from the data source and the metadata cache is stored at a computing resource of the one or more computing devices, and the computing resource is different from a cluster that fetches the metadata.

10 . The non-transitory computer-readable medium of claim 8 , wherein the identified metadata comprises a timestamp associated with the update condition.

11 . The non-transitory computer-readable medium of claim 8 , wherein the stored instructions executable by the one more processors to further cause the one or more computing devices to:

determine that the identified metadata is not consistent with the set of data objects for the query operation;

fetch, from the data source, updated metadata for the set of data objects; and

execute the query operation based on the updated metadata.

12 . The non-transitory computer-readable medium of claim 11 , wherein the instructions to fetch updated metadata further comprise stored instructions executable by the one more processors to cause the one or more computing devices to:

identify a cluster identifier configured to fetch the updated metadata from the data source; and

send a request to the identified cluster based on the cluster identifier, causing the identified cluster to fetch the updated metadata from the data source.

13 . The non-transitory computer-readable medium of claim 8 , wherein the metadata cache stores the metadata according to a hierarchical namespace.

14 . The non-transitory computer-readable medium of claim 8 , wherein the stored instructions executable by the one more processors to further cause the one or more computing devices to:

identify second metadata corresponding to a second set of data objects in the metadata cache;

determine whether the identified second metadata for the second set of data objects meets a refresh deadline;

fetch, responsive to the identified second metadata meeting the update condition, updated second metadata from the data source; and

store the updated second metadata in the metadata cache.

15 . A system comprising:

one or more computer processors; and

one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the system to:

receive, from a client device, a request to perform a query operation on a set of data objects stored in a data source, wherein the data source is remote from the system;

prior to accessing the set of data objects in the data source,

access a metadata cache storing metadata of one or more data objects, the metadata cache located at the system, and the metadata describing information of the one or more data objects stored in the data source;

identify metadata corresponding to the set of data objects for the query operation in the metadata cache;

determine whether the identified metadata for the set of data objects meets an update condition;

execute, responsive to determining that the identified metadata does not meet the update condition, the query operation based on the identified metadata; and

provide one or more results of executing the query operation to the client device.

16 . The system of claim 15 , wherein the metadata in the metadata cache is fetched from the data source and the metadata cache is stored at a computing resource of the system, and the computing resource is different from a cluster that fetches the metadata.

17 . The system of claim 15 , wherein the identified metadata comprises a timestamp associated with the update condition.

18 . The system of claim 15 , wherein the stored instructions executed by the one or more computer processors to further cause the system to:

determine that the identified metadata is not consistent with the set of data objects for the query operation;

fetch, from the data source, updated metadata for the set of data objects; and

execute the query operation based on the updated metadata.

19 . The system of claim 18 , wherein the instructions to fetch updated metadata further comprise stored instructions executed by the one or more computer processors to cause the system to:

identify a cluster identifier configured to fetch the updated metadata from the data source; and

send a request to the identified cluster based on the cluster identifier, causing the identified cluster to fetch the updated metadata from the data source.

20 . The system of claim 15 , wherein the metadata cache stores the metadata according to a hierarchical namespace.