Optimized reconciliation between cloud networks
Techniques are described for optimizing reconciliation between cloud networks. In some examples, a data platform may store a local copy of data, in which a remote cloud service stores a source copy of the data. The data platform may initiate a resynchronization of the local copy with the source copy stored by the remote cloud service and obtain an indication the resynchronization is incomplete. The data platform may determine if a first subset of the data satisfies one or more validity checks and determine if a second subset of the data is out-of-sync with the source copy of the data at the remote cloud service based on the one or more validity checks. According to at least one example, The data platform initiates a full download of the second subset of the data to the storage system of the data platform from the remote cloud service.
1 . A data platform comprising:
processing circuitry;
a storage system;
non-transitory computer readable media storing instructions; and
wherein the instructions, when executed by the processing circuitry, configure the processing circuitry to:
store, by the processing circuitry, a local copy of data via the storage system of the data platform, wherein a remote cloud service stores a source copy of the data on a remote storage system of the remote cloud service;
initiate, by the processing circuitry, a resynchronization of the local copy of the data stored by the storage system of the data platform with the source copy stored by the remote cloud service;
obtain, by the processing circuitry, an indication the resynchronization of the local copy of the data from the source copy is incomplete;
obtain, by the processing circuitry, remote metadata attributes from the remote cloud service that correspond to the data stored via the storage system without downloading file data content corresponding to the source copy of the data from the remote storage system;
compare, by the processing circuitry, the remote metadata attributes with corresponding local metadata attributes for the data stored via the storage system to determine a first subset of the data stored via the storage system of the data platform satisfies one or more validity checks;
determine, by the processing circuitry, that a second subset of the data stored via the storage system of the data platform is out-of-sync with the source copy of the data stored by the remote cloud service based on the one or more validity checks; and
initiate, by the processing circuitry, a full download of the second subset of the data to the storage system of the data platform from the remote cloud service.
2 . The data platform of claim 1 , wherein the instructions, when executed, further configure the processing circuitry to:
obtain the remote metadata attributes from the remote cloud service without downloading file data content corresponding to the source copy of the data;
determine local metadata attributes for the data stored via the storage system of the data platform; and
validate the first subset of the data stored via the storage system of the data platform satisfies the one or more validity checks based on a comparison of the remote metadata attributes with the local metadata attributes for the data stored via the storage system of the data platform.
3 . The data platform of claim 1 , wherein the instructions, when executed, further configure the processing circuitry to:
obtain the remote metadata attributes from the remote cloud service without downloading file data content corresponding to the source copy of the data;
determine local metadata attributes for the data stored via the storage system of the data platform; and
determine the second subset of the data stored via the storage system of the data platform is out-of-sync with the source copy of the data at the remote cloud service based on a comparison of the remote metadata attributes with the local metadata attributes for the data stored via the storage system of the data platform.
4 . The data platform of claim 3 , wherein the remote metadata attributes and the local metadata attributes describe one or more of:
a file size of one or more files within the data;
a checksum value of one or more files within the data;
a file presence status or file deleted status of one or more files within the data; or
a file name of one or more files within the data.
5 . The data platform of claim 1 :
wherein to obtain the indication the resynchronization of the local copy of the data from the source copy is incomplete includes the instructions, when executed, to further configure the processing circuitry to receive an instruction from the remote cloud service to perform a full download of the data to resynchronize the local copy of the data with the source copy of the data stored by the remote cloud service; and
wherein the instructions, when executed, further configure the processing circuitry to download in full the second subset of the data without download of the first subset of the data determined to satisfy the one or more validity checks.
6 . The data platform of claim 1 , wherein to obtain the indication the resynchronization of the local copy of the data from the source copy is incomplete includes the instructions, when executed, to further configure the processing circuitry to obtain at least one of:
an indication from the remote cloud service that a determined data delta requires a full download of the data from the source copy of the data stored by the remote cloud service;
an instruction from the remote cloud service to perform a full download of the source copy of the data from the remote cloud service;
an indication from the remote cloud service that token or link configured to access the source copy of the data from the remote cloud service has expired; or
an indication from the remote cloud service that a requested download of the data from the remote cloud service will be throttled.
7 . The data platform of claim 1 , wherein to obtain the indication the resynchronization of the local copy of the data from the source copy is incomplete includes the instructions, when executed, to further configure the processing circuitry to obtain at least one of:
an error message from the remote cloud service;
a determined latency for the resynchronization that satisfies a time-out threshold;
a determined throttling condition at the remote cloud service affecting the resynchronization of the local copy of the data from the source copy of the data stored by the remote cloud service;
a determined quantity of change operations on the data satisfies a maximum data-delta threshold;
an elapsed time since the data was last fully downloaded from the source copy of the data stored by the remote cloud service satisfies a maximum incremental backup threshold; or
a determined bandwidth consumption that satisfies a bandwidth threshold.
8 . The data platform of claim 1 , wherein the data comprises at least one of:
a user mailbox stored by the remote cloud service;
a file stored by the remote cloud service;
a directory stored by the remote cloud service;
a virtual drive stored by the remote cloud service;
a user data repository stored by the remote cloud service; or
an organization data repository stored by the remote cloud service.
9 . The data platform of claim 1 , wherein to initiate the resynchronization of the local copy of the data from the source copy stored by the remote cloud service includes the instructions, when executed, to further configure the processing circuitry to resynchronize the local copy of the data with the source copy stored by the remote cloud service without a full download of the data from the source copy stored by the remote cloud service.
10 . The data platform of claim 1 , wherein to initiate the resynchronization of the local copy of the data from the source copy stored by the remote cloud service includes the instructions, when executed, to further configure the processing circuitry to:
initiate an incremental download of the local copy of the data from the source copy stored by the remote cloud service; or
initiate a partial download of the local copy of the data from the source copy stored by the remote cloud service.
11 . A method comprising:
storing a local copy of data via a storage system of a data platform, wherein a remote cloud service stores a source copy of the data on a remote storage system of the remote cloud service;
initiating a resynchronization of the local copy of the data stored by the storage system of the data platform with the source copy stored by the remote cloud service;
obtaining an indication the resynchronization of the local copy of the data from the source copy is incomplete;
obtaining remote metadata attributes from the remote cloud service that correspond to the data stored via the storage system without downloading file data content corresponding to the source copy of the data from the remote storage system;
comparing the remote metadata attributes with corresponding local metadata attributes for the data stored via the storage system to determine a first subset of the data stored via the storage system of the data platform satisfies one or more validity checks;
determining that a second subset of the data stored via the storage system of the data platform is out-of-sync with the source copy of the data stored by the remote cloud service based on the one or more validity checks; and
initiating a full download of the second subset of the data to the storage system of the data platform from the remote cloud service.
12 . The method of claim 11 , further comprising:
obtaining the remote metadata attributes from the remote cloud service without downloading file data content corresponding to the source copy of the data;
determining local metadata attributes for the data stored via the storage system of the data platform; and
validating the first subset of the data stored via the storage system of the data platform satisfies the one or more validity checks based on a comparison of the remote metadata attributes with the local metadata attributes for the data stored via the storage system of the data platform.
13 . The method of claim 11 , further comprising:
obtaining the remote metadata attributes from the remote cloud service without downloading file data content corresponding to the source copy of the data;
determining local metadata attributes for the data stored via the storage system of the data platform; and
determining the second subset of the data stored via the storage system of the data platform is out-of-sync with the source copy of the data at the remote cloud service based on a comparison of the remote metadata attributes with the local metadata attributes for the data stored via the storage system of the data platform.
14 . The method of claim 13 , wherein the remote metadata attributes and the local metadata attributes describe one or more of:
a file size of one or more files within the data;
a checksum value of one or more files within the data;
a file presence status or file deleted status of one or more files within the data; or
a file name of one or more files within the data.
15 . The method of claim 11 , further comprising:
receiving an instruction from the remote cloud service to perform a full download of the data to resynchronize the local copy of the data with the source copy of the data stored by the remote cloud service; and
downloading, in full, the second subset of the data without downloading the first subset of the data determined to satisfy the one or more validity checks.
16 . The method of claim 11 , wherein obtaining the indication the resynchronization of the local copy of the data from the source copy is incomplete includes obtaining at least one of:
receiving an indication from the remote cloud service that a determined data delta requires a full download of the data from the source copy of the data stored by the remote cloud service;
receiving an instruction from the remote cloud service to perform a full download of the source copy of the data from the remote cloud service;
receiving an indication from the remote cloud service that token or link configured to access the source copy of the data from the remote cloud service has expired; or
receiving an indication from the remote cloud service that a requested download of the data from the remote cloud service will be throttled.
17 . The method of claim 11 , wherein obtaining the indication the resynchronization of the local copy of the data from the source copy is incomplete includes at least one of:
obtaining an error message from the remote cloud service;
determining latency for the resynchronization satisfies a time-out threshold;
determining a throttling condition at the remote cloud service is affecting the resynchronization of the local copy of the data from the source copy of the data stored by the remote cloud service;
determining a quantity of change operations on the data satisfies a maximum data-delta threshold;
determining an elapsed time since the data was last fully downloaded from the source copy of the data stored by the remote cloud service satisfies a maximum incremental backup threshold; or
determining bandwidth consumption satisfies a bandwidth threshold.
18 . The method of claim 11 , wherein the data comprises at least one of:
a user mailbox stored by the remote cloud service;
a file stored by the remote cloud service;
a directory stored by the remote cloud service;
a virtual drive stored by the remote cloud service;
a user data repository stored by the remote cloud service; or
an organization data repository stored by the remote cloud service.
19 . The method of claim 11 , further comprising:
synchronizing the local copy of the data with the source copy stored by the remote cloud service without a full download of the data from the source copy stored by the remote cloud service.
20 . Computer-readable storage media comprising instructions that, when executed, configure processing circuitry to:
store a local copy of data via a storage system of a data platform, wherein a remote cloud service stores a source copy of the data on a remote storage system of the remote cloud service;
initiate a resynchronization of the local copy of the data stored by the storage system of the data platform with the source copy stored by the remote cloud service;
obtain an indication the resynchronization of the local copy of the data from the source copy is incomplete;
obtain remote metadata attributes from the remote cloud service that correspond to the data stored via the storage system without downloading file data content corresponding to the source copy of the data from the remote storage system;
compare the remote metadata attributes with corresponding local metadata attributes for the data stored via the storage system to determine a first subset of the data stored via the storage system of the data platform satisfies one or more validity checks;
determine that a second subset of the data stored via the storage system of the data platform is out-of-sync with the source copy of the data stored by the remote cloud service based on the one or more validity checks; and
initiate a full download of the second subset of the data to the storage system of the data platform from the remote cloud service.