Data management across cloud environments
Techniques are described for data management across cloud environments. An example method comprises restoring, by the data platform, a selected copy of the one or more copies from the one or more second storage systems by, for each chunk of the selected copy: identifying, by the data platform, the chunk in the chunk metadata, determining, by the data platform, whether a matching chunk is stored on the first storage system based on the chunk metadata, responsive to determining the matching chunk is stored on the first storage system, retrieving, by the data platform, the matching chunk from the first storage system, wherein the matching chunk is included in the selected copy, and responsive to determining the matching chunk is not stored on the first storage system, retrieving, by the data platform, the chunk from the one or more second storage systems, wherein the chunk is included in the selected copy.
1 . A method comprising:
storing, by a data platform implemented by a computing system, a first plurality of chunks storing data for an object of a file system and a second plurality of chunks storing data for a copy of the object, wherein the first plurality of chunks are stored on a first storage system and the second plurality of chunks are stored on one or more second storage systems;
storing, by the data platform, chunk metadata for both the first plurality of chunks and the second plurality of chunks at the first storage system that is deployed local to the computing system that implements the data platform; and
restoring, by the data platform, a selected copy of the one or more copies from the one or more second storage systems by, for each chunk of the selected copy:
identifying, by the data platform and from the chunk metadata, a data access cost for restoring a matching chunk, that matches the chunk, from the first storage system, wherein the data access cost differs based on whether restoring the matching chunk from the first storage system includes a data egress from a different computing system to the computing system that implements the data platform;
determining, by the data platform, whether the data access cost for restoring the matching chunk from the first storage system is lower than that of restoring the chunk from the one or more second storage systems;
responsive to determining that the data access cost for restoring the matching chunk from the first storage system is lower than that of restoring the chunk from the second storage system, retrieving, by the data platform, the matching chunk from the first storage system, wherein the matching chunk is included in the selected copy; and
responsive to determining that the data access cost for restoring the matching chunk from the first storage system is not lower than that of restoring the chunk from the second storage system, retrieving, by the data platform, the chunk from the one or more second storage systems, wherein the chunk is included in the selected copy.
2 . The method of claim 1 , further comprising, after restoring the selected copy, storing, by the data platform, the selected copy on a storage system selected from one or more of the first storage system and the one or more second storage systems.
3 . The method of claim 1 , wherein each of at least a subset of the one or more copies represents a backup of the object for a particular time.
4 . The method of claim 1 , wherein the one or more second storage systems are deployed to the different computing system.
5 . The method of claim 1 , wherein each of the first storage system and the one or more second storage systems are provided by distinct cloud service providers.
6 . The method of claim 5 , wherein retrieving the chunk from the one or more second storage systems is associated with a higher cost than retrieving the matching chunk from the first storage system.
7 . The method of claim 5 , further comprising locating the chunk in the one or more second storage systems is associated with a higher cost than locating the matching chunk from the first storage system.
8 . The method of claim 1 , further comprising receiving, by the data platform, an indication of the selected copy via an input device.
9 . A computing system comprising:
a memory storing instructions; and
processing circuitry that executes the instructions to:
store a first plurality of chunks storing data for an object of a file system and a second plurality of chunks storing data for a copy of the object, wherein the first plurality of chunks are stored on a first storage system and the second plurality of chunks are stored on one or more second storage systems;
store chunk metadata for the first plurality of chunks and the second plurality of chunks at the first storage system that is deployed local to the computing system that implements a data platform; and
restore a selected copy of the one or more copies from the one or more second storage systems by, for each chunk of the selected copy:
identify, from the chunk metadata, a data access cost for restoring a matching chunk, that matches the chunk, from the first storage system, wherein the data access cost differs based on whether restoring the matching chunk from the first storage system includes a data egress from a different computing system to the computing system;
determine whether the data access cost for restoring the matching chunk from the first storage system is lower than that of restoring the chunk from the one or more second storage system;
responsive to determining that the data access cost for restoring the matching chunk from the first storage system is lower than that of restoring the chunk from the second storage system, retrieve the matching chunk from the first storage system, wherein the matching chunk is included in the selected copy; and
responsive to determining that the data access cost for restoring the matching chunk from the first storage system is not lower than that of restoring the chunk from the second storage system, retrieve the chunk from the one or more second storage systems, wherein the chunk is included in the selected copy.
10 . The computing system of claim 9 , wherein the processing circuitry further executes the instructions to, after restoring the selected copy, store the selected copy on a storage system selected from one or more of the first storage system and the one or more second storage systems.
11 . The computing system of claim 9 , wherein each of at least a subset of the one or more copies represents a backup of the object for a particular time.
12 . The computing system of claim 9 , wherein the one or more second storage systems are deployed to the different computing system.
13 . The computing system of claim 9 , wherein each of the first storage system and the one or more second storage systems are provided by distinct cloud service providers.
14 . The computing system of claim 13 , wherein retrieving the chunk from the one or more second storage systems is associated with a higher cost than retrieving the matching chunk from the first storage system.
15 . The computing system of claim 13 , further comprising locating the chunk in the one or more second storage systems is associated with a higher cost than locating the matching chunk from the first storage system.
16 . The computing system of claim 9 , wherein the processing circuitry further executes the instructions to receive an indication of the selected copy via an input device.
17 . A non-transitory computer-readable storage medium comprising instructions that, when executed, cause processing circuitry of a computing system to:
store a first plurality of chunks storing data for an object of a file system and a second plurality of chunks storing data for a copy of the object, wherein the first plurality of chunks are stored on a first storage system and the second plurality of chunks are stored on one or more second storage systems;
store chunk metadata for the first plurality of chunks and the second plurality of chunks at the first storage system that is deployed local to the computing system that implements a data platform; and
restore a selected copy of the one or more copies from the one or more second storage systems by, for each chunk of the selected copy:
identify, from the chunk metadata, a data access cost for restoring a matching chunk, that matches the chunk, from the first storage system, wherein the data access cost differs based on whether restoring the matching chunk from the first storage system includes a data egress from a different computing system to the computing system;
determine whether the data access cost for restoring the matching chunk from the first storage system is lower than that of restoring the chunk from the one or more second storage system;
responsive to determining that the data access cost for restoring the matching chunk from the first storage system is lower than that of restoring the chunk from the second storage system, retrieve the matching chunk from the first storage system, wherein the matching chunk is included in the selected copy; and
responsive to determining that the data access cost for restoring the matching chunk from the first storage system is not lower than that of restoring the chunk from the second storage system, retrieve the chunk from the one or more second storage systems, wherein the chunk is included in the selected copy.
18 . The computer-readable storage medium of claim 17 , wherein, when further executed, the instructions cause the processing circuitry of the computing system to, after restoring the selected copy, store the selected copy on a storage system selected from one or more of the first storage system and the one or more second storage systems.