Techniques for scalable distributed system backups
Techniques discussed herein manage backups of a service cell (SC). Each SC may include a data plane that is isolated from other SCs and comprises a distributed computing cluster (a cluster). A manifest that specifies one or more backup policies may be used to generate a full backup or a partial backup of a data set stored by the cluster. In accordance with the manifest, a signal may be sent to nodes of the cluster. In response, the nodes may transmit locally-stored data (e.g., data segments) to specified locations at a remote storage. The system may maintain a mapping of which segments correspond to data that was stored in the cluster at a time corresponding to a full or partial backup.
1. A computer-implemented method, comprising:
implementing an isolated hosting environment of a cloud-computing environment, the isolated hosting environment comprising a management plane and a data plane that are isolated from other hosting environments, the isolated hosting environment comprising a distributed computing cluster configured to operate in the data plane;
obtaining, by a computing component of the data plane, a manifest that specifies one or more backup policies related to generating at least one of a full backup or a partial backup of a data set stored by the distributed computing cluster;
in accordance with the one or more policies, transmitting one or more requests from the computing component to one or more respective nodes of the distributed computing cluster, the one or more respective nodes individually being configured to transmit one or more segments stored at a respective node to a remote storage location in response to a respective request;
obtaining, by the computing component of the data plane from a first node of the one or more nodes, a first segment identifier identifying a first segment that is stored at the first node and at the remote storage location;
obtaining, by the computing component of the data plane from a second node of the one or more nodes, a second segment identifier identifying a second segment that is stored at the second node and within the full backup; and
maintaining, by the computing component, a mapping comprising an association between the first segment identifier, the second segment identifier, and an indicator indicating the association relates to the full backup, wherein the mapping enables the full backup to be reconstructed using the association between the first segment identifier and the second segment identifier.
2. The computer-implemented method of claim 1 , further comprising:
initiating the partial backup of the data set based at least in part on the manifest;
identifying a set of segments already stored at the remote storage location;
obtaining, by the computing component from the first node, a plurality of segment identifiers identifying a plurality of segments of the data set that are stored at the first node;
determining, by the computing component, a subset of segments from the plurality of segments, the subset of segments excluding the set of segments already stored at the remote storage location, the subset of segments comprising changes in particular data stored at the first node after the first backup request;
transmitting, to the first node, a second backup request requesting storage of the subset of segments at the remote storage location, the first node being configured to store the subset of segments at the remote storage location in response to the second backup request; and
maintaining, by the computing component, an additional mapping to map at least one or more segment identifiers corresponding to the subset of segments to another indicator that indicates the additional mapping relates to the partial backup of the data set.
3. The computer-implemented method of claim 1 , wherein the computing component obtains the manifest from the management plane based at least in part on requesting the manifest from the management plane.
4. The computer-implemented method of claim 1 , wherein the remote storage location is remote with respect to the isolated hosting environment.
5. The computer-implemented method of claim 1 , further comprising:
transmitting, to the management plane, feedback data indicating that the full backup was completed; and
receiving, from the management plane, a subsequent request for either another full backup or another partial backup, the management plane being configured to determine, based at least in part on the manifest set and historical feedback data, that a backup has been missed.
6. The computer-implemented method of claim 1 , wherein the one or more nodes execute a search indexing engine configured to identify segments of data stored at a given node.
7. The computer-implemented method of claim 1 , wherein the remote storage location comprises a super set of the data locally stored at each of the nodes of the distribute computing cluster.
8. A cloud-computing system, comprising:
one or more processors; and
one or more memories storing computer-executable instructions that, when executed with the one or more processors, cause the cloud-computing system to:
implement an isolated hosting environment of a cloud-computing environment, the isolated hosting environment comprising a management plane and a data plane that are isolated from other hosting environments, the isolated hosting environment comprising a distributed computing cluster configured to operate in the data plane;
obtain, by a computing component of the data plane, a manifest that specifies one or more backup policies related to generating at least one of a full backup or a partial backup of a data set stored by the distributed computing cluster;
in accordance with the one or more policies, transmit one or more requests from the computing component to one or more respective nodes of the distributed computing cluster, the one or more respective nodes individually being configured to transmit one or more segments stored at a respective node to a remote storage location in response to a respective request;
obtain, by the computing component of the data plane from a first node of the one or more nodes, a first segment identifier identifying a first segment that is stored at the first node and at the remote storage location;
obtain, by the computing component of the data plane from a second node of the one or more nodes, a second segment identifier identifying a second segment that is stored at the second node and within the full backup; and
maintain, by the computing component, a mapping comprising an association between the first segment identifier, the second segment identifier, and an indicator indicating the association relates to the full backup, wherein the mapping enables the full backup to be reconstructed using the association between the first segment identifier and the second segment identifier.
9. The computing system of claim 8 , wherein executing the instructions further causes the computing system to:
initiate the partial backup of the data set based at least in part on the manifest;
identify a set of segments already stored at the remote storage location;
obtain, by the computing component from the first node, a plurality of segment identifiers identifying a plurality of segments of the data set that are stored at the first node;
identify, by the computing component, a subset of segments from the plurality of segments, the subset of segments excluding the set of segments already stored at the remote storage location, the subset of segments comprising changes in particular data stored at the first node after the first backup request;
transmit, to the first node, a second backup request requesting storage of the subset of segments at the remote storage location, the first node being configured to store the subset of segments at the remote storage location in response to the second backup request; and
maintain, by the computing component, an additional mapping to map at least one or more segment identifiers corresponding to the subset of segments to another indicator that indicates the additional mapping relates to the partial backup of the data set.
10. The computing system of claim 8 , wherein the computing component obtains the manifest from the management plane based at least in part on requesting the manifest from the management plane.
11. The computing system of claim 8 , wherein the remote storage location is remote with respect to the isolated hosting environment.
12. The computing system of claim 8 , wherein executing the instructions further causes the computing system to:
transmit, to the management plane, feedback data indicating that the full backup was completed; and
receive, from the management plane, a subsequent request for either another full backup or the partial backup, the management plane being configured to determine, based at least in part on the manifest set and historical feedback data, that a backup has been missed.
13. The computing system of claim 8 , wherein the one or more nodes execute a search indexing engine configured to identify segments of data stored at a given node.
14. The computing system of claim 8 , wherein the remote storage location comprises a super set of the data locally stored at each of the nodes of the distribute computing cluster.
15. A non-transitory computer-readable storage medium comprising executable instructions that, when executed with one or more processors of a cloud-computing system, cause the cloud-computing system to:
implement an isolated hosting environment of a cloud-computing environment, the isolated hosting environment comprising a management plane and a data plane that are isolated from other hosting environments, the isolated hosting environment comprising a distributed computing cluster configured to operate in the data plane;
obtain, by a computing component of the data plane, a manifest that specifies one or more backup policies related to generating at least one of a full backup or a partial backup of a data set stored by the distributed computing cluster;
in accordance with the one or more policies, transmit one or more requests from the computing component to one or more respective nodes of the distributed computing cluster, the one or more respective nodes individually being configured to transmit one or more segments stored at a respective node to a remote storage location in response to a respective request;
obtain, by the computing component of the data plane from a first node of the one or more nodes, a first segment identifier identifying a first segment that is stored at the first node and at the remote storage location;
obtain, by the computing component of the data plane from a second node of the one or more nodes, a second segment identifier identifying a second segment that is stored at the second node and within the full backup; and
maintain, by the computing component, a mapping comprising an association between the first segment identifier, the second segment identifier, and an indicator indicating the association relates to the full backup, wherein the mapping enables the full backup to be reconstructed using the association between the first segment identifier and the second segment identifier.
16. The non-transitory computer-readable storage medium of claim 15 , wherein executing the instructions further causes the computing system to:
initiate the partial backup of the data set based at least in part on the manifest;
identify a set of segments already stored at the remote storage location;
obtain, by the computing component from the first node, a plurality of segment identifiers identifying a plurality of segments of the data set that are stored at the first node;
identify, by the computing component, a subset of segments from the plurality of segments, the subset of segments excluding the set of segments already stored at the remote storage location, the subset of segments comprising changes in particular data stored at the first node after the first backup request;
transmit, to the first node, a second backup request requesting storage of the subset of segments at the remote storage location, the first node being configured to store the subset of segments at the remote storage location in response to the second backup request;
and maintain, by the computing component, an additional mapping to map at least one or more segment identifiers corresponding to the subset of segments to another indicator that indicates the additional mapping relates to the partial backup of the data set.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the remote storage location is remote with respect to the isolated hosting environment.
18. The non-transitory computer-readable storage medium of claim 15 , wherein executing the instructions further causes the computing system to:
transmit, to the management plane, feedback data indicating that the full backup was completed; and
receive, from the management plane, a subsequent request for either another full backup or the partial backup, the management plane being configured to determine, based at least in part on the manifest set and historical feedback data, that a backup has been missed.
19. The non-transitory computer-readable storage medium of claim 15 , wherein the one or more nodes execute a search indexing engine configured to identify segments of data stored at a given node.
20. The non-transitory computer-readable storage medium of claim 15 , wherein the remote storage location comprises a super set of the data locally stored at each of the nodes of the distribute computing cluster.