Disaster recovery in a cell model for an extensibility platform
According to one or more embodiments, an example method herein comprises: operating in a particular cell of a multi-celled architecture for an extensibility platform, the particular cell having a plurality of datastores with stored data; determining a backup schedule for the plurality of datastores; sending commands to individual datastore backup controllers based on the backup schedule to cause the individual datastore backup controllers to produce backups of the stored data to a respective local backup volume; and sending, in response to receiving an indication of completion of the backups, the backups of the respective local backup volume to a cloud storage service.
1 . A method, comprising:
operating, by a backup manager, in a particular cell of a multi-celled architecture for an extensibility platform, the particular cell having a plurality of datastores with stored data, wherein the stored data comprises configuration data and observability data;
determining, by the backup manager, a backup schedule for the plurality of datastores, wherein the backup schedule identifies, for each respective datastore, which of the configuration data and observability data are to be included in a backup;
sending, from the backup manager, commands to individual datastore backup controllers based on the backup schedule to cause the individual datastore backup controllers to produce backups of the stored data to a respective local backup volume; and
sending, from the backup manager and in response to receiving an indication of completion of the backups, the backups of the respective local backup volume to a cloud storage service.
2 . The method as in claim 1 , further comprising:
restoring particular data from a particular backup from the cloud storage service; and
moving the particular backup to a particular local backup volume, wherein a particular datastore backup controller writes the particular backup to a particular datastore to restore the particular data.
3 . The method as in claim 1 , wherein the backup schedule is determined to schedule independent backups for the configuration data and the observability data.
4 . The method as in claim 1 , wherein the backup schedule is determined to schedule backups for only the configuration data.
5 . The method as in claim 1 , wherein observability data comprises one or more of metrics, events, logs, or traces.
6 . The method as in claim 1 , further comprising:
receiving configuration on how many incremental backups to store on the cloud storage service.
7 . The method as in claim 1 , further comprising:
providing a history of backups.
8 . The method as in claim 1 , further comprising:
cleaning up aged backups in the cloud storage service.
9 . The method as in claim 1 , wherein the backup schedule is determined to stagger backups among the plurality of datastores.
10 . The method as in claim 1 , further comprising:
determining whether to command an incremental backup or a full backup.
11 . The method as in claim 10 , wherein an incremental backup is commanded when a full backup exists, and a full backup is commanded when a full backup does not yet exist.
12 . The method as in claim 1 , further comprising:
receiving the indication as a domain event.
13 . The method as in claim 1 , further comprising:
sending commands on a domain event bus.
14 . The method as in claim 1 , further comprising:
performing restoration testing and validation on backups.
15 . A tangible, non-transitory, computer-readable medium having computer-executable instructions stored thereon that, when executed by a processor on a computer, cause the computer to perform a backup manager process comprising:
operating in a particular cell of a multi-celled architecture for an extensibility platform, the particular cell having a plurality of datastores with stored data, wherein the stored data comprises configuration data and observability data;
determining a backup schedule for the plurality of datastores, wherein the backup schedule identifies, for each respective datastore, which of the configuration data and observability data are to be included in a backup;
sending commands to individual datastore backup controllers based on the backup schedule to cause the individual datastore backup controllers to produce backups of the stored data to a respective local backup volume; and
sending, in response to receiving an indication of completion of the backups, the backups of the respective local backup volume to a cloud storage service.
16 . The tangible, non-transitory, computer-readable medium as in claim 15 , wherein the backup manager process further comprises:
restoring particular data from a particular backup from the cloud storage service; and
moving the particular backup to a particular local backup volume, wherein a particular datastore backup controller writes the particular backup to a particular datastore to restore the particular data.
17 . The tangible, non-transitory, computer-readable medium as in claim 15 , wherein the backup schedule is determined to schedule independent backups for the configuration data and the observability data.
18 . An apparatus, comprising:
one or more network interfaces to communicate with a network;
a processor coupled to the one or more network interfaces and configured to execute one or more processes; and
a memory configured to store a backup manager process that is executable by the processor, the backup manager process, when executed, configured to:
operate in a particular cell of a multi-celled architecture for an extensibility platform, the particular cell having a plurality of datastores with stored data, wherein the stored data comprises configuration data and observability data;
determine a backup schedule for the plurality of datastores, wherein the backup schedule identifies, for each respective datastore, which of the configuration data and observability data are to be included in a backup;
send commands to individual datastore backup controllers based on the backup schedule to cause the individual datastore backup controllers to produce backups of the stored data to a respective local backup volume; and
send, in response to receiving an indication of completion of the backups, the backups of the respective local backup volume to a cloud storage service.