Cell-based backup for recovering from system failures in a multi-tenant computing system
A multi-tenant computing system provides services to a number of different tenant organizations. To address the problem of failure of portions of the system, the hardware infrastructure of the system is located at a number of different geographical locations. The various tenants are assigned to one of a set of “cells,” each cell corresponding to one of the geographical locations. Additionally, each cell has another one of the cells assigned to it as a backup cell, and the data of each cell is replicated within its assigned backup cell. At system run time, if a failure is detected within one of the cells, the network redirection is used within the multi-tenant system to reflect that the backup cell for the failing cell is now handling requests for the failing cell. Upon determination that the failing cell has been repaired and is now again correctly functioning, the network redirection is no longer employed, such that the (formerly) failing cell again handles its own requests.
1. A computer-implemented method for failure recovery in a distributed multi-tenant system comprising a plurality of tenants, each tenant being an organization having a plurality of users and having an associated server and database and belonging to a corresponding cell from a plurality of cells each comprising one or more servers and one or more databases, the computer-implemented method comprising:
for each cell of the plurality of cells:
assigning one of the plurality of cells to serve as a backup to the cell, the cell being geographically separate from the backup cell, the backup cell handling client requests for the cell while it is assigned as backup for the cell, and
replicating the databases of the tenants of the cell to the backup cell; and
receiving, from a user of a first one of the organizations, a request for single sign-on to a plurality of third-party applications to which the user has access via the organization;
responsive to detecting failure in a first cell of the plurality of cells responsible for performing the single sign-on:
identifying a first backup cell for the first cell;
using network redirection within the multi-tenant system to cause the request to be delivered to the first backup cell, such that the backup cell handles both requests for the first cell and requests for the first backup cell; and
the first backup cell transparently signing the user on to the plurality of third-party applications to which the user has access.
2. The computer-implemented method of claim 1 , further comprising:
responsive to detecting that the failure in the first cell has been repaired:
ceasing to use the network redirection, such that the first cell handles requests for the first cell and the first backup cell handles requests for the first backup cell.
3. The computer-implemented method of claim 1 , wherein each of the tenants is assigned to exactly one of the cells.
4. The computer-implemented method of claim 1 , further comprising detecting the failure in the first cell by computing a metric comprising at least one of average request latency for the first cell or request failure rate for the first cell.
5. The computer-implemented method of claim 1 , wherein using network redirection comprises modifying a DNS CNAME record.
6. A non-transitory computer-readable storage medium storing instructions for failure recovery in a distributed multi-tenant system comprising a plurality of tenants, each tenant being an organization having a plurality of users and having an associated server and database and belonging to a corresponding cell from a plurality of cells each comprising one or more servers and one or more databases, the instructions when executed by a computer processor performing actions comprising:
for each cell of the plurality of cells:
assigning one of the plurality of cells to serve as a backup to the cell, the cell being geographically separate from the backup cell, the backup cell handling client requests for the cell while it is assigned as backup for the cell, and
replicating the databases of the tenants of the cell to the backup cell; and
receiving, from a user of a first one of the organizations, a request for single sign-on to a plurality of third-party applications to which the user has access via the organization;
responsive to detecting failure in a first cell of the plurality of cells responsible for performing the single sign-on:
identifying a first backup cell for the first cell;
using network redirection within the multi-tenant system to cause the request to be delivered to the first backup cell, such that the backup cell handles both requests for the first cell and requests for the first backup cell; and
the first backup cell transparently signing the user on to the plurality of third-party applications to which the user has access.
7. The non-transitory computer-readable storage medium of claim 6 , the actions further comprising:
responsive to detecting that the failure in the first cell has been repaired:
ceasing to use the network redirection, such that the first cell handles requests for the first cell and the first backup cell handles requests for the first backup cell.
8. The non-transitory computer-readable storage medium of claim 6 , wherein each of the tenants is assigned to exactly one of the cells.
9. The non-transitory computer-readable storage medium of claim 6 , the actions further comprising detecting the failure in the first cell by computing a metric comprising at least one of average request latency for the first cell or request failure rate for the first cell.
10. The non-transitory computer-readable storage medium of claim 6 , wherein using network redirection comprises modifying a DNS CNAME record.
11. A multi-tenant system having a plurality of tenants, each tenant being an organization having a plurality of users and having an associated server and database and belonging to a corresponding cell from a plurality of cells each comprising one or more servers and one or more databases, the multi-tenant system comprising:
a computer processor; and
a non-transitory computer-readable storage medium storing instructions that when executed by the computer processor performing actions comprising:
for each cell of the plurality of cells:
assigning one of the plurality of cells to serve as a backup to the cell, the cell being geographically separate from the backup cell, the backup cell handling client requests for the cell while it is assigned as backup for the cell, and
replicating the databases of the tenants of the cell to the backup cell; and
receiving, from a user of a first one of the organizations, a request for single sign-on to a plurality of third-party applications to which the user has access via the organization;
responsive to detecting failure in a first cell of the plurality of cells responsible for performing the single sign-on:
identifying a first backup cell for the first cell;
using network redirection within the multi-tenant system to cause the request to be delivered to the first backup cell, such that the backup cell handles both requests for the first cell and requests for the first backup cell; and
the first backup cell transparently signing the user on to the plurality of third-party applications to which the user has access.
12. The multi-tenant system of claim 11 , the actions further comprising:
responsive to detecting that the failure in the first cell has been repaired:
ceasing to use the network redirection, such that the first cell handles requests for the first cell and the first backup cell handles requests for the first backup cell.
13. The multi-tenant system of claim 11 , wherein each of the tenants is assigned to exactly one of the cells.
14. The multi-tenant system of claim 11 , the actions further comprising detecting the failure in the first cell by computing a metric comprising at least one of average request latency for the first cell or request failure rate for the first cell.
15. The multi-tenant system of claim 11 , wherein using network redirection comprises modifying a DNS CNAME record.