Distributed database configuration
Replicas are selected in a large distributed network, and the roles for these replicas are identified. In one example, a leader is selected from among candidate computing dusters. To make this selection, an activity monitor predicts or monitors the workload of one or more clients. Different activities of the workload are given corresponding weights. The delay in performing requested activities, modified by these weights is found, and the candidate leader with the lowest weighted delay is selected as the leader.
1. A system comprising:
a plurality of computing clusters each comprising computer memory and a computer processor;
a distributed database running on one or more of the plurality of the computing clusters and that interacts with a client application running on a client computer, the distributed database configured to:
store data of the distributed database in shards distributed among computing clusters of the distributed database; and
scale the one or more of the plurality of computing clusters that interact with the client application running on the client computer based on a delay threshold;
an activity monitor service configured to:
monitor interactions between the client application and the distributed database; and
generate, from the monitoring of the interactions between the client application and the distributed database, workload data describing the interactions between the client application and the distributed database; and
determine, for each respective computing cluster of the plurality of computing clusters, using the workload data, a delay in performing a particular operation for the client application from among a plurality of operations available to each respective computing cluster; and
a task assigning service configured to:
determine that the workload data satisfies the delay threshold corresponding to the delay of each computing cluster of the plurality of computing clusters in performing the particular operation for the client application; and
in response to determining that the workload data satisfies the delay threshold corresponding to the delay of each computing cluster of the plurality of computing clusters in performing the particular operation for the client application, assign a particular subset computing of the plurality of computing clusters a replica role in the distributed database that allows the particular subset of the computing clusters to provide redundancy to satisfy a redundancy request from a client associated with the client application.
2. The system of claim 1 , wherein the task assigning service is further configured to assign, based on the workload data, a second computing cluster to a leader role in the distributed database, wherein assigning the second computing cluster to a leader role comprises identifying a respective computing cluster having a lowest delay in performing the particular operation for the client application.
3. The system of claim 2 , wherein to identify the respective cluster having the lowest delay in performing the particular operation for the client application, the task assigning service is configured to:
identify a frequency for the particular operation;
identify a delay value of the particular operation for each respective computing cluster; and
find, for each respective computing cluster, a weighted delay value that incorporates the frequency of the particular operation.
4. The system of claim 2 , wherein the lowest delay corresponds to a tail delay based on a delay distributed between two or more computing clusters of the plurality of computing clusters, the delay distribution defined by a round-trip time between the two or more computing clusters of the plurality of computing clusters.
5. The system of claim 1 , wherein generating, from the monitoring of the interactions between the client application and the distributed database, the workload data describing the interactions between the client application and the distributed database comprises:
receiving historical data comprising historical interactions between the client and the distributed database; and
generating predicted workload data describing the interactions between the client application and the distributed database based on the received historical data.
6. The system of claim 1 , wherein the plurality of computing clusters execute computing tasks configured to create virtual machines.
7. The system of claim 1 , wherein assigning the particular subset of the plurality of computing clusters the replica role in the distributed database occurs subsequent to the task assigning service assigning a second computing cluster to a leader role in the distributed database.
8. The system of claim 1 , wherein the activity monitor service and the task assigning service are running on one or more of the computing clusters.
9. The system of claim 1 , wherein one of the computing clusters of the plurality of computing clusters comprises the client computer.
10. A method comprising:
monitoring interactions between a client application and a distributed database, wherein the distributed database runs on at least one or more of computing clusters and interacts with a client application running on a client computer, wherein each computing cluster comprises computer memory and a computer processor, wherein the distributed database is configured to store data of the distributed database in shards distributed among computing clusters of the distributed database and scale the one or more of the plurality of computing clusters that interact with the client application running on the client computer based on a delay threshold;
generating, from the monitoring of the interactions between the client application and the distributed database, workload data describing the interactions between the client application and the distributed database;
determining, for each respective computing cluster of the plurality of computing clusters, using the workload data, a delay in performing a particular operation for the client application from among a plurality of operations available to each respective computing cluster;
determining that the workload data satisfies the delay threshold corresponding to the delay of each computing cluster of the plurality of computing clusters in performing the particular operation for the client application; and
in response to determining that the workload data satisfies the delay threshold corresponding to the delay of each computing cluster of the plurality of computing clusters in performing the particular operation for the client application, assigning a particular subset of the plurality of computing clusters a replica role in the distributed database that allows the particular subset of the plurality of computing clusters to provide redundancy to satisfy a redundancy request from a client associated with the client application.
11. The method of claim 10 , further comprising:
determining a respective computing cluster having a lowest delay in performing the particular for the client application; and
assigning, based on the determination of the respective computing cluster having the lowest delay in performing the particular operation for the client application, a second computing cluster to a leader role in the distributed database.
12. The method of claim 11 , wherein determining the respective computing cluster having the lowest delay in performing the particular operation for the client application comprises:
identifying a frequency for the particular operation;
identifying a delay value of the particular operation for each respective computing cluster; and
finding, for each respective computing cluster, a weighted delay value that incorporates the frequency of the particular operation.
13. The method of claim 11 , wherein the lowest delay corresponds to a tail delay based on a delay distributed between two or more computing clusters of the plurality of computing clusters, the delay distribution defined by a round-trip time between the two or more computing clusters of the plurality of computing clusters.
14. The method of claim 10 , wherein generating, from the monitoring of the interactions between the client application and the distributed database, the workload data describing the interactions between the client application and the distributed database comprises:
receiving historical data comprising historical interactions between the client and the distributed database; and
generating predicted workload data describing the interactions between the client application and the distributed database based on the received historical data.
15. The method of claim 10 , wherein the plurality of computing clusters execute computing tasks configured to create virtual machines.
16. The method of claim 10 , wherein assigning the particular subset of the plurality of computing clusters the replica role in the distributed database occurs subsequent to a task assigning service assigning a second computing cluster to a leader role in the distributed database.
17. The method of claim 10 , wherein generating the workload data describing the interactions between the client application and the distributed database occurs at an activity monitor service and wherein determining that the workload data satisfies the delay threshold corresponding to the delay of each computing cluster of the plurality of computing clusters in performing the particular operation for the client application occurs at a task assigning service, and wherein the activity monitor service and the task assigning service are executing on one or more of the computing clusters.
18. The method of claim 10 , wherein one of the computing clusters of the plurality of computing clusters comprises the client computer.