Systems and methods for database zone sharding and API integration
Systems and methods are provided to enable control and placement of data repositories. In some embodiments, the system segments data into zones. A website, for example, may need to segment data according to location. In this example, a zone may be created for North America and another zone may be created for Europe. Data related to operations executed in North America, for example, can be placed in the North America zone and data related to transactions in Europe can be placed in the Europe zone. According to some embodiments, the system may use zones to accommodate a range of deployment scenarios.
1. A database management system comprising:
a distributed database comprising a plurality of shards storing a plurality of data items;
at least one processor configured to:
determine a first range of shard key values;
analyze operations on the distributed database and select a target shard to optimize a system hosting the target shard or portion of the target shard for storage of data associated with the first range of shard key values;
map the first range of shard key values to the target shard or the portion of the target shard;
route a first data item to the target shard or the portion of the target shard, wherein the act of routing comprises:
mapping the first data item to the first range of shard key values; and
responsive to the mapping, storing the first data item in the target shard or the portion of the target shard.
2. The database system of claim 1 , further comprising a first datacenter and a second datacenter and the at least one processor is configured to:
host a first node of the target shard or the portion of the target shard on the first datacenter; and
host a second node of the target shard or the portion of the target shard on the second datacenter.
3. The database system of claim 2 , wherein the at least one processor is configured to:
route an operation to the first node when the first datacenter is operational; and
route the operation to the second node when the first datacenter is not operational.
4. The database system of claim 2 , wherein:
the first datacenter is located in a first geographic region and the second datacenter is located in a second geographic region;
the first node is a primary node of the target shard or the portion of the target shard and the second node is a secondary node of the target shard or the portion of the target shard; and
the at least one processor is configured to:
determine that a majority of write requests over a period of time originated from the second geographic region and, in response, trigger an election setting the second node to the primary node of the target shard or the portion of the target shard.
5. The database system of claim 2 , wherein:
the first datacenter is located in a first geographic region and the second datacenter is located in a second geographic region;
the first node is a primary node of the target shard or the portion of the target shard and the second node is a secondary node of the target shard or the portion of the target shard; and
the at least one processor is configured to:
determine that a majority of read requests over a period of time originated from the first geographic region and, in response, trigger an election setting the first node to a secondary node of the target shard or the portion of the target shard.
6. The database system of claim 2 , wherein the first datacenter is located in a first geographic region and the second datacenter is located in a second geographic region, and the at least one processor is configured to:
route a first operation associated with the first geographic region to the first node hosted by the first datacenter; and
route a second operation associated with the second geographic region to the second node hosted by the second datacenter.
7. The database system of claim 1 , further comprising storage hardware that provides a first level of performance and storage hardware that provides a second level of performance, wherein the at least one processor is configured to:
store the target shard or the portion of the target shard on the storage hardware that provides the first level of performance;
determine that an operation requires the first level of performance; and
route the operation to the target shard or the portion of the target shard in response to determining that the operation requires the first level of performance.
8. The database system of claim 7 , wherein the at least one processor is configured to:
determine that a threshold number of data items stored on the target shard or the portion of the target shard are to be accessible with the second level of performance and, in response, migrate the target shard or the portion of the target shard to the storage hardware that provides the second level of performance for storage.
9. The database system of claim 8 , wherein the at least one processor is configured to:
determine that the threshold number of data items stored on the target shard or the portion of the target shard are to be accessible with the second level of performance based on times that at least some of the data items were generated.
10. A computer-implemented method of managing a distributed database, the method comprising:
storing a plurality of data items among a plurality of shards in the distributed database;
determining a first range of key values;
analyzing operations on the distributed database and selecting a target shard or portion of a target shard to optimize a system hosting the target shard or the portion of the target shard for storage of data associated with the first range of shard key values;
mapping the first range of shard key values to the target shard or the portion of the target shard;
routing a first data item to the target shard or the portion of the target shard, wherein the act of routing comprises:
mapping the first data item to the first range of shard key values; and
responsive to the mapping, storing the first data item in the target shard or the portion of the target shard.
11. The method of claim 10 , further comprising:
hosting a first node of the target shard or the portion of the target shard on a first datacenter; and
hosting a second node of the target shard or the portion of the target shard on a second datacenter.
12. The method of claim 11 , further comprising:
routing an operation to the first node when the first datacenter is operational; and
routing the operation to the second node when the first datacenter is not operational.
13. The method of claim 11 , wherein:
the first datacenter is located in a first geographic region and the second datacenter is located in a second geographic region; and
the method further comprises acts of:
setting the first node as a primary node of the target shard or the portion of the target shard and the second node as a secondary node of the target shard or the portion of the target shard; and
determining that a majority of write requests over a period of time originated from the second geographic region and, in response, triggering an election setting the second node to the primary node of the target shard or the portion of the target shard.
14. The method of claim 11 , wherein:
the first datacenter is located in a first geographic region and the second datacenter is located in a second geographic region; and
the method further comprises acts of:
setting the first node as a primary node of the target shard or the portion of the target shard and the second node as a secondary node of the target shard or the portion of the target shard; and
determining that a majority of read requests over a period of time originated from the first geographic region and, in response, triggering an election setting the first node to a secondary node of the target shard or the portion of the target shard.
15. The method of claim 11 , wherein the first datacenter is located in a first geographic region and the second datacenter is located in a second geographic region, and the method further comprises acts of:
routing a first operation associated with the first geographic region to the first node hosted by the first datacenter; and
routing a second operation associated with the second geographic region to the second node hosted by the second datacenter.
16. The method of claim 10 , further comprising:
storing data in storage hardware that provides a first level of performance and data in storage hardware that provides a second level of performance;
storing the target shard or the portion of the target shard on the storage hardware that provides the first level of performance;
determining that an operation requires the first level of performance; and
routing the operation to the target shard or the portion of the target shard in response to determining that the operation requires the first level of performance.
17. The method of claim 16 , further comprising:
determining that a threshold number of data items stored on the target shard or the portion of the target shard are to be accessible with the second level of performance and, in response, migrating the target shard or the portion of the target shard to the storage hardware that provides the second level of performance.
18. At least one non-transitory computer-readable storage medium storing instructions, the instructions, when executed by at least one processor, cause the at least one processor to perform a method comprising:
storing a plurality of data items among a plurality of shards in the distributed database;
determining a first range of key values;
analyzing operations on the distributed database and selecting a target shard or portion of the target shard to optimize a system hosting the target shard or the portion of the target shard for storage of data associated with the first range of shard key values;
mapping the first range of shard key values to the target shard or the portion of the target shard;
routing a first data item to the target shard or the portion of the target shard, wherein the act of routing comprises:
mapping the first data item to the first range of shard key values; and
responsive to the mapping, storing the first data item in the target shard or the portion of the target shard.
19. The method of claim 18 , further comprising:
hosting a first node of the target shard or the portion of the target shard on a first datacenter located in a first geographic region;
hosting a second node of the target shard or the portion of the target shard on a second datacenter located in a second geographic region;
setting the first node as a primary node of the target shard or the portion of the target shard and the second node as a secondary node of the target shard or the portion of the target shard; and
determining that a majority of write requests over a period of time originated from the second geographic region and, in response, triggering an election setting the second node to the primary node of the target shard or the portion of the target shard.
20. The method of claim 18 , further comprising:
hosting a first node of the target shard or the portion of the target shard on a first datacenter located in a first geographic region;
hosting a second node of the target shard or the portion of the target shard on a second datacenter located in a second geographic region;
setting the first node as a primary node of the target shard or the portion of the target shard and the second node as a secondary node of the target shard or the portion of the target shard; and determining that a majority of read requests over a period of time originated from the first geographic region and, in response, triggering an election setting the first node to a secondary node of the target shard or the portion of the target shard.