Data placement and sharding
Data sharding partitions data associated with an application into multiple shards in which each shard stores a subset of the data. A shard is further divided into multiple microshards in which each microshard contains a subset of the shard data. A set of microshards are assigned to a specified shard. The assignment of microshards can change, e.g., when new shards are added and microshards are migrated from one shard to another shard. A microshard map, which indicates a mapping between the microshards and shards, is updated automatically when the assignments change. After identifying the shard to which a specified microshard is assigned using the microshard map, a data access request from a client device is automatically forwarded to the appropriate shard. Because data can be accessed efficiently regardless of whether the number of shards changes, the data sharding facilitates provisioning of the shards on a need basis.
1. A computer-implemented method comprising:
storing data associated with an application across multiple shards, wherein:
each of the multiple shards stores a subset of the data and is hosted by at least one of multiple server devices;
the subset of the data within each of the multiple shards is stored across multiple microshards; and
each of the multiple microshards is associated with a unique microshard identifier;
defining, by a shard manager component, a function involving at least one mathematical operation, wherein the function:
takes a specific microshard identifier of a specific microshard; and
outputs a shard identifier of the multiple shards to which the specific microshard has been assigned;
transmitting the function to multiple client devices;
managing, in response to a triggering event, a migration of the specific microshard from the first one of the multiple shards to a second one of the multiple shards;
receiving, at one of the multiple client devices during the migration, a data access request, the data access request specifying the specific microshard identifier;
using, by the one of the multiple client devices during the migration, the specific microshard identifier and the function to identify the shard identifier of the first one of the multiple shards; and
forwarding, by the one of the multiple client devices during the migration, the data access request to a specific one of the multiple server devices hosting the first one of the multiple shards.
2. The computer-implemented method of claim 1 , further comprising:
receiving, by the one of the multiple client devices after forwarding the data access reques, a write-unavailable message from the specific one of the multiple server devices; and
updating, by the one of the multiple client devices in response to the write-unavailable message, a status of the migration shared by the multiple client devices to indicate that the migration is in process.
3. The computer-implemented method of claim 2 , wherein updating the status of the migration comprises updating, by the one of the multiple client devices, a memcache to reflect that the specific microshard is not available for writing.
4. The computer-implemented method of claim 3 , wherein the memcache stores statuses of microshards that are being migrated, and wherein the memcache is a storage system shared between the multiple client devices.
5. The computer-implemented method of claim 3 further comprising:
receiving, during the migration, an additional data access request, wherein the additional data access request specifies the specific microshard identifier;
querying the memcache using the specific microshard identifier to determine that the specific microshard is still in transit from the first one of the multiple shards to the second one of the multiple shards; and
batching the data access request and the additional data access request for later forwarding to the second one of the multiple shards.
6. The computer-implemented method of claim 2 , further comprising storing the data access request for later forwarding to the second one of the multiple shards.
7. The computer-implemented method of claim 1 , further comprising:
determining that the data access request is a request to write data to the specific microshard; and
using the specific microshard identifier to determine that a status of the migration is set to not started, wherein the data access request is forwarded to the first one of the multiple shards in response to the status of the migration being set to not started.
8. The computer-implemented method of claim 1 , further comprising:
using, by the one of the multiple client devices, the specific microshard identifier to determine that a status of the migration is set to completed; and
forwarding the data access request to the second one of the multiple shards.
9. The computer-implemented method of claim 1 , wherein managing the migration of the specific microshard from the first one of the multiple shards to the second one of the multiple shards comprises instructing, in response to a completion of the migration, the shard manager component to define an additional function involving the at least one mathematical operation, wherein the additional function:
takes the specific microshard identifier of the specific microshard; and
outputs a shard identifier of the second one of the multiple shards.
10. The computer-implemented method of claim 1 , further comprising, in response to a determination that the data access request is a write request and a status of the migration is not started, writing, at the first one of the multiple shards, specified data to the specific microshard.
11. The computer-implemented method of claim 1 , further comprising, in response to a determination that the data access request is a read request and a status of the migration is not started or in process, retrieving, at the first one of the multiple shards, requested data from the specific microshard.
12. A system, comprising:
multiple server devices, each comprising at least one processor, that store data associated with an application across multiple shards, wherein:
each of the multiple shards stores a subset of the data and is hosted by at least one of the multiple server devices;
the subset of the data within each of the multiple shards is stored across multiple microshards; and
each of the multiple microshards is associated with a unique microshard identifier;
a shard manager component that:
manages placement of the data across the multiple shards; and
defines a function involving at least one mathematical operation, wherein the function:
takes a specific microshard identifier of a specific microshard; and
outputs a shard identifier of a first one of the multiple shards to which the specific microshard has been assigned; and
transmits the function to multiple client devices;
a migration controller component configured to manage, in response to a triggering event, a migration of the specific microshard from the first one of the multiple shards to a second one of the multiple shards; and
a mapping component that:
receives, at one of the multiple client devices during the migration, a data access request, the data access request specifying the specific microshard identifier;
uses, at the one of the multiple client devices during the migration, the specific microshard identifier and the function to identify the shard identifier of the first one of the multiple shards; and
forwards, from the one of the multiple client devices during the migration, the data access request to a specific one of the multiple server devices hosting the first one of the multiple shards.
13. The system of claim 12 , wherein
the mapping component further:
receives, from the specific one of the multiple server devices, a write-unavailable message; and
updates, in response to the write-unavailable message, a status of the migration to indicate that the migration is in process.
14. The system of claim 13 further comprising a memcache, wherein the mapping component updates the status of the migration by notifying the memcache that the specific microshard is not available for writing.
15. The system of claim 14 , wherein the memcache stores statuses of microshards that are being migrated, and wherein the memcache is a storage system shared between the multiple client devices.
16. The system of claim 13 , wherein the mapping component further:
receives, during the migration, an additional data access request, wherein the additional data access request specifies the specific microshard identifier;
uses the specific microshard identifier to determine that the specific microshard is still in transit from the first one of the multiple shards to the second one of the multiple shards; and
batches the data access request and the additional data access request for later forwarding to the second one of the multiple shards.
17. A non-transitory computer-readable storage medium storing computer-readable instructions, comprising:
instruction for storing data associated with an application across multiple shards, wherein:
each of the multiple shards stores a subset of the data and is hosted by at least one of multiple server devices;
the subset of the data within each of the multiple shards is stored across multiple microshards; and
each of the multiple microshards is associated with a unique microshard identifier;
instruction for defining, by a shard manager component, a function involving at least one mathematical operation, wherein the function:
takes a specific microshard identifier of a specific microshard; and
outputs a shard identifier of a first one of the multiple shards to which the specific microshard has been assigned;
instructions for transmitting the function to multiple client devices;
instruction for managing, in response to a triggering event, a migration of the specific microshard from the first one of the multiple shards to a second one of the multiple shards;
instruction for receiving, at one of the multiple client devices during the migration, a data access request, the data access request specifying the specific microshard identifier;
instruction for using, by the one of the multiple client devices during the migration, the specific microshard identifier and the function to identify the shard identifier of the first one of the multiple shards; and
instructions for forwarding, by the one of the multiple client devices during the migration, the data access request to a specific one of the multiple server devices hosting the first one of the multiple shards.
18. The computer-implemented method of claim 1 further comprising:
defining, by the shard manager component in response to the triggering event, an additional function involving the at least one mathematical operation, wherein the additional function:
takes the specific microshard identifier of the specific microshard; and
outputs a shard identifier of the second one of the multiple shards;
transmitting the additional function to the multiple client devices;
receiving an additional data access request, wherein the additional data access request specifies the specific microshard identifier;
using the specific microshard identifier and the additional function to identify the shard identifier of the second one of the multiple shards; and
forwarding the additional data access request to a specific one of the multiple server devices hosting the second one of the multiple shards.
19. The computer-implemented method of claim 1 , wherein the triggering event comprises one or more of:
an addition of a new shard; or
a removal of one of the multiple shards.
20. The computer-implemented method of claim 1 , wherein the triggering event comprises an increase in data traffic on one or more of the multiple shards.