Eventual consistency in a deduplicated cloud storage system
One example method includes receiving a write request that includes a data structure version to be written, wherein the data structure version is associated with a unique identifier, storing the data structure version in association with the unique identifier, receiving a read request for a most recent version of the data structure and, when the stored data structure version is not the most recent version of the data structure, examining respective unique identifiers of each of a group of other stored data structure versions to determine which stored data structure version is the most recent. Finally, the example method includes returning the most recent data structure version, notwithstanding that one or more other data structure versions existed at the time that the read request was received.
1. A system, comprising:
one or more hardware processors; and
a computer readable storage medium having stored therein computer-executable instructions which, when executed by the one or more hardware processors, perform the following operations:
receiving a write request that includes a data structure version to be written, wherein the data structure version is assigned a unique identifier that is generated based on a hash of contents of the data structure, the data structure comprising a similarity group that records a mapping between a compression region comprising a compressed group of data portions, and fingerprints of the data portions that make up the compression region;
appending, to a name of the similarity group, a transaction ID that is incremented for each update that is made to the similarity group;
storing the data structure version in association with the unique identifier;
receiving a read request for a most recent version of the data structure;
when the stored data structure version is not the most recent version of the data structure, examining respective unique identifiers of each of a plurality of other stored data structure versions to determine which stored data structure version is the most recent; and
returning the most recent data structure version, notwithstanding that one or more other data structure versions existed at the time that the read request was received.
2. The system as recited in claim 1 , wherein the read request is received from a deduplication microservice.
3. The system as recited in claim 1 , wherein when the size of the similarity group exceeds a threshold, the operations further comprise creating a new subgroup having a subgroup ID that has a relatively higher value than a prior in time subgroup ID.
4. The system as recited in claim 1 , wherein as between the similarity group and one or more other similarity groups, the operations further comprise performing data deduplication against the similarity group with the highest subgroup ID and, within that highest subgroup ID, the highest transaction ID.
5. The system as recited in claim 1 , wherein namespace consistency is achieved before the stored data structure version is available to fulfill the read request.
6. The system as recited in claim 1 , wherein the compression region is shared by multiple objects.
7. The system as recited in claim 1 , wherein the operations are performed in a data protection environment that implements an eventual consistency model.
8. The system as recited in claim 1 , wherein determining which stored data structure version is the most recent comprises performing a list call that includes a data structure name prefix that is common to two or more of the stored data structure versions.
9. The system as recited in claim 1 , wherein the unique identifier of a data structure version prevents that data structure version from being overwritten by another data structure version with a different unique identifier.
10. The system as recited in claim 1 , wherein the operations further comprise deleting all but the newest data structure version.
11. The system as recited in claim 1 , wherein as between successive unique identifiers, a second one of the successive unique identifiers reflects a time increment relative to a first one of the successive unique identifiers.
12. The system as recited in claim 1 , wherein as between a second unique identifier that succeeds a first unique identifier, the second unique identifier has a value that is greater than a value of the first unique identifier.
13. The system as recited in claim 1 , wherein the unique identifier associated with the data structure version included in the write request is assigned either by a component of a datacenter, or by an entity that issued the write request.
14. The system as recited in claim 1 , wherein one or more of the data structure versions comprise deduplicated data.