Efficient erasure coding of mid-size data objects
The disclosed technology generally describes a data protection scheme that for “mid-size” objects directly writes divided object data fragments, and performs erasure coding to directly write object coding fragments, to distributed storage locations in a node cluster. A storage container such as a chunk allocated for mid-size objects is distributed among the storage cluster nodes. When a mid-size object (e.g., between 24 megabytes and 128 megabytes) is to be created, the object data is divided into object data fragments and encoded into object coding fragments, with the data object fragments and object coding fragments written/appended to the distributed storage locations, without needing a preliminary protection scheme.
1. A system, comprising:
a processor, and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, the operations comprising:
obtaining, in a node cluster data storage system, object data corresponding to an object to be protected with an erasure coding scheme;
determining that the object data is of a size that is to be protected without a preliminary protection scheme; and
storing the object based on the erasure coding scheme, comprising dividing the object data into object data fragments, performing erasure coding on the object data to obtain object coding fragments, and writing the object data fragments and the object coding fragments to a group of data structure fragments distributed in the node cluster data storage system,
wherein the group of data structure fragments are associated with a distributed chunk data structure in the node cluster data storage system, and wherein storing the object comprises writing the object data fragments to a same offset location within the distributed chunk data structure.
2. The system of claim 1 , wherein the object to be protected is a new object to be created, and wherein the operations further comprise:
acknowledging creation of the new object after storing the new object.
3. The system of claim 1 , wherein the operations further comprise:
pre-allocating the group of data structure fragments.
4. The system of claim 1 , wherein the determining that the object data is of the size that is to be protected without the preliminary protection scheme comprises:
determining that the object data is of the size that is greater than or equal to a defined lower size limit.
5. The system of claim 4 , wherein the defined lower size limit is based on a defined segment size multiplied by a value representing a total number of the object data fragments.
6. The system of claim 5 , wherein the defined segment size is two megabytes and the number of object data fragments is twelve.
7. The system of claim 1 , wherein the determining that the object data is of the size that is to be protected without the preliminary protection scheme comprises:
determining that the object data is of the size that is less than a defined upper size limit.
8. The system of claim 7 , wherein the defined upper size limit is based on a chunk size of a chunk data structure corresponding to the group of data structure fragments.
9. The system of claim 1 , wherein the erasure coding scheme corresponds to a division of the object data into twelve object data fragments and an encoding of four object coding fragments.
10. A method, comprising:
obtaining, by a system comprising a processor in a node cluster data storage system, object data corresponding to an object to be created and stored in a protected state via erasure coding;
determining, based on the object data being determined to be of a size greater than or equal to a first size limit and below a second size limit greater than the first size limit, that the object data is to be stored in the protected state without a preliminary protection scheme; and
creating the object in the protected state, comprising:
dividing the object data into a group of object data fragments;
erasure encoding the object data into a group of object coding fragments; and
writing the group of object data fragments and the group of object coding fragments into data structure fragments pre-allocated among distributed nodes of a node cluster, wherein the writing the group of object data fragments and the group of object coding fragments to the data structure fragments comprises writing respective object data fragments and object coding fragments to same offset locations in a distributed chunk data structure.
11. The method of claim 10 , wherein the group of object data fragments comprises twelve object data fragments, wherein the group of object coding fragments comprises four object coding fragments, and wherein the writing the group of object data fragments and the group of object coding fragments comprises writing the group of object data fragments and the group of object coding fragments into sixteen data structure fragments of the data structure fragments in a pre-allocated chunk data structure distributed among the distributed nodes of the node cluster.
12. The method of claim 10 , wherein the determining, based on the object data being determined to be of the size greater than or equal to the first size limit and below the second size limit, that the object data is to be stored in the protected state without the preliminary protection scheme comprises:
determining that the size of the object data is greater than or equal to twenty-four megabytes and is less than one-hundred twenty-eight megabytes.
13. The method of claim 10 , further comprising:
acknowledging creation of the object after writing the group of object data fragments and the group of object coding fragments into the data structure fragments.
14. The method of claim 10 , further comprising:
sealing the distributed chunk data structure in response to determining that the distributed chunk data structure has entered a threshold fullness state.
15. A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, the operations comprising:
receiving object data corresponding to an object to be created and stored in a protected state in a node cluster according to an erasure coding scheme;
determining, based on a size of the object data, that the object is a mid-size object that does not receive application of a preliminary protection scheme;
processing the object data into object data fragments and object coding fragments according to the erasure coding scheme;
writing respective object data fragments of the object data fragments to respective distributed chunk data fragment portions of a distributed chunk data structure that is distributed among nodes of the node cluster; and
writing respective object coding fragments of the object coding fragments to respective distributed chunk coding fragment portions of the distributed chunk data structure, wherein the respective distributed chunk data fragment portions and the respective distributed chunk coding fragment portions have a same offset location in the distributed chunk data structure.
16. The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise:
maintaining a chunk data structure identifier in association with an offset value representing the same offset location.
17. The non-transitory machine-readable medium of claim 15 , wherein the processing the object data into the object data fragments and the object coding fragments according to the erasure coding scheme comprises:
dividing the object data into twelve object data fragments and encoding the object data into four object coding fragments.
18. The system of claim 1 , wherein the operations further comprise:
maintaining a chunk data structure identifier in association with an offset value representing the same offset location.
19. The method of claim 10 , further comprising:
maintaining a chunk data structure identifier in association with an offset value representing the same offset location.
20. The non-transitory machine-readable medium of claim 15 , wherein the preliminary protection scheme comprises mirroring, and wherein erasure coding is performed on the object.