Efficient compaction on log-structured distributed file system using erasure coding for resource consumption reduction

1. A computer-implemented method for performing data compaction in a storage system comprising a plurality of storage nodes, the method comprising:

in response to determining, by a processor, that data compaction is triggered, regrouping valid data from a first set of data chunks stored in the storage system into a second set of data chunks such that a respective data chunk from the second set of data chunks comprises contiguous valid data slices, wherein a respective valid data slice in the first set of data chunks is stored at a particular physical location and is included in the second set of data chunks; and

performing error-correction-coding protection on the second set of data chunks, which comprises:

locating an existing erasure-coding codeword comprising the valid data slice in the first set of data chunks;

constructing a new erasure-coding codeword in response to the existing erasure-coding codeword comprising invalid data; and

storing the constructed new erasure-coding codeword in the storage system, wherein the valid data slice included in the constructed new erasure-coding codeword is stored at the same particular physical location.

2. The computer-implemented method of claim 1 , wherein the existing erasure-coding codeword comprises multiple portions that are separately stored in a predetermined set of storage nodes; and

wherein storing the constructed new error-correction-coding codeword comprises writing the constructed new error-correction-coding codeword into the same predetermined set of storage nodes.

3. The computer-implemented method of claim 1 , wherein constructing the new error-correction-coding codeword further comprises:

in response to receiving new data to be stored in the storage system, replacing a portion of the existing error-correction-coding codeword that comprises invalid data with a corresponding portion of the received new data; and

computing new parity based on the corresponding portion of the received new data and remaining portions of the existing error-correction-coding codeword that comprise valid data.

4. The computer-implemented method of claim 3 , wherein writing the constructed new error-correction-coding codeword further comprises:

writing the corresponding portion of the received new data in a same storage node storing the replaced portion of the existing error-correction-coding codeword that comprises invalid data;

writing the computed new parity in same storage nodes storing parity of the existing parity-coding codeword; and

keeping the remaining portions of the existing error-correction-coding codeword that comprise valid data in their current physical locations.

5. The computer-implemented method of claim 1 , wherein constructing the new error-correction-coding codeword further comprises:

in response to not receiving new data to be stored in the storage system, replacing a portion of the existing error-correction-coding codeword that comprises invalid data with a corresponding portion of valid data currently located on a same storage node; and

computing new parity based on the corresponding portion of valid data and remaining portions of the existing error-correction-coding codeword that comprise valid data.

6. The computer-implemented method of claim 5 , wherein storing the constructed new error-correction-coding codeword further comprises:

writing the computed new parity in same storage nodes storing parity of the existing parity-coding codeword; and

keeping the remaining portions of the existing error-correction-coding codeword that comprise valid data and the corresponding portion of valid data in their current physical locations.

7. The computer-implemented method of claim 1 , further comprising invalidating parity portions of the existing error-correction-coding codeword such that the invalidated parity portions are subject to garbage collection along with the portion of the existing error-correction-coding codeword that comprises invalid data.

8. A computer system, comprising:

a processor; and

a storage device coupled to the processor and storing instructions, which when executed by the processor cause the processor to perform a method for performing data compaction in a storage system comprising a plurality of storage nodes, the method comprising:

in response to determining that data compaction is triggered, regrouping valid data from a first set of data chunks stored in the storage system into a second set of data chunks such that a respective data chunk from the second set of data chunks comprises contiguous valid data slices, wherein a respective valid data slice in the first set of data chunks is stored at a particular physical location and is included in the second set of data chunks; and

performing error-correction-coding protection on the second set of data chunks, which comprises:

locating an existing erasure-coding codeword comprising the valid data slice in the first set of data chunks;

constructing a new erasure-coding codeword in response to the existing erasure-coding codeword comprising invalid data; and

storing the newly constructed erasure-coding codeword in the storage system, wherein the valid data slice included in the constructed new erasure-coding codeword is stored at the same particular physical location.

9. The computer system of claim 8 , wherein the existing error-correction-coding codeword comprises multiple portions that are separately stored in a predetermined set of storage nodes; and

wherein storing the constructed new error-correction-coding codeword comprises writing the constructed new error-correction-coding codeword into the same predetermined set of storage nodes.

10. The computer system of claim 8 , wherein constructing the new error-correction-coding codeword further comprises:

in response to receiving new data to be stored in the storage system, replacing a portion of the existing error-correction-coding codeword that comprises invalid data with a corresponding portion of the received new data; and

computing new parity based on the corresponding portion of the received new data and remaining portions of the existing error-correction-coding codeword that comprise valid data.

11. The computer system of claim 10 , wherein storing the constructed new error-correction-coding codeword further comprises:

writing the corresponding portion of the received new data in a same storage node storing the replaced portion of the existing error-correction-coding codeword that comprises invalid data;

writing the computed new parity in same storage nodes storing parity of the existing parity-coding codeword; and

keeping the remaining portions of the existing error-correction-coding codeword that comprise valid data in their current physical locations.

12. The computer system of claim 8 , wherein constructing the new error-correction-coding codeword further comprises:

in response to not receiving new data to be stored in the storage system, replacing the portion of the existing error-correction-coding codeword that comprises invalid data with a corresponding portion of valid data currently located on a same storage node; and

computing new parity based on the corresponding portion of valid data and the remaining portions of the existing error-correction-coding codeword that comprise valid data.

13. The computer system of claim 12 , wherein storing the constructed new error-correction-coding codeword further comprises:

writing the computed new parity in same storage nodes storing parity of the existing parity-coding codeword; and

keeping the remaining portions of the existing error-correction-coding codeword that comprise valid data and the corresponding portion of valid data in their current physical locations.

14. The computer system of claim 8 , wherein the method further comprises invalidating parity portions of the existing error-correction-coding codeword such that the invalidated parity portions are subject to garbage collection along with the portion of the existing error-correction-coding codeword that comprises invalid data.

15. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for performing data compaction in a storage system comprising a plurality of storage nodes, the method comprising:

in response to determining, by a processor, that data compaction is triggered, regrouping valid data from a first set of data chunks stored in the storage system into a second set of data chunks such that a respective data chunk from the second set of data chunks comprises contiguous valid data slices, wherein a respective valid data slice in the first set of data chunks is stored at a particular physical location and is included in the second set of data chunks; and

performing error-correction-coding protection on the second set of data chunks, which comprises:

locating an existing erasure-coding codeword comprising the valid data slice in the first set of data chunks;

constructing a new erasure-coding codeword in response to the existing erasure-coding codeword comprising invalid data; and

storing the newly constructed erasure-coding codeword in the storage system, wherein the valid data slice included in the newly constructed erasure-coding codeword is stored at the same particular physical location.

16. The computer-readable storage medium of claim 15 , wherein the existing error-correction-coding codeword comprises multiple portions that are separately stored in a predetermined set of storage nodes; and

wherein storing the constructed new error-correction-coding codeword comprises writing the constructed new error-correction-coding codeword into the predetermined set of storage nodes.

17. The computer-readable storage medium of claim 15 , wherein constructing the new error-correction-coding codeword further comprises:

in response to receiving new data to be stored in the storage system, replacing the portion of the existing error-correction-coding codeword that comprises invalid data with a corresponding portion of the received new data; and

computing new parity based on the corresponding portion of the received new data and remaining portions of the existing error-correction-coding codeword that comprise valid data.

18. The computer-readable storage medium of claim 17 , wherein storing the constructed new error-correction-coding codeword comprises:

writing the corresponding portion of the received new data in a same storage node storing the replaced portion of the existing error-correction-coding codeword that comprises invalid data;

writing the computed new parity in same storage nodes storing parity of the existing parity-coding codeword; and

keeping the remaining portions of the existing error-correction-coding codeword that comprise valid data in their current physical locations.

19. The computer-readable storage medium of claim 15 , wherein constructing the new error-correction-coding codeword further comprises:

in response to not receiving new data to be stored in the storage system, replacing the portion of the existing error-correction-coding codeword that comprises invalid data with a corresponding portion of valid data currently located on a same storage node; and

computing new parity based on the corresponding portion of valid data and remaining portions of the existing error-correction-coding codeword that comprise valid data.

20. The computer-readable storage medium of claim 19 , wherein storing the constructed new error-correction-coding codeword further comprises:

writing the computed new parity in same storage nodes storing parity of the existing parity-coding codeword; and

keeping the remaining portions of the existing error-correction-coding codeword that comprise valid data and the corresponding portion of valid data in their current physical locations.