Method and system for improved write performance in erasure-coded storage systems
Methods, computer program products, computer systems, and the like are disclosed that provide for improved write performance in erasure-coded storage systems in an efficient and effective manner. These can include identifying a data segment, identifying metadata, persisting the data segment to a storage system, storing the metadata in the journal, and persisting the metadata to the storage system. In such embodiments, the metadata is associated with the data segment by virtue of the metadata comprising a fingerprint of the data segment. Further, in such embodiments, the persisting the data segment to the storage system is performed without storing the data segment in a journal, and the storage system is an erasure-coded storage system.
1. A computer-implemented method, implemented in a computer system, comprising:
identifying a data segment;
identifying metadata, wherein
the metadata is associated with the data segment at least by virtue of the metadata comprising a fingerprint of the data segment;
persisting the data segment to a storage system, wherein
the storage system comprises erasure-coded storage,
the persisting the data segment writes the data segment beginning at an erasure-coded stripe boundary in the erasure-coded storage, and
the persisting the data segment to the storage system is performed without storing the data segment in a journal;
storing the metadata in the journal; and
persisting the metadata to the storage system.
2. The method of claim 1 , further comprising:
determining whether the data segment is a duplicate data segment;
in response to the data segment being the duplicate data segment, updating existing metadata stored in the journal, wherein
the existing metadata is associated with an existing data segment; and
in response to the data segment not being the duplicate data segment, performing the persisting of the data segment, the storing of the metadata, and the persisting of the metadata.
3. The method of claim 1 , wherein
the persisting the data segment to the storage system stores the data segment in a container data file, and
the persisting the metadata to the storage system stores the metadata in a container metadata file.
4. The method of claim 3 , further comprising:
opening the container data file; and
appending another data segment to the container data file, wherein
the appending comprises writing the another data segment to the container data file, and
the writing the another data segment starts at another erasure-coded stripe boundary of the erasure-coded storage within the container data file.
5. The method of claim 3 , wherein
the erasure-coded storage is comprised in an erasure-coded storage system.
6. The method of claim 5 , wherein
the container data file is stored in the erasure-coded storage system such that the container data file begins at the erasure-coded stripe boundary.
7. The method of claim 5 , wherein
the persisting of the data segment stores the data segment at an erasure-coded stripe boundary of the erasure-coded storage within the container data file.
8. The method of claim 3 , further comprising:
updating a container metadata file header of the container metadata file, wherein
a storage size of the container metadata file header is equal to a sector size of the storage system.
9. The method of claim 8 , further comprising:
persisting the container metadata file header into the container metadata file.
10. A computer program product comprising:
a plurality of instructions, comprising
a first set of instructions, executable by a processor of a computer system, configured to identify a data segment,
a second set of instructions, executable by the processor, configured to identify metadata, wherein
the metadata is associated with the data segment at least by virtue of the metadata comprising a fingerprint of the data segment,
a third set of instructions, executable by the processor, configured to persist the data segment to a storage system, wherein
the storage system comprises erasure-coded storage,
the third set of instructions persists the data segment by causing writing of the data segment to begin at an erasure-coded stripe boundary in the erasure-coded storage, and
the data segment is persisted to the storage system without storing the data segment in a journal,
a fourth set of instructions, executable by a processor of a computer system, configured to store the metadata in the journal, and
a fifth set of instructions, executable by a processor of a computer system, configured to persist the metadata to the storage system; and
a non-transitory computer-readable storage medium, wherein the instructions are encoded in the non-transitory computer-readable storage medium.
11. The computer program product of claim 10 , wherein the instructions further comprise:
a sixth set of instructions, executable by the processor, configured to determine whether the data segment is a duplicate data segment;
a seventh set of instructions, executable by the processor, configured to, in response to the data segment being the duplicate data segment, update existing metadata stored in the journal, wherein
the existing metadata is associated with an existing data segment; and
an eighth set of instructions, executable by the processor, configured to, in response to the data segment not being the duplicate data segment, cause execution of the third set of instructions, the fourth set of instructions, and the fifth set of instructions.
12. The computer program product of claim 10 , further comprising:
a sixth set of instructions, executable by the processor, configured to open a container data file, wherein
the data segment is persisted to the storage system by storing the data segment in the container data file, and
the metadata is persisted to the storage system by storing the metadata in a container metadata file; and
a seventh set of instructions, executable by the processor, configured to append another data segment to the container data file, wherein
the another data segment is appended to the container data file by writing the another data segment to the container data file, and
the writing the another data segment starts at another erasure-coded stripe boundary of the erasure-coded storage within the container data file.
13. The computer program product of claim 10 , wherein
the data segment is persisted to the storage system by storing the data segment in a container data file,
the metadata is persisted to the storage system by storing the metadata in a container metadata file, and
the erasure-coded storage is comprised in an erasure-coded storage system.
14. The computer program product of claim 13 , wherein
the container data file is stored in the erasure-coded storage system such that the container data file begins at the erasure-coded stripe boundary.
15. The computer program product of claim 10 , further comprising:
a sixth set of instructions, executable by the processor, configured to update a container metadata file header of a container metadata file, wherein
the data segment is persisted to the storage system by storing the data segment in a container data file,
the metadata is persisted to the storage system by storing the metadata in the container metadata file, and
a storage size of the container metadata file header is equal to a sector size of the storage system; and
a seventh set of instructions, executable by the processor, configured to persist the container metadata file header into the container metadata file.
16. A computer system comprising:
one or more processors;
a computer-readable storage medium coupled to the one or more processors; and
a plurality of instructions, encoded in the computer-readable storage medium and configured to cause the one or more processors to
identify a data segment,
identify metadata, wherein
the metadata is associated with the data segment at least by virtue of the metadata comprising a fingerprint of the data segment,
persist the data segment to a storage system, wherein
the storage system comprises erasure-coded storage,
the data segment is persisted by writing the data segment beginning at an erasure-coded stripe boundary in the erasure-coded storage, and
the data segment is persisted to the storage system without storing the data segment in a journal,
store the metadata in the journal, and
persist the metadata to the storage system.
17. The computer system of claim 16 , wherein the plurality of instructions comprise further instructions configured to cause the one or more processors to:
determine whether the data segment is a duplicate data segment;
in response to the data segment being the duplicate data segment, update existing metadata stored in the journal, wherein
the existing metadata is associated with an existing data segment; and
the data segment not being the duplicate data segment, cause execution of the instructions configured to cause the one or more processors to persist the data segment, to store the metadata, and to persist the metadata.
18. The computer system of claim 16 , wherein the plurality of instructions comprise further instructions configured to cause the one or more processors to:
open a container data file, wherein
the data segment is persisted to the storage system by storing the data segment in the container data file, and
the metadata is persisted to the storage system by storing the metadata in a container metadata file; and
append another data segment to the container data file, wherein
the another data segment is appended to the container data file by writing the another data segment to the container data file, and
the writing the another data segment starts at another erasure-coded stripe boundary of the erasure-coded storage within the container data file.
19. The computer system of claim 16 , wherein
the data segment is persisted to the storage system by storing the data segment in a container data file,
the metadata is persisted to the storage system by storing the metadata in a container metadata file,
the erasure-coded storage is comprised in an erasure-coded storage system, and
the container data file is stored in the erasure-coded storage system such that the container data file begins at the erasure-coded stripe boundary.
20. The computer system of claim 16 , wherein the plurality of instructions comprise further instructions configured to cause the one or more processors to:
update a container metadata file header of a container metadata file, wherein
the data segment is persisted to the storage system by storing the data segment in a container data file,
the metadata is persisted to the storage system by storing the metadata in the container metadata file, and
a storage size of the container metadata file header is equal to a sector size of the storage system; and
persist the container metadata file header into the container metadata file.