Methods and systems of efficiently storing data
The present disclosure provides methods, systems, and non-transitory computer readable media for efficiently storing data. The methods include segmenting a parcel of data into one or more data chunks according to a physical block size of the secondary storage unit, wherein the one or more data chunks include a partial data chunk and zero or more full data chunks; sending each full data chunk of the zero or more full data chunks to the secondary storage unit to be written to a selected physical block of the secondary storage unit; collecting, in a collection buffer, the current partial data chunk and at least another partial data chunk; and sending a combination of the current partial data chunk and a subset of the plurality of other partial data chunks to the secondary storage unit to be written to a selected physical block of the secondary storage unit, wherein said combination fills substantially all of a physical-block-sized data chunk.
1. A method for storing data in a secondary storage unit, the method comprising:
segmenting a parcel of data into one or more data chunks according to a physical block size of the secondary storage unit, wherein the one or more data chunks include a partial data chunk and zero or more full data chunks;
sending each full data chunk of the zero or more full data chunks to the secondary storage unit to be written to a selected physical block of the secondary storage unit;
collecting, in a collection buffer, the current partial data chunk and at least another partial data chunk;
sending a combination of the current partial data chunk and a subset of the plurality of other partial data chunks to the secondary storage unit to be written to a selected physical block of the secondary storage unit, wherein said combination fills substantially all of a physical-block-sized data chunk; and
recording, for the parcel of data, metadata that directly maps the parcel of data and the physical blocks where the parcel of data's data chunks were written without mapping the parcel of data to logical blocks.
2. The method of claim 1 , further comprising:
segmenting the parcel of data into two or more sub-parcels; and
selecting two or more secondary storage units to store the sub-parcels of data on.
3. The method of claim 1 , wherein the secondary storage units are selected on the basis of:
current or historical IO utilization of the secondary storage unit,
current or historical capacity utilization of the secondary storage unit,
performance characteristics of the secondary storage unit, or
characteristics of the parcel of data.
4. The method of claim 1 , wherein the metadata includes physical block addresses for the physical block where the parcel of data's data chunks were written and indexes of the parcel of data's data chunks in each of the physical blocks.
5. The method of claim 4 , wherein:
each written data chunk starts with a header and ends with a footer;
the header and footer are unique within their respective data chunk; and
the header marks the index of the data chunk.
6. The method of claim 1 , further comprising sending the partial data chunks collected in the collection buffer to the secondary storage unit to be written to selected physical blocks of the secondary storage unit in response to an exceptional event.
7. The method of claim 6 , wherein the exceptional event is one of a loss of power, a time out of a timer, or a command to flush the collection buffer.
8. A non-transitory computer readable medium that stores a set of instructions that is executable by at least one processor of a computer system to cause the computer system to perform a method for storing data in a secondary storage unit, the method comprising:
segmenting a parcel of data into one or more data chunks according to a physical block size of the secondary storage unit, wherein the one or more data chunks include a partial data chunk and zero or more full data chunks;
sending each full data chunk of the zero or more full data chunks to the secondary storage unit to be written to a selected physical block of the secondary storage unit;
collecting, in a collection buffer, the current partial data chunk and at least another partial data chunk;
sending a combination of the current partial data chunk and a subset of the plurality of other partial data chunks to the secondary storage unit to be written to a selected physical block of the secondary storage unit, wherein said combination fills substantially all of a physical-block-sized data chunk; and
recording, for the parcel of data, metadata that directly maps the parcel of data and the physical blocks where the parcel of data's data chunks were written without mapping the parcel of data to logical blocks.
9. The non-transitory computer readable medium of claim 8 , wherein the secondary storage units are selected on the basis of:
current or historical IO utilization of the secondary storage unit,
current or historical capacity utilization of the secondary storage unit,
performance characteristics of the secondary storage unit, or
characteristics of the parcel of data.
10. The non-transitory computer readable medium of claim 8 , wherein the metadata includes physical block addresses for the physical block where the parcel of data's data chunks were written and indexes of the parcel of data's data chunks in each of the physical blocks.
11. The non-transitory computer readable medium of claim 10 , wherein:
each written data chunk starts with a header and ends with a footer;
the header and footer are unique within their respective data chunk; and
the header marks the index of the data chunk.
12. The non-transitory computer readable medium of claim 8 , wherein the set of instructions is executable by the at least one processor of the computer system to cause the computer system to further perform sending the partial data chunks collected in the collection buffer to the secondary storage unit to be written to selected physical blocks of the secondary storage unit in response to an exceptional event.
13. A system for storing data in a secondary storage unit, comprising:
a memory storing a set of instructions; and
one or more processors configured to execute the set of instructions to cause the system to perform:
segmenting a parcel of data into one or more data chunks according to a physical block size of the secondary storage unit, wherein the one or more data chunks include a partial data chunk and zero or more full data chunks;
sending each full data chunk of the zero or more full data chunks to the secondary storage unit to be written to a selected physical block of the secondary storage unit;
collecting, in a collection buffer, the current partial data chunk and at least another partial data chunk;
sending a combination of the current partial data chunk and a subset of the plurality of other partial data chunks to the secondary storage unit to be written to a selected physical block of the secondary storage unit, wherein said combination fills substantially all of a physical-block-sized data chunk; and
recording, for the parcel of data, metadata that directly maps the parcel of data and the physical blocks where the parcel of data's data chunks were written without mapping the parcel of data to logical blocks.
14. The system of claim 13 , wherein the secondary storage units are selected on the basis of:
current or historical IO utilization of the secondary storage unit,
current or historical capacity utilization of the secondary storage unit,
performance characteristics of the secondary storage unit, or
characteristics of the parcel of data.
15. The system of claim 13 , wherein the metadata includes physical block addresses for the physical block where the parcel of data's data chunks were written and indexes of the parcel of data's data chunks in each of the physical blocks.
16. The system of claim 15 , wherein:
each written data chunk starts with a header and ends with a footer;
the header and footer are unique within their respective data chunk; and
the header marks the index of the data chunk.
17. The system of claim 13 , wherein the one or more processors are configured to execute the set of instructions to cause the system to further perform sending the partial data chunks collected in the collection buffer to the secondary storage unit to be written to selected physical blocks of the secondary storage unit in response to an exceptional event.
18. A system comprising:
a host device; and
a secondary storage unit communicatively coupled to the host device, wherein
the host device is configured to:
segment a parcel of data into one or more data chunks according to a physical block size of the secondary storage unit, wherein the one or more data chunks include a partial data chunk and zero or more full data chunks,
send each full data chunk of the zero or more full data chunks to the secondary storage unit to be written to a selected physical block of the secondary storage unit,
collect, in a collection buffer, the current partial data chunk and at least another partial data chunk, send a combination of the current partial data chunk and a subset of the plurality of other partial data chunks to the secondary storage unit to be written to a selected physical block of the secondary storage unit, wherein said combination fills substantially all of a physical-block-sized data chunk, and
record, for the parcel of data, metadata that directly maps the parcel of data and the physical blocks where the parcel of data's data chunks were written without mapping the parcel of data to logical blocks; and
the secondary storage unit is configured to:
write each received full data chunk of the zero or more full data chunks to the selected physical block of the secondary storage unit,
write the combination of the current partial data chunk and the subset of the plurality of other partial data chunks to the selected physical block of the secondary storage unit.