Systems and methods for formatting and reconstructing raw data
A method for formatting raw data comprises accessing the raw data, the raw data comprising sparse data segments, which are empty of any data, and non-sparse data segments, which comprise data, and generating a formatted data stream comprising one or more atomic blocks, each atomic block corresponding to a metadata file and to a portion of the non-sparse data segments of the raw data. Generating one atomic block comprises browsing the raw data and, upon locating one sparse data segment, populating the corresponding metadata file with offsets indicative of a beginning and an end of the located sparse data segment and populating the atomic block with a concatenation of at least portions of the non-sparse segments of the raw data located before and after the located sparse segment. If the atomic block exceeds a maximum size, another atomic block and another corresponding metadata file are populated.
1. A method of formatting raw data, comprising:
accessing the raw data from a storage location on a first data storage device, the raw data containing sparse data segments and non-sparse data segments, the sparse data segments being empty of any data and the non-sparse data segments containing data;
reading the accessed raw data to locate the sparse data segments and the non-sparse data segments;
generating a formatted data stream comprising one or more formatted atomic blocks based on the read raw data, the one or more formatted atomic blocks having a maximum block size defined by a size of a destination device memory and associated with a metadata file, each one of the one or more formatted atomic blocks configured with a format that includes a field indicating a data size of the non-sparse data segments of the raw data, a field containing at least a portion of the non-sparse data segments, and a field containing the metadata file identifying specific beginning and end locations, in which the formatting of the one or more atomic blocks comprises:
when one of the atomic blocks does not exceed the maximum block size:
populate the metadata file of the one of the atomic blocks with offsets identifying a beginning of each of the located sparse data segments and an end of each of the located sparse data segments; and
populate the one of the atomic blocks with a concatenation of at least a portion of a first non-sparse segment of the raw data located before the sparse segment with at least a portion of a second non-sparse data segment of the raw data located after the sparse data segment; and
when one of the atomic blocks exceeds the maximum size:
populate the metadata file of another one of the atomic blocks with offsets indicative of the beginning of the located sparse data segment and the end of the located sparse data segment; and
populate the other one of the atomic blocks with a concatenation of the at least a portion of the first non-sparse data segment of the raw data located before the sparse data segment with the at least a portion of the second non-sparse data segment of the raw data located after the sparse data segment; and
reconstructing the raw data on a second data storage device from the formatted data stream, the second data storage device being distinct from the first data storage device, the reconstructing being based on the metadata file, the reconstructing comprising:
browsing the metadata file; and
iteratively reconstructing the raw data by writing a corresponding data content for each non-sparse data segment identified by the metadata file and executing a command to define a segment of zeros for each sparse data segment identified by the metadata file, wherein the segment of zeros comprises multiple zeros,
wherein one atomic block is streamed from the first data storage device to the second data storage device while the generating of the formatted data stream and the iterative reconstructing of the raw data are being executed in parallel.
2. The method of claim 1 , wherein a size of the segment of zeroes corresponds to a size of the corresponding sparse data segment.
3. The method of claim 1 , wherein generating the formatted data stream further comprises generating a footer, the footer comprising a value relating to a number of the one or more formatted atomic blocks and being concatenated to the formatted data stream.
4. A method of formatting raw data, comprising:
accessing the raw data on a first data storage device, the raw data containing sparse data segments and non-sparse data segments, the sparse data segments being empty of any data and the non-sparse data segments containing data;
reading the raw data to locate the sparse data segments and the non-sparse data segments;
generating a formatted data stream based on the read raw data in which a size of the formatted data stream is limited by a size of a destination device memory, the formatted data stream comprising a field indicating a data size of the non-sparse data segments of the raw data, a field containing at least a portion of the non-sparse data segments, and a field containing a metadata file, the metadata file identifying specific locations of located sparse segments, wherein the generating comprises, upon locating one of the sparse data segments in the raw data, assembling the metadata file by:
populating the metadata file with offsets identifying a beginning of each of the located sparse data segments and an end of each of the located sparse data segments; and
populating the metadata file by concatenating each of the located non-sparse segments in successive fashion of the raw data located between each sparse data segments; and
reconstructing the raw data on a second data storage device from the formatted data stream, the second data storage device being distinct from the first data storage device, the reconstructing being based on the metadata file, the reconstructing comprising:
browsing the metadata file; and
iteratively reconstructing the raw data by writing a corresponding data content for each non-sparse data segment identified by the metadata file and executing a command to define a segment of zeros for each sparse data segment identified by the metadata file, wherein the segment of zeros comprises multiple zeros,
wherein the formatted data stream is streamed from the first data storage device to the second data storage device while the generating of the formatted data stream and the reconstructing of the raw data are being executed in parallel.
5. The method of claim 4 , wherein a size of the segment of zeroes corresponds to a size of the corresponding sparse data segment.
6. The method of claim 4 , wherein:
the raw data comprises data blocks and is reconstructed on a block device; and
a command to define a block of zero which size corresponds to the size of the sparse segment is associated with the block device.
7. The method of claim 4 , further comprising, prior to reconstructing the raw data from the formatted data stream, executing a decompression algorithm on the formatted data stream.
8. The method of claim 4 , wherein the formatted data stream is streamed from a first data storage device to a second data storage device while the formatting of the raw data and the reconstructing of the raw data are being executed in parallel.
9. The method of claim 1 , wherein generating the formatted data stream further comprises generating a footer, the footer comprising one or more of the following values: a number of read bytes of the raw data, a number of written bytes including written bytes of the metadata file and the footer, and/or a ratio between the number of read bytes and the number of written bytes; the footer being concatenated to the formatted data stream.
10. The method of claim 1 , wherein the raw data comprises a disk-image containing a content and/or a structure of an entire data storage device.
11. The method of claim 1 , further comprising, subsequent to generating the formatted data stream, executing a compression algorithm on the formatted data stream.
12. A networking device for Operating System (OS) deployment on a bare-metal server, the networking device being configured to perform the method of claim 1 .