Enhanced data compression in distributed datastores
A method for performing write operations on a set of one or more physical disks of a set of one or more host machines is provided. The method receives a data block to write on at least one physical disk in the set of physical disks and generates a first set of one or more compressed sectors based on the received data block. The method writes (i) a first entry having a first header and the first set of compressed sectors to a data log that is maintained in a cache, and (ii) the first set of compressed sectors to a bank in memory. The method further determines if a size of data including compressed sectors in the bank satisfies a threshold, and when the size of data in the bank satisfies the threshold, writes the data to the at least one physical disk in the set of physical disks.
1. A method for performing write operations on a set of one or more physical disks of a set of one or more host machines, the method comprising:
receiving a data block to write on at least two physical disks in the set of physical disks of the set of host machines;
generating a first set of one or more compressed sectors based on the received data block;
maintaining a data log in a cache comprising one or more physical disks, each entry of the data log comprising a header and a set of compressed sectors, the header including metadata describing the set of compressed sectors;
writing a first entry to the data log, the first entry comprising a first header and the first set of one or more compressed sectors;
writing the first entry to a bank in memory of at least one of the set of host machines;
determining if a size of data in the bank satisfies a threshold after writing the first entry in the bank; and
when the size of data in the bank satisfies the threshold, writing the data including the first set of one or more compressed sectors to the at least two physical disks in the set of physical disks of the set of host machines.
2. The method of claim 1 , wherein writing the first entry to the data log comprises mirroring the first entry to multiple physical disks without decompressing the first set of one or more compressed sectors.
3. The method of claim 1 , wherein writing the data to the at least two physical disks in the set of physical disks of the set of host machines comprises transmitting the data including the compressed sectors to each of the set of host machines.
4. The method of claim 1 , wherein the threshold comprises a size corresponding to a full stripe comprising a plurality of data blocks.
5. The method of claim 4 , wherein the plurality of data blocks is spread across the set of physical disks in the one or more host machines.
6. The method of claim 5 , further comprising:
generating a stripe summary comprising a set of summary blocks, wherein each summary block is written on one of the set of physical disks and comprises metadata describing compressed blocks written on the one physical disk for the stripe, wherein the metadata describing the compressed blocks comprises a compression size of each block.
7. The method of claim 6 , wherein each host machine comprising one of the set of physical disks performs at least one of garbage collection, checksum calculation, or hash calculation without decompressing data blocks in the physical disk using the summary block of the one physical disk.
8. A non-transitory computer readable medium comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations for performing write operations on a set of one or more physical disks of a set of one or more host machines, the operations comprising:
receiving a data block to write on at least two physical disks in the set of physical disks of the set of host machines;
generating a first set of one or more compressed sectors based on the received data block;
maintaining a data log in a cache comprising one or more physical disks, each entry of the data log comprising a header and a set of compressed sectors, the header including metadata describing the set of compressed sectors;
writing a first entry to the data log, the first entry comprising a first header and the first set of one or more compressed sectors;
writing the first entry to a bank in memory of at least one of the set of host machines;
determining if a size of data in the bank satisfies a threshold after writing the first entry in the bank; and
when the size of data in the bank satisfies the threshold, writing the data including the first set of one or more compressed sectors to the at least two physical disks in the set of physical disks of the set of host machines.
9. The non-transitory computer readable medium of claim 8 , wherein writing the first entry to the data log comprises mirroring the first entry to multiple physical disks without decompressing the first set of one or more compressed sectors.
10. The non-transitory computer readable medium of claim 8 , wherein the threshold comprises a size corresponding to a full stripe comprising a plurality of data blocks.
11. The non-transitory computer readable medium of claim 10 , wherein the plurality of data blocks is spread across the set of physical disks in the one or more host machines.
12. The non-transitory computer readable medium of claim 11 , wherein the operations further comprise:
generating a stripe summary comprising a set of summary blocks, wherein each summary block is written on one of the set of physical disks and comprises metadata describing compressed blocks written on the one physical disk for the stripe, wherein the metadata describing the compressed blocks comprises a compression size of each block.
13. The non-transitory computer readable medium of claim 12 , wherein each host machine comprising one of the set of physical disks performs at least one of garbage collection, checksum calculation, or hash calculation without decompressing data blocks in the physical disk using the summary block of the one physical disk.
14. A computer system, comprising:
a memory; and
a processor coupled to the memory, the processor being configured to:
receive a data block to write on at least two physical disks in a set of physical disks of a set of host machines;
generate a first set of one or more compressed sectors based on the received data block;
maintain a data log in a cache comprising one or more physical disks, each entry of the data log comprising a header and a set of compressed sectors, the header including metadata describing the set of compressed sectors;
write a first entry to the data log, the first entry comprising a first header and the first set of one or more compressed sectors;
write the first entry to a bank in memory of at least one of the set of host machines;
determine if a size of data in the bank satisfies a threshold after writing the first entry in the bank; and
when the size of data in the bank satisfies the threshold, write the data including the first set of one or more compressed sectors to the at least two physical disks in the set of physical disks of the set of host machines.
15. The computer system of claim 14 , wherein writing the first entry to the data log comprises mirroring the first entry to multiple physical disks without decompressing the first set of one or more compressed sectors.
16. The computer system of claim 14 , wherein the threshold comprises a size corresponding to a full stripe comprising a plurality of data blocks spread across the set of physical disks in the one or more host machines.
17. The computer system of claim 16 , wherein the processor is further configured to:
generate a stripe summary comprising a set of summary blocks, wherein each summary block is written on one of the set of physical disks and comprises metadata describing compressed blocks written on the one physical disk for the stripe, wherein the metadata describing the compressed blocks comprises a compression size of each block.