Performance optimization and support compatibility of data compression with hardware accelerator
One embodiment provides a computer implemented method of data compression using a hardware accelerator. A first thread pool for compression jobs, and a first polling thread is allocated for polling the status of a hardware accelerator. A compression thread is retrieved from the first thread pool in response to a compression request from a file system. Multiple source data buffers from the file system are aggregated into a compression unit, and a scatter gather list and destination buffer are submitted to the hardware accelerator. A checksum of result data is calculated from the destination buffer. A zlib header is added to the result data, and the checksum is added as a zlib footer to the result data.
1. A computer implemented method of data compression using a hardware accelerator, the method comprising:
allocating a first thread pool for compression jobs and a first polling thread for polling a status of a hardware accelerator;
retrieving a compression thread from the first thread pool in response to a compression request from a file system;
aggregating multiple source data buffers from the file system into a compression unit;
submitting a scatter gather list and a destination buffer to the hardware accelerator;
calculating a first checksum of result data from the destination buffer;
adding a zlib header to the result data; and
adding the first checksum as a zlib footer to the result data.
2. The computer implemented method as in claim 1 , wherein the hardware accelerator is a QuickAssist Technology accelerator.
3. The computer implemented method as in claim 1 , further comprising:
linking the source data buffers in the compression unit into the scatter gather list.
4. The computer implemented method as in claim 1 , further comprising:
periodically polling completion registers of the hardware accelerator using the first polling thread.
5. The computer implemented method as in claim 1 , wherein aggregating the source data buffers into the compression unit is performed until the compression unit grows to a preset value.
6. The computer implemented method as in claim 1 , wherein the scatter gather list and the destination buffer are submitted to the hardware accelerator simultaneously.
7. The computer implemented method as in claim 1 , wherein the compression thread is blocked after submission of the scatter gather list and the destination buffer to the hardware accelerator.
8. The computer implemented method as in claim 1 , further comprising:
allocating a second thread pool for decompression jobs and a second polling thread for polling the status of the hardware accelerator;
retrieving a decompression thread from the second thread pool in response to a decompression request from the file system;
receiving the decompression request and a source data buffer by the decompression thread;
moving a start buffer point to skip the zlib header;
calculating a length of deflate data blocks;
submitting the source data buffer and the destination buffer to the hardware accelerator;
retrieving, by the decompression thread, result data from the hardware accelerator;
calculating a second checksum of the result data from the hardware accelerator; and
comparing the second checksum with the first checksum.
9. The computer implemented method as in claim 8 , further comprising:
reporting an error if the second checksum and the first checksum are not the same.
10. A data compression system comprising:
one or more processing devices;
a hardware accelerator for compressing and decompressing data; and
data compression logic to compress data via the one or more processing devices, the data compression logic configured to:
allocate a first thread pool for compression jobs and a first polling thread for polling a status of the hardware accelerator;
retrieve a compression thread from the first thread pool in response to a compression request from a file system;
aggregate multiple source data buffers from the file system into a compression unit;
submit a scatter gather list and a destination buffer to the hardware accelerator;
calculate a first checksum of result data from the destination buffer;
add a zlib header to the result data; and
add the first checksum as a zlib footer to the result data.
11. The deduplication data storage system as in claim 10 , wherein the hardware accelerator is a QuickAssist Technology accelerator.
12. The deduplication data storage system as in claim 10 , wherein the data compression logic is further configured to:
link the source data buffers in the compression unit into the scatter gather list.
13. The deduplication data storage system as in claim 10 , wherein the data compression logic is further configured to:
periodically poll completion registers of the hardware accelerator using the first polling thread.
14. The deduplication data storage system as in claim 10 , wherein the data compression logic is further configured to:
allocate a second thread pool for decompression jobs and a second polling thread for polling the status of the hardware accelerator;
retrieve a decompression thread from the second thread pool in response to a decompression request from a file system;
receive the decompression request and a source data buffer by the decompression thread;
move a start buffer point to skip the zlib header;
calculate a length of deflate data blocks;
submit the source data buffer and the destination buffer to the hardware accelerator;
retrieve, by the decompression thread, result data from the hardware accelerator;
calculate a second checksum of the result data from the hardware accelerator; and
compare the second checksum with the first checksum.
15. The deduplication data storage system as in claim 14 , wherein the data compression logic is further configured to:
report an error if the second checksum and the first checksum are not the same.
16. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform a data compression operation, the operation comprising:
allocating a first thread pool for compression jobs and a first polling thread for polling a status of a hardware accelerator;
retrieving a compression thread from the first thread pool in response to a compression request from a file system;
aggregating multiple source data buffers from the file system into a compression unit;
submitting a scatter gather list and a destination buffer to the hardware accelerator;
calculating a first checksum of result data from the destination buffer;
adding a zlib header to the result data; and
adding the first checksum as a zlib footer to the result data.
17. The non-transitory computer-readable medium as in claim 16 , wherein the hardware accelerator is a QuickAssist Technology accelerator.
18. The non-transitory computer-readable medium as in claim 16 , the operation further comprising:
linking the source data buffers in the compression unit into the scatter gather list.
19. The non-transitory computer-readable medium as in claim 16 , the operation further comprising:
allocating a second thread pool for decompression jobs and a second polling thread for polling the status of the hardware accelerator;
retrieving a decompression thread from the second thread pool in response to a decompression request from a file system;
receiving the decompression request and a source data buffer by the decompression thread;
moving a start buffer point to skip the zlib header;
calculating a length of deflate data blocks;
submitting the source data buffer and destination buffer to the hardware accelerator;
retrieving, by the decompression thread, result data from the hardware accelerator;
calculating a second checksum of the result data from the hardware accelerator; and
comparing the second checksum with the first checksum.
20. The non-transitory computer-readable medium as in claim 19 , the operation further comprising:
reporting an error if the second checksum and the first checksum are not the same.