Method and system for reduced data movement compression using in-storage computing and a customized file system
One embodiment facilitates data compression. During operation, the system receives, by a host computing device, a request to write a first set of data. The system writes the first set of data to a first storage device of a storage component. A storage component controller performs compression on the first set of data to obtain compressed data. The controller transmits, to a file system component of the host computing device, metadata associated with the compressed data. The file system component inserts in a journal an entry based on the metadata associated with the compressed data. The system writes, by the controller, the compressed data to a second storage device of the storage component, thereby enhancing performance of the host computing device by reducing a number of data transfers involved in compressing and writing of the first set of data.
1. A computer-implemented method for facilitating data compression, the method comprising:
receiving, by a host computing device, a request to write a first set of data;
writing the first set of data to a first storage device of a storage component;
performing, by a controller of the storage component, compression on the first set of data to obtain compressed data;
comparing metadata of the compressed data to metadata of the first set of data to determine an amount of compression gain; and
responsive to determining that the amount of compression gain is greater than a predetermined threshold:
transmitting, by the controller to a file system component of the host computing device, metadata associated with the compressed data;
inserting, in a journal by the file system component, an entry based on the metadata associated with the compressed data; and
writing, by the controller, the compressed data to a second storage device of the storage component.
2. The method of claim 1 , further comprising:
writing, by the host computing device, a plurality of uncompressed files to a volatile memory of the host computing device;
combining, by the file system component, the plurality of uncompressed files;
concatenating, by the file system component, metadata for the combined files;
writing, by the file system component, the concatenated metadata in the journal based on a format of the concatenated metadata; and
writing the uncompressed files to the first storage device of the storage component.
3. The method of claim 1 , further comprising:
in response to determining that the amount of compression gain is not greater than the predetermined threshold, writing the first set of data to the second storage device of the storage component.
4. The method of claim 1 , further comprising:
exposing the first storage device of the storage component to the host computing device, which allows the host computing device to write the first set of data or the compressed data to the first storage device of the storage component.
5. The method of claim 1 , wherein in response to detecting a condition which triggers compaction of the journal, the method further comprises:
updating the journal by replacing out-of-date versions of metadata entries associated with a file with a most recent version of a metadata entry for the file.
6. The method of claim 5 , wherein the condition which triggers compaction of the journal is based on one or more of:
whether a current size of the journal is greater than a predetermined size; and
whether a predetermined time interval has passed.
7. The method of claim 1 , further comprising:
receiving, by the host computing device, a request to read a second set of data;
identifying, by the file system component in the journal, metadata associated with the second set of data;
in response to determining, based on the identified metadata, that the second set of data is stored as compressed data:
loading the compressed data corresponding to the second set of data from the second storage device of the storage component to the first storage device of the storage component; and
extracting, by the controller, the second set of data by performing decompression on the compressed data corresponding to the second set of data;
in response to determining, based on the identified metadata, that the second set of data is not stored as compressed data, reading the second set of data from the second storage device of the storage component; and
transmitting the requested second set of data to the host computing device.
8. The method of claim 1 ,
wherein the storage component is a solid state drive (SSD),
wherein the first storage device of the storage component is an internal volatile memory which includes a dynamic random access memory (DRAM) buffer,
wherein the second storage device is a non-volatile memory,
wherein the compression is performed by a microprocessor of the SSD controller, and
wherein the first set of data is written to the internal DRAM buffer of the SSD via a Peripheral Component Interconnect Express (PCIe) bus.
9. A computer system for facilitating data compression, the system comprising:
a processor; and
a memory coupled to the processor and storing instructions, which when executed by the processor cause the processor to perform a method, the method comprising:
receiving, by a host computing device, a request to write a first set of data;
writing the first set of data to a first storage device of a storage component;
performing, by a controller of the storage component, compression on the first set of data to obtain compressed data;
comparing metadata of the compressed data to metadata of the first set of data to determine an amount of compression gain; and
responsive to determining that the amount of compression gain is greater than a predetermined threshold:
transmitting, by the controller to a file system component of the host computing device, metadata associated with the compressed data;
inserting, in a journal by the file system component, an entry based on the metadata associated with the compressed data; and
writing, by the controller, the compressed data to a second storage device of the storage component.
10. The computer system of claim 9 , further comprising:
writing, by the host computing device, a plurality of uncompressed files to a volatile memory of the host computing device;
combining, by the file system component, the plurality of uncompressed files;
concatenating, by the file system component, metadata for the combined files;
writing, by the file system component, the concatenated metadata in the journal based on a format of the concatenated metadata; and
writing the uncompressed files to the first storage device of the storage component.
11. The computer system of claim 9 , wherein the method further comprises:
in response to determining that the amount of compression gain is not greater than the predetermined threshold, writing the first set of data to the second storage device of the storage component.
12. The computer system of claim 9 , wherein the method further comprises:
exposing the first storage device of the storage component to the host computing device, which allows the host computing device to write the first set of data or the compressed data to the first storage device of the storage component.
13. The computer system of claim 9 , wherein in response to detecting a condition which triggers compaction of the journal, the method further comprises:
updating the journal by replacing out-of-date versions of metadata entries associated with a file with a most recent version of a metadata entry for the file.
14. The computer system of claim 13 , wherein the condition which triggers compaction of the journal is based on one or more of:
whether a current size of the journal is greater than a predetermined size; and
whether a predetermined time interval has passed.
15. The computer system of claim 9 , wherein the method further comprises:
receiving, by the host computing device, a request to read a second set of data;
identifying, by the file system component in the journal, metadata associated with the second set of data;
in response to determining, based on the identified metadata, that the second set of data is stored as compressed data:
loading the compressed data corresponding to the second set of data from the second storage device of the storage component to the first storage device of the storage component; and
extracting, by the controller, the second set of data by performing decompression on the compressed data corresponding to the second set of data;
in response to determining, based on the identified metadata, that the second set of data is not stored as compressed data, reading the second set of data from the second storage device of the storage component; and
transmitting the requested second set of data to the host computing device.
16. The computer system of claim 9 , wherein the storage component is a solid state drive (SSD), wherein the first storage device of the storage component is an internal volatile memory which includes a dynamic random access memory (DRAM) buffer, wherein the second storage device is a non-volatile memory, wherein the compression is performed by a microprocessor of the SSD controller, and wherein the first set of data is written to the internal DRAM buffer of the SSD via a Peripheral Component Interconnect Express (PCIe) bus.
17. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, the method comprising:
receiving, by a host computing device, a request to write a first set of data;
writing the first set of data to a first storage device of a storage component;
performing, by a controller of the storage component, compression on the first set of data to obtain compressed data;
comparing metadata of the compressed data to metadata of the first set of data to determine an amount of compression gain; and
responsive to determining that the amount of compression gain is greater than a predetermined threshold:
transmitting, by the controller to a file system component of the host computing device, metadata associated with the compressed data;
inserting, in a journal by the file system component, an entry based on the metadata associated with the compressed data; and
writing, by the controller, the compressed data to a second storage device of the storage component.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the method further comprises:
in response to determining that the amount of compression gain is not greater than the predetermined threshold, writing the first set of data to the second storage device of the storage component.
19. The non-transitory computer-readable storage medium of claim 17 , further comprising:
exposing the first storage device of the storage component to the host computing device, which allows the host computing device to write the first set of data or the compressed data to the first storage device of the storage component.
20. The non-transitory computer-readable storage medium of claim 17 , wherein in response to detecting a condition which triggers compaction of the journal, the method further comprises:
updating the journal by replacing out-of-date versions of metadata entries associated with a file with a most recent version of a metadata entry for the file.