Data compaction method and device
Provided is a data compaction method that includes: determining, by an electronic device, a plurality of Sorted String Table (SSTable) files on which a compaction operation is to be performed among SSTable files stored in a storage device connected to the electronic device, the data in the SSTable files stored in the storage device is stored based on a Log Structured Merge Tree (LSM-Tree) mechanism; based on receiving the plurality of SSTable files from the storage device, performing, by the electronic device, merge sorting on data in data blocks of the plurality of SSTable files based on indexes of the data in the data blocks of the plurality of SSTable files, to obtain reconstructed data blocks; and sending, by the electronic device, the reconstructed data blocks to the storage device, the reconstructed data blocks being used by the storage device to obtain reconstructed SSTable files.
1. A data compaction method comprising:
determining, by an electronic device, a plurality of Sorted String Table (SSTable) files on which a compaction operation is to be performed among SSTable files stored in a storage device connected to the electronic device, wherein data in the SSTable files stored in the storage device is stored based on a Log Structured Merge Tree (LSM-Tree) mechanism;
based on receiving the plurality of SSTable files from the storage device, performing, by the electronic device, merge sorting on data in data blocks of the plurality of SSTable files based on indexes of the data in the data blocks of the plurality of SSTable files, to obtain reconstructed data blocks; and
sending, by the electronic device, the reconstructed data blocks to the storage device, wherein the reconstructed data blocks are used by the storage device to obtain reconstructed SSTable files,
wherein each SSTable file comprises a data block and an index block, and indexes in each index block correspond to data in the data block corresponding to the index block.
2. The method of claim 1 , wherein the performing the merge sorting on the data in the data blocks of the plurality of SSTable files comprises:
parsing the data blocks of the plurality of SSTable files to obtain the data in the data blocks of the plurality of SSTable files;
performing the merge sorting on the data in the data blocks of the plurality of SSTable files based on the indexes of the data in the data blocks of the plurality of SSTable files; and
reconstructing data obtained through the merge sorting into data blocks according to a data block format.
3. The method of claim 2 , wherein the indexes of the data in the data blocks of the plurality of SSTable files are obtained by parsing each index block of the plurality of SSTable files by the storage device.
4. The method of claim 3 , wherein the storage device is a SmartSSD comprising a storage unit and a computing unit, the SSTable files stored in the storage device are stored in the storage unit, and the indexes of the data in the data blocks of the plurality of SSTable files are obtained by parsing each index block of the plurality of SSTable files by the computing unit.
5. The method of claim 2 , further comprising:
based on receiving a key-value pair, storing a value of the key-value pair in a storage unit of the storage device according to a size of a key of the key-value pair, and storing a key-address pair comprising the key of the key-value pair and an address in a form of an SSTable file in the storage device based on the LSM-Tree mechanism, wherein the address is a storage addresses of the value in the storage device, and
wherein the data in each of the data blocks of the plurality of SSTable files is a key-address pair.
6. The data compaction method of claim 1 , wherein the storage device is a universal flash storage (UFS) device.
7. A data compaction method, comprising:
parsing, by a storage device, index blocks of a plurality of Sorted String Table (SSTable) files on which a compaction operation is to be performed, the plurality of SSTable files stored based on a Log Structured Merge Tree (LSM-Tree) mechanism in the storage device, to obtain indexes of data in data blocks of the plurality of SSTable files;
sending, by the storage device, the indexes of the data in the data blocks of the plurality of SSTable files to an electronic device to perform merge sorting on the data in the data blocks of the plurality of SSTable files and obtain reconstructed data blocks;
performing, by the storage device, index reconstruction on data in the reconstructed data blocks to obtain reconstructed index blocks corresponding to the reconstructed data blocks; and
stitching, by the storage device, each of the reconstructed data blocks and a corresponding index block among the reconstructed index blocks to obtain a reconstructed SSTable file.
8. The method of claim 7 , wherein the storage device is a SmartSSD comprising a computing unit and a storage unit, and
wherein the computing unit is configured to perform the parsing the index blocks of the plurality of SSTable files, the performing the index reconstruction on the data in the reconstructed data blocks, and the stitching each of the reconstructed data blocks and the corresponding index block.
9. An electronic device comprising:
a determining unit configured to determine a plurality of Sorted String Table (SSTable) files on which a compaction operation is to be performed among SSTable files stored in a storage device, wherein data in the SSTable files stored in the storage device is stored based on a Log Structured Merge Tree (LSM-Tree) mechanism;
a merge sorting unit configured to perform merge sorting on data in data blocks of the plurality of SSTable files based on indexes of the data in the data blocks of the plurality of SSTable files received from the storage device, to obtain reconstructed data blocks; and
a sending unit configured to send the reconstructed data blocks to the storage device, wherein the reconstructed data blocks are used by the storage device to obtain reconstructed SSTable files,
wherein each SSTable file comprises a data block and an index block, wherein indexes in each index block correspond to data in a data block corresponding to the index block.
10. The electronic device of claim 9 , wherein the merge sorting unit is further configured to:
parse the data blocks of the plurality of SSTable files to obtain the data in the data blocks of the plurality of SSTable files;
perform the merge sorting on the data in the data blocks of the plurality of SSTable files based on the indexes of the data in the data blocks of the plurality of SSTable files; and
reconstruct data obtained through the merge sorting into data blocks according to a data block format.
11. The electronic device of claim 10 , wherein the indexes of the data in the data blocks of the plurality of SSTable files are obtained by parsing each index block of the plurality of SSTable files by the storage device.
12. The electronic device of claim 11 , wherein the storage device is a Smart SSD comprising a storage unit and a computing unit, the SSTable files stored in the storage device are stored in the storage unit, and the indexes of the data in the data blocks of the plurality of SSTable files are obtained by parsing each index block of the plurality of SSTable files by the computing unit.
13. The electronic device of claim 10 , further comprising:
a storing unit configured to store a value of a key-value pair in a storage unit of the storage device according to a size of a key of the key-value pair, and store a key-address pair comprising the key of the key-value pair and an address in a form of an SSTable file in the storage device based on the LSM-Tree mechanism, wherein the address is a storage addresses of the value in the storage device, and
wherein the data in each of the data blocks of the plurality of SSTable files is a key-address pair.
14. The electronic device of claim 9 , wherein the electronic device further comprises the storage device comprising:
a parsing unit configured to parse index blocks of the plurality of SSTable files on which the compaction operation is to be performed, to obtain indexes of data in data blocks of the plurality of SSTable files;
a sending unit configured to send the indexes of the data in the data blocks of the plurality of SSTable files to the electronic device, to perform merge sorting on the data in the data blocks of the plurality of SSTable files and obtain reconstructed data blocks;
an index reconstruction unit configured to perform index reconstruction on data in the reconstructed data blocks to obtain reconstructed index blocks corresponding to the reconstructed data blocks; and
a stitching unit configured to stitch each of the reconstructed data blocks and a corresponding index block among the reconstructed index blocks to obtain a reconstructed SSTable files.
15. The electronic device of claim 9 , wherein the storage device is SmartSSD.
16. The electronic device of claim 9 , wherein the electronic device and the storage device constitute a universal flash storage (UFS) system, wherein the electronic device is a UFS host, and wherein the storage device is a UFS device, wherein the electronic device is connected to the storage device through a UFS interface.
17. The electronic device of claim 9 , wherein the electronic device and the storage device constitute a data center system, wherein the electronic device is included in an application server, and wherein the storage device is included in a storage server.