IP Library › Granted Patent US 12,443,345
Granted Patent B2
US 12,443,345 · App. 18/453,967 · Granted Oct 14, 2025

Multi-level data storage device and operation method thereof

Inventors: Kyoungho Koo (Daejeon, KR); Youjip Won (Daejeon, KR)
Assignees: SK hynix Inc.; Korea Advanced Institute of Science and Technology
G06F3/061G06F3/0653G06F3/0673G06F2212/7208
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,443,345
App. No.
18/453,967
Granted
Oct 14, 2025
Kind
B2
Abstract

A multi-level data storage device includes a first storage device; a second storage device located at a lower level than the first storage device; an input/output (I/O) control circuit configured to control a first write operation for the first storage device and a second write operation for the second storage device; and an imbalance control circuit configured to calculate an imbalance index corresponding to a write set that is generated when a sum of a number of first write operations and a number of the second write operations becomes a predetermined number, and configured to control the I/O control circuit to control imbalance of write operations performed in the multi-level data storage device by controlling the first write operation or the second write operation based on the imbalance index.

Claims (31)

1. A multi-level data storage device comprising:

a first storage device;

a second storage device located at a lower level than the first storage device;

an input/output (I/O) control circuit configured to control a first write operation for the first storage device and a second write operation for the second storage device; and

an imbalance control circuit configured to calculate an imbalance index corresponding to a write set that is generated when a sum of a number of first write operations and a number of second write operations becomes a predetermined number, and configured to control the I/O control circuit to control imbalance of write operations performed in the multi-level data storage device by controlling the first write operation or the second write operation based on the imbalance index,

wherein the imbalance control circuit calculates the imbalance index by dividing an average latency of the second write operations included in the write set by an average latency of the first write operations included in the write set, and

wherein, when the write set is a first write set, the imbalance control circuit controls the I/O control circuit to reduce an imbalance index for a second write set when the imbalance index for the first write set indicates an imbalance state, the second write set being generated to follow the first write set.

2. The multi-level storage device of claim 1 , wherein the imbalance control circuit monitors a minimum value of an imbalance index and generates a threshold value from the minimum value to be compared with the imbalance index.

3. The multi-level storage device of claim 1 , wherein the imbalance control circuit controls the I/O control circuit to insert a predetermined waiting time between first write operations included in the second write set when the imbalance state is determined.

4. The multi-level storage device of claim 1 , wherein the first storage device and the second storage device each store a key-value (KV) set according to levels of a log-structured merge (LSM) tree,

wherein the first storage device stores a plurality of KV sets corresponding to level 0 of the LSM tree, and

wherein when the imbalance state is determined, the imbalance control circuit controls the I/O control circuit to change a KV set corresponding to the level 0 into one or more KV sets corresponding to level 1 to keep the one or more KV sets corresponding to the level 1 in the first storage device.

5. The multi-level storage device of claim 4 , wherein when a used storage space of the first storage device exceeds a space threshold, the imbalance control circuit controls the I/O control circuit to move the one or more KV sets corresponding to the level 1 stored in the first storage device to the second storage device.

6. The multi-level storage device of claim 4 , wherein if the imbalance state is not determined, the imbalance control circuit controls the I/O control circuit to store the one or more KV sets corresponding to the level 1 in the second storage device when the KV set corresponding to the level 0 is changed to the one or more KV sets in the level 1.

7. An operation method of a multi-level data storage device including a first storage device storing data by performing a first write operation and a second storage device storing data by moving the data stored in the first storage device to the second storage device by performing a second write operation, the operation method comprising:

generating a first write set when a sum of a number of first write operations and a number of second write operations becomes a predetermined number;

calculating an imbalance index corresponding to the first write set by using an average latency of the first write operations and an average latency of the second write operations corresponding to the first write set;

determining an imbalance state based on the imbalance index; and

controlling imbalance of write operations performed in the multi-level data storage device by controlling the first write operation or the second write operation so that an imbalance index corresponding to a second write set is reduced when the imbalance state is determined, the second write set being generated to follow the first write set,

wherein calculating the imbalance index includes dividing an average latency of second write operations by an average latency of first write operations, corresponding to the first write set.

8. The operation method of claim 7 , wherein determining the imbalance state includes:

monitoring a minimum value of the imbalance index and determining a threshold value from the minimum value; and

comparing the threshold value with the imbalance index.

9. The operation method of claim 7 , wherein controlling the imbalance includes inserting a predetermined waiting time between first write operations in the second write set.

10. The operation method of claim 7 , wherein the first storage device and the second storage device each store a key-value (KV) set according to levels of a log-structured merge (LSM) tree, and the first storage device stores a plurality of KV sets corresponding to level 0 of the LSM tree,

wherein when the imbalance state is determined, controlling the imbalance includes:

changing a KV set corresponding to the level 0 to one or more KV sets corresponding to level 1; and

storing the one or more KV sets corresponding to the level 1 in the first storage device without performing the second write operation.

11. The operation method of claim 10 , wherein controlling the imbalance further includes:

determining whether a used storage space of the first storage device exceeds a space threshold; and

performing a second write operation to move the one or more KV sets stored in the first storage device to the second storage device when the storage space exceeds the storage threshold.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 23, 2023
From: KOO, KYOUNGHO; WON, YOUJIP
To: SK HYNIX INC.; KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY
Reel/Frame 064686/0887 →
Priority Claims (2)
KR 10-2022-0174131 · Dec 13, 2022 · national
KR 10-2023-0098624 · Jul 28, 2023 · national
Continuity (1)
Related Publication 20240192853A1 · Jun 13, 2024
References Cited (46)
US 10761777B2 · Malina et al. · 2020 [cited by applicant]
US 20090019243A1 · Hur · 2009 [cited by examiner]
US 20130031307A1 · Itoh · 2013 [cited by examiner]
US 20190035473A1 · Rajamani · 2019 [cited by examiner]
US 20190384528A1 · Grosz · 2019 [cited by examiner]
KR 102264119B1 · 2021 [cited by applicant]
Ashok Anand et al., “Cheap and Large CAMS for High Performance Data-Intensive Networked Systems”, NSDI '10: 7th USENIX Symposium on Networked Systems Design and Implementation, Apr. 28-30, 2010, vol. 10, pp. 433-448, Sa… [cited by applicant]
Timothy G. Armstrong et al., “Linkbench: a Database Benchmark Based on the Facebook Social Graph,” In Proceedings of the 2013 ACM SIGMOD Inter-national Conference on Management of Data, Jun. 22-27, 2013, pp. 1185-1196, … [cited by applicant]
Andrew Audibert, “Scalable Metadata Service in Alluxio: Storing Billions of Files,” May 10, 2019, https://www.alluxio.io/blog/scalable-metadata-service-in-alluxio-storing-billions-of-files/. [cited by applicant]
Oana Balmau et al., “SILK: Preventing latency spikes in Log-Structured merge Key-Value stores,” 2019 USENIX Annual Technical Conference, Jul. 10-12, 2019, pp. 753-766, https://www.usenix.org/conference/atc19/presentatio… [cited by applicant]
Oana Balmau et al. “Silk+ preventing latency spikes in log-structured merge key-value stores running heterogeneous workloads”, ACM Transactions on Computer Systems (TOCS), May 2020, 36(4), p. 1-27, https://dl.acm.org/do… [cited by applicant]
Doug Beaver et al., “Finding a needle in haystack: Facebook's photo storage,” in the Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI '10), Oct. 4-6, 2010, pp. 47-60. [cited by applicant]
Zhichao Cao et al., “Characterizing, Modeling, and Benchmarking rocksDB Key-Value Workloads at Facebook,” in the Proceedings of the 18th USENIX Conference on File and Storage Technologies (FAST '20), Feb. 25-27, 2020, p… [cited by applicant]
Fay Chang et al., “Bigtable: A distributed storage system for structured data,” ACM Transactions on Computer Systems (TOCS), Jun. 2008, vol. 26, No. 2, pp. 1-26. [cited by applicant]
Hao Chen et al. “SpanDB: A Fast, Cost-Effective LSM-tree Based KV Store on Hybrid Storage”, 19th USENIX Conference on File and Storage Technologies., Feb. 23-25, 2021, pp. 17-32, https://www.usenix.org/conference/fast21… [cited by applicant]
Austin T. Clements et al. “Scalable Address Spaces Using RCU Balanced Trees”, In Architectural Support for Programming Languages and Operating Systems, Mar. 3-7, 2012, pp. 199-210. [cited by applicant]
Brian F. Cooper et al. “Pnuts: Yahoo!'s Hosted Data Serving Platform,” PVLDB '08, 2008, 1(2), Aug. 23-28, 2008, pp. 1277-1288, Auckland, New Zealand. [cited by applicant]
Brian F. Cooper et al., “Benchmarking Cloud Serving Systems with YCSB,” In Proceeding of the 1st ACM Symposium on Cloud computing (SoCC '10), Jun. 10-11, 2010, pp. 143-154. [cited by applicant]
Biplob Debnath et al., “Flashstore: High Throughput Persistent Key-Value Store,” Proceedings of the VLDB Endowment, vol. 3, No. 2, 2010, pp. 1414-1425. [cited by applicant]
Biplob Debnath et al., “Skimpystash: Ram space skimpy key-value store on flash-based storage,” In Proceedings of the 2011 ACM SIGMOD International Conference on Management of data, Jun. 12-16, 2011, pp. 25-36. [cited by applicant]
Giuseppe DeCandia et al., “Dynamo: Amazon's highly available key-value store,” ACM SIGOPS operating systems review, 41(6):205-220, 2007, pp. 205-220. [cited by applicant]
Krijn Doekemeijer and Animesh Trivedi, “Key-value stores on flash storage devices: A survey,” May 11, 2022, pp. 1-38, arXiv preprint arXiv:2205.07975, 2022. [cited by applicant]
Tyler Harter et al., “Analysis of HDFS under HBase: A facebook messages case study,” In 12th USENIX Conference on File and Storage Technologies (FAST 14), Feb. 17-20, 2014, pp. 199-212. [cited by applicant]
Gui Huang et al., “X-Engine: An optimized storage engine for large-scale e-commerce transaction processing,” In Proceedings of the 2019 International Conference on Management of Data (SIGMOD '19), Jun. 30-Jul. 5, 2019, … [cited by applicant]
J Stuart Hunter, “The exponentially weighted moving average,” Journal of quality technology, 18(4):203-210, 1986. [cited by applicant]
Junsu Im,et al . . . “Pink: High-speed in-storage key-value store with bounded tails,” In 2020 USENIX Annual Technical Conference (USENIX ATC 20), pp. 173-187, 2020. [cited by applicant]
“Intel Optane memory—Responsive Memory, Accelerated Performance,” 2021, https://www.intel.com/content/www/us/en/products/details/memory-storage/optane-memory.html. [cited by applicant]
“Optane ssd 905p,” 2022, https://www.intel.co.kr/content/ www/kr/ko/products/sku/148607/intel-optane-ssd-905p-series-380gb-m-2-110mm-pcie-x4-20nm-3d-xpoint/specifications.html. [cited by applicant]
Krish K.R. et al., “hats: A heterogeneity-aware tiered storage for hadoop,” In 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, pp. 502-511. IEEE, 2014. [cited by applicant]
Chunbo Lai et al., “Atlas: Baidu's key-value storage system for cloud data,” In 2015 31st Symposium on Mass Storage Systems and Technologies (MSST), pp. 1-14. IEEE, 2015. [cited by applicant]
Wenjie Li et al., “Hilsm: an lsm-based key-value store for hybrid nvm-ssd storage systems,” In Proceedings of the 17th ACM International Conference on Computing Frontiers, pp. 208-216, May 11-13, 2020. [cited by applicant]
Lanyue Lu et al., “Wisckey: Separating keys from values in ssd-conscious storage,” ACM Trans. Storage 13, 1, Article 5, Mar. 2017. [cited by applicant]
Chris Mellor, “Toshiba flashes 100TB QLC flash drive, may go on sale within months. really,” Micron, Aug. 10, 2016, https://www.theregister.com/2016/08/10/toshiba100tbqlcssd/. [cited by applicant]
“QLC NAND Technology,” Micron, 2020 https://www.micron.com/products/advancedsolutions/qlc-nand. [cited by applicant]
Neelima Premsankar, ‘What is a “Heterogeneous Memory” Storage Engine (HSE)?’ Micron Insight, Jun. 17, 2020, https://www.micron.com/about/blog/2020/june/what-is-a-heterogeneous-storage-engine. [cited by applicant]
Patrick O'Neil et al., “The log-structured merge-tree (lsm-tree),” Acta Informatica, 33(4):351-385, 1996. [cited by applicant]
Ashwini Raina et al., “Prismdb: Read-aware log-structured merge trees for heterogeneous storage,” Sep. 24, 2020, pp. 1-16. [cited by applicant]
Pandian Raju et al., “Pebblesdb: Building key-value stores using fragmented log-structured merge trees,” In Proceedings of the 26th Symposium on Operating Systems Principles, Oct. 28-31, 2017, pp. 497-514. [cited by applicant]
“870 QVO,” 2022, https://semiconductor.samsung.com/consumer-storage/internal-ssd/870qvo/. [cited by applicant]
“Samsung 970 evo ssd,” 2022, https://semiconductor.samsung.com/consumer-storage/internal-ssd/970evo/. [cited by applicant]
“Z-ssd redefining fast responsiveness,” Samsung Semiconductor Global Website, 2018, https://www.samsung.com/semiconductor/ssd/z-ssd/. [cited by applicant]
Roshan Sumbaly et al., “Serving large-scale batch computed data with project voldemort,” In Proceedings of FAST '12: 10th USENIX Conference on File and Storage Technologies, Feb. 15-17, 2012, pp. 223-235. [cited by applicant]
Khikmatullo Tulkinbekov and Deok-Hwan Kim. “Casedb: Light-weight key-value store for edge computing environment,” Aug. 25, 2020, pp. 149775-149786, IEEE Access, 8:149775-149786. [cited by applicant]
Kan Wu et al., “Towards an unwritten contract of intel optane SSD,” in Proceedings of the 11th USENIX Conference on Hot Topics in Storage and File Systems, Jul. 2019. [cited by applicant]
Ting Yao et al., “Matrixkv: Reducing write stalls and write amplification in lsm-tree based kv stores with matrix container in nvm,” in Proceedings of 2020 USENIX Annual Technical Conference, pp. 17-31, Jul. 15-17, 2020. [cited by applicant]
Hobin Yoon et al., “Mutant: Balancing storage cost and latency in lsm-tree data stores,” in Proceedings of the ACM Symposium on Cloud Computing (SoCC '18), Oct. 11-13, 2018, pp. 162-173. [cited by applicant]