IP Library › Granted Patent US 12,743,233
Granted Patent B2
US 12,743,233 · App. 18/406,767 · Granted Sep 22, 2026

Accumulating I/O operations of data and checksums into a single combined I/O operation for implementation by an underlying storage device layer

Inventors: James Alastair Taylor (Livermore, CA); Suhas Girish Urkude (San Ramon, CA)
Assignee: NetApp, Inc.
G06F3/0659G06F3/0611G06F3/0673
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,743,233
App. No.
18/406,767
Granted
Sep 22, 2026
Kind
B2
Abstract

Techniques are provided for combining data block and checksum block I/O into a single I/O operation. Many storage systems utilize checksums to verify the integrity of data blocks stored within storage devices managed by a storage stack. However, when a storage system reads a data block from a storage device, a corresponding checksum must also be read to verify integrity of the data in the data block. This results in increased latency because two read operations are being processed through the storage stack and are being executed upon the storage device. To reduce this latency and improve I/O operations per second, a single combined I/O operation corresponding to a contiguous range of blocks including the data block and the checksum block is processed through the storage stack instead of two separate I/O operations. Additionally, I/O operation may be combined into a single request that is executed upon the storage device.

Claims (48)

1 . A system, comprising:

one or more processors; and

a non-transitory computer-readable medium, coupled to the one or more processors, having stored therein instructions that when executed by the one or more processors cause the one or more processors to:

receive a first I/O operation targeting a set of data blocks of a storage device and a second I/O operation targeting checksums for the set of data blocks, wherein the checksums are stored in a checksum block;

combine the first I/O operation and the second I/O operation into a combined I/O operation targeting the set of data blocks and the checksums; and

transmit the combined I/O operation to access the set of data blocks, and the checksums from the storage device.

2 . The system of claim 1 , embodying a set of instructions, which when executed by the one or more processors further cause the one or more processors to:

determine that the set of data blocks and the checksum block form a dis-contiguous set of data blocks within the storage device; and

in response to determining that set of data blocks and the checksum block form the dis-contiguous set of data blocks, combine the first I/O operation and the second I/O operation into the combined I/O operation.

3 . The system of claim 1 , embodying a set of instructions, which when executed by the one or more processors further cause the one or more processors to:

in response to determining that a plurality of I/O operations are not being received at the rate faster than the rate at which the plurality of I/O operations are being transmitted, refrain from combining the combined I/O operation with other I/O operations, wherein the combined I/O operation is transmitted to an underlying storage device layer.

4 . The system of claim 1 , embodying a set of instructions, which when executed by the one or more processors further cause the one or more processors to:

receive the first I/O operation at a storage stack.

5 . The system of claim 1 , embodying a set of instructions, which when executed by the one or more processors further cause the one or more processors to:

store, in a zone, the set of data blocks and the checksum block.

6 . The system of claim 1 , wherein a second checksum block, for a second set of data, is stored in a second zone different from a zone storing the checksum block.

7 . The system of claim 1 , embodying a set of instructions, which when executed by the one or more processors further cause the one or more processors to route I/O operations through an intermediary layer to a storage layer.

8 . The system of claim 1 , embodying a set of instructions, which when executed by the one or more processors further cause the one or more processors to:

combine a plurality of I/O operations targeting data blocks and checksum blocks of the storage device into a single combined I/O operation; and

transmit the single combined I/O operation to access the data blocks and the checksum blocks in the storage device.

9 . The system of claim 1 , embodying a set of instructions, which when executed by the one or more processors further cause the one or more processors to:

receive a response to the combined I/O operation; and

transmit a first I/O response for the first I/O operation and a second I/O response for the second I/O operation.

10 . The system of claim 1 , embodying a set of instructions, which when executed by the one or more processors further cause the one or more processors to:

generate a delay between receiving and transmitting I/O operations to the storage device, and

accumulate one or more I/O operations during the delay in order to combine the one or more I/O operations into a single combined I/O operation to transmit to the storage device.

11 . The system of claim 1 , embodying a set of instructions, which when executed by the one or more processors further cause the one or more processors to:

generate a delay between receiving and transmitting I/O operations to the storage device, and

accumulate one or more I/O operations during the delay in order to combine the one or more I/O operations into a single combined I/O operation; and

adjust the delay to either increase or decrease the delay.

12 . The system of claim 1 , embodying a set of instructions, which when executed by the one or more processors further cause the one or more processors to:

adjust a delay to combine one or more I/O operations into a single combined I/O operation, wherein the delay is adjusted based upon at least one of a number of I/O operations being received during the delay or a round trip time latency between a storage layer and an underlying storage device layer.

13 . The system of claim 1 , embodying a set of instructions, which when executed by the one or more processors further cause the one or more processors to:

combine a read I/O operation and a write I/O operation into a single combined I/O operation to transmit to the storage device.

14 . A method comprising:

receiving a first I/O operation targeting a set of data blocks of a storage device and a second I/O operation targeting checksums for the set of data blocks, wherein the checksums are stored in a checksum block;

combining the first I/O operation and the second I/O operation into a combined I/O operation targeting the set of data blocks and the checksums; and

transmitting the combined I/O operation to access the set of data blocks, and the checksums from the storage device.

15 . The method of claim 14 , further comprising:

receiving a response to the combined I/O operation from the storage device; and

transmitting separate I/O responses in response to the first I/O operation and the second I/O operation.

16 . A non-transitory machine readable medium comprising instructions, which when executed by a machine, causes the machine to:

receive a first I/O operation targeting a set of data blocks of a storage device and a second I/O operation targeting checksums for the set of data blocks, wherein the checksums are stored in a checksum block;

combine the first I/O operation and the second I/O operation into a combined I/O operation targeting the set of data blocks and the checksums; and

transmit the combined I/O operation to access the set of data blocks, and the checksums from the storage device.

17 . The non-transitory machine readable medium of claim 16 , wherein the instructions cause the machine to:

receive a response to the combined I/O operation from an underlying storage device layer; and

transmit, by a storage layer, separate I/O responses through a storage stack in response to the first I/O operation and the second I/O operation.

Continuity (2)
Continuation 17716978 · Apr 8, 2022
Related Publication 20240143234A1 · May 2, 2024
References Cited (59)
US 8190784B1 · Raizen et al. · 2012 [cited by applicant]
US 8244810B1 · Haldar · 2012 [cited by applicant]
US 8886845B1 · Clark et al. · 2014 [cited by applicant]
US 9495102B2 · Bisht · 2016 [cited by applicant]
US 9891866B1 · Lazier et al. · 2018 [cited by applicant]
US 10922231B1 · Gray et al. · 2021 [cited by applicant]
US 10949289B1 · Monakhov et al. · 2021 [cited by applicant]
US 11954348B2 · Taylor et al. · 2024 [cited by applicant]
US 12474851B2 · Taylor et al. · 2025 [cited by applicant]
US 20040153746A1 · Talagala et al. · 2004 [cited by applicant]
US 20080025126A1 · Jewell et al. · 2008 [cited by applicant]
US 20080282105A1 · Deenadhayalan et al. · 2008 [cited by applicant]
US 20130061110A1 · Zvibel · 2013 [cited by applicant]
US 20130091307A1 · Lai et al. · 2013 [cited by applicant]
US 20130132801A1 · Hafner et al. · 2013 [cited by applicant]
US 20130262398A1 · Gygi et al. · 2013 [cited by applicant]
US 20140229657A1 · Karamov et al. · 2014 [cited by applicant]
US 20150081967A1 · Pan · 2015 [cited by applicant]
US 20150220275A1 · Oh · 2015 [cited by examiner]
US 20170357572A1 · Okubo · 2017 [cited by examiner]
US 20180150220A1 · Pandian et al. · 2018 [cited by applicant]
US 20190087126A1 · Chun · 2019 [cited by applicant]
US 20190146867A1 · Flynn · 2019 [cited by examiner]
US 20200341915A1 · Lecrone et al. · 2020 [cited by applicant]
US 20220130438A1 · Jung · 2022 [cited by applicant]
US 20230325098A1 · Taylor et al. · 2023 [cited by applicant]
US 20230325115A1 · Taylor et al. · 2023 [cited by applicant]
US 20240256164A1 · Taylor et al. · 2024 [cited by applicant]
Non-Final Office Action mailed on May 16, 2023 for U.S. Appl. No. 17/716,978, filed Apr. 8, 2022, 31 pages. [cited by applicant]
Notice of Allowance mailed on Oct. 17, 2023 for U.S. Appl. No. 17/716,978, filed Apr. 8, 2022, 4 pages. [cited by applicant]
Notice of Allowance mailed on Sep. 1, 2023 for U.S. Appl. No. 17/716,978, filed Apr. 8, 2022, 7 pages. [cited by applicant]
A2HOSTING., “How to Generate and Verify File Checksums on Linux,” Search the KB, Reprinted from the Internet at: https://www.a2hosting.com/kb/developer-corner/linux/working-with-file-checksums/, 9 pages. [cited by applicant]
Astra Control Services., “Astra Control Service Documentation”, Aug. 2022, NetApp, Astra Control Services, Reprinted from the Internet at: https://docs.netapp.com/us-en/astra-control-service/, 114 pages. [cited by applicant]
Astra-Protect., Application Data Protection, Mobility and Storage for Cloud Native Applications, 2022, Cloud Central, Astra Family, Astra-Protect, Move and Store Your Kubernetes Persistent Data, Reprinted from the Inter… [cited by applicant]
FERROUSLEPIDOPTERA., “Scatter Gather List,” Everything2.com, 2000, 1 page. [cited by applicant]
Gao K., et al., “Combining I/O Operations for Multiple Array Variables in Parallel NetCDF,” IEEE International Conference on Cluster Computing, 2009, 10 pages. [cited by applicant]
Hasan F., “How Does Checksum Work?,” A Concise Shot of Dev Knowledge, Reprinted from the Internet at: https://www.educative.io/answers/how-does-checksum-work, 5 pages. [cited by applicant]
Kwon Y., et al., “Turn Your Storage Stack into a File System,” 2017, 7 pages. [cited by applicant]
LIFEWIRE., “What Is a Checksum?”, Tim Fisher, Checksum Definition, Examples, and More, Software Apps, Windows, Lifewire Tech for Humans, Feb. 2022, Reprinted from the Internet at: https://www.lifewire.com/what-does-chec… [cited by applicant]
Mark B., “External NetApp Storage for Azure Stack”, Nov. 2019, 23 pages. [cited by applicant]
Moyer B., “How Does Scatter/Gather Work?,” Promises of Single-Cycle Access Are True, Butt . . . , EE Journal, 2017, Reprinted from the Internet at: https://www.eejournal.com/article/20170209-scatter-gather/, 14 pages. [cited by applicant]
Netapp.,“Astra Trident 22.07 documentation”, Aug. 12, 2022, https://docs.netapp.comlus-enltridentlindex.html/, 283 pages. [cited by applicant]
Netapp., “Design and Architecture Guide, Storage Configuration for Trident,” 2021, 13 pages. Retrieved from Internet URL: https://netapp-trident.readthedocs.io/en/stable-v21.04/dag/kubernetes/index.html. [cited by applicant]
NetApp; ONTAP 9 Documentation; ONTAP 9; Aug. 18, 2022; https://docs.netapp.com/us-en/ontap/index.html; 2712 Pgs. [cited by applicant]
Netapp., “ONTAP Guarantee”, Ontap / Performance-admin / Guarantee-throughput-qos-task.adoc, Retrieved on Feb. 6, 2023, 8 pages. [cited by applicant]
Netapp., “The Key Pillars of the Astra Data Store Architecture”, https://cloud.netapp.comibloglastra-big-kubernetes-applications-get-an-enterprise-grade-platform-with-astra-data-store, Apr. 13, 2022, 5 pages. [cited by applicant]
Netapp.,“Get Started Astra Data Store”, Jul. 25, 2022, https://docs.netapp.comlus-enlastra-data-storelgetstartedlrequirements.html/, 41 pages. [cited by applicant]
Notice of Allowance mailed on Aug. 24, 2023 for U.S. Appl. No. 17/716,977, filed Apr. 8, 2022, 9 pages. [cited by applicant]
Notice of Allowance mailed on Dec. 8, 2023 for U.S. Appl. No. 17/716,977, filed Apr. 8, 2022, 5 pages. [cited by applicant]
Notice of Allowance mailed on Oct. 24, 2023 for U.S. Appl. No. 17/716,977, filed Apr. 8, 2022, 2 pages. [cited by applicant]
Oracle., “Scatter-Gather Access,” Programming Interfaces Guide, Oracle Corporation and/or its Affiliates, 2010, Reprinted from the Internet at: https://docs.oracle.com/cd/E19683-01/816-5042/rsmapi-18/index.html, 3 pages. [cited by applicant]
Oracle., “Using Scatter/Gather DMA,” Windows Drivers/Micorsoft Docs, Windows Hardware Developer, Kernel-Mode Driver Architecture, 2021, Reprinted from the Internet at: https://docs.oracle.com/cd/E19683-01/816-5042/rsmap… [cited by applicant]
Parisi J., et al., “NFS in NetApp ONTAP Best Practice and Implementation Guide,” National Science Foundation (.gov), 2023, 171 pages. [cited by applicant]
Teofilo L., “NetApp Storage QoS Tutorial”, Flackbox, Retrieved on Feb. 6, 2023, 27 pages. Retrieved from Internet URL: https://www.flackbox.com/netapp-storage-qos-tutorial. [cited by applicant]
Zhaofeng Z., “How to Design a Storage Layer for Structured Data Storage Requirements,” Alibaba Cloud Storage, 2020, Reprinted from the Internet at: https://www.alibabacloud.com/blog/how-to-design-a-storage-layer-for-str… [cited by applicant]
Non-Final Office Action mailed on Feb. 12, 2025 for U.S. Appl. No. 18/629,333, filed Apr. 8, 2024, 18 pages. [cited by applicant]
Notice of Allowance mailed on Apr. 11, 2024 for U.S. Appl. No. 17/716,978, filed Apr. 8, 2022, 07 pages. [cited by applicant]
Corrected Notice of Allowance mailed on Aug. 6, 2025 for U.S. Appl. No. 18/629,333, filed Apr. 8, 2024, 02 pages. [cited by applicant]
Notice of Allowance mailed on Jul. 23, 2025 for U.S. Appl. No. 18/629,333, filed Apr. 8, 2024, 05 pages. [cited by applicant]