IP Library Granted Patent US 12,499,013
Granted Patent B2
US 12,499,013 · App. 18/401,916 · Granted Dec 16, 2025

Tracking writes and snapshot creation/deletion in memory to improve asynchronous replication performance and support lower recovery point objectives (RPOs)

Inventors: David Bernard (Westford, MA); Mayank Ajmera (Cary, NC); Vamsi K. Vankamamidi (Hopkinton, MA); Vikram Prabhakar (Apex, NC)
Assignee: Dell Products L.P.
G06F11/1451G06F11/1456G06F11/1464
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,499,013
App. No.
18/401,916
Granted
Dec 16, 2025
Kind
B2
Abstract

Techniques for tracking incoming writes and snap creation/deletion in memory to improve asynchronous replication and support lower RPOs. In the techniques, a storage system uses its data cache to receive data specified in write requests issued by storage clients, while dedicating an amount of the cache memory to track and record offsets/lengths of writes directed to source volumes. At the end of each replication interval, the storage system obtains a list of the recorded offsets/lengths for each source volume, identifies and reads areas of the source volume that were written to during the replication interval using the list, and replicates data from the identified areas to a destination volume. Because the list of recorded offsets/lengths of incoming writes for the source volume is compiled and available from volatile cache memory, it can be generated and accessed very quickly using reduced processing/memory resources, allowing for lower RPOs in asynchronous replication processes.

Claims (80)

1 . A method comprising:

receiving, at a storage system, first data specified in a plurality of first writes directed to a source volume in a first tracking session, the first tracking session having a first tracking session ID and corresponding to a first replication interval of an asynchronous replication process;

processing, by a first core of a central processing unit (CPU), a first portion of the plurality of first writes from the first tracking session, the first core being associated with a first hash table containing at least a first key-value pair entry, a key of the first key-value pair entry corresponding to the first tracking session ID, a value of the first key-value pair entry corresponding to a pointer to a first buffer for the first tracking session;

processing, by a second core of the CPU, a second portion of the plurality of first writes from the first tracking session, the second core being associated with a second hash table containing at least a second key-value pair entry, a key of the second key-value pair entry corresponding to the first tracking session ID, a value of the second key-value pair entry corresponding to a pointer to a second buffer for the first tracking session;

recording, in the first buffer for the first tracking session, offsets and lengths of the first portion of the plurality of first writes processed by the first core;

recording, in the second buffer for the first tracking session, offsets and lengths of the second portion of the plurality of first writes processed by the second core;

in response to an end of the first tracking session, obtaining a first list of the recorded offsets and lengths of the first portion of the plurality of first writes and the second portion of the plurality of first writes;

identifying first areas of the source volume written to in the first tracking session using the obtained first list; and

replicating the first data from the identified first areas of the source volume to a destination volume.

2 . The method of claim 1 further comprising:

receiving, at the storage system, second data specified in a plurality of second writes directed to the source volume in a second tracking session, the second tracking session having a second tracking session ID and corresponding to a second replication interval of the asynchronous replication process, the first tracking session and the second tracking session corresponding to two successive tracking sessions;

processing, by the first core of the CPU, a first portion of the plurality of second writes from the second tracking session, the first hash table containing a third key-value pair entry, a key of the third key-value pair entry corresponding to the second tracking session ID, a value of the third key-value pair entry corresponding to a pointer to a first buffer for the second tracking session;

processing, by the second core of the CPU, a second portion of the plurality of second writes in the second tracking session, the second hash table containing a fourth key-value pair entry, a key of the fourth key-value pair entry corresponding to the second tracking session ID, a value of the fourth key-value pair entry corresponding to a pointer to a second buffer for the second tracking session;

recording, in the first buffer for the second tracking session, offsets and lengths of the first portion of the plurality of second writes processed by the first core; and

recording, in the second buffer for the second tracking session, offsets and lengths of the second portion of the plurality of second writes processed by the second core.

3 . The method of claim 2 further comprising:

in response to an end of the second tracking session, obtaining a second list of the recorded offsets and lengths of the first portion of the plurality of second writes and the second portion of the plurality of second writes;

identifying second areas of the source volume written to in the second tracking session using the obtained second list; and

replicating the second data from the identified second areas of the source volume to the destination volume.

4 . The method of claim 2 wherein a first user snapshot of the source volume is created during the second tracking session, and wherein the method further comprises:

recording creation of the first user snapshot in the first buffer for the second tracking session associated with the first core that processed the first portion of the plurality of second writes directed in the second tracking session; and

recording the creation of the first user snapshot in the second buffer for the second tracking session associated with the second core that processed the second portion of the plurality of second writes in the second tracking session.

5 . The method of claim 4 further comprising:

determining that the storage system lacks sufficient volatile memory space for recording offsets and lengths of a plurality of third writes directed to the source volume in a third tracking session; and

having determined that the storage system lacks sufficient volatile memory space for recording the offsets and lengths of the plurality of third writes directed to the source volume in the third tracking session:

creating a second user snapshot of the source volume during the third tracking session; and

reverting to issuing, by the storage system, a snapshot differential bitmap request to obtain data differences between the first user snapshot of the source volume and the second user snapshot of the source volume.

6 . The method of claim 5 further comprising:

determining that the storage system has returned to a state with sufficient volatile memory space for recording offsets and lengths of a plurality of fourth writes directed to the source volume in a fourth tracking session; and

having determined that the storage system has returned to the state with sufficient volatile memory space for recording the offsets and lengths of the plurality of fourth writes directed to the source volume in the fourth tracking session:

returning to recording, by the storage system, the offsets and lengths of the plurality of fourth writes in at least one buffer for the fourth tracking session.

7 . The method of claim 2 wherein a user snapshot of the source volume is deleted during the second tracking session, and wherein the method further comprises:

recording deletion of the user snapshot in the first buffer for the second tracking session associated with the first core that processed the first portion of the plurality of second writes in the second tracking session; and

recording the deletion of the user snapshot in the second buffer for the second tracking session associated with the second core that processed the second portion of the plurality of second writes in the second tracking session.

8 . The method of claim 2 further comprising:

tagging each first write with the first tracking session ID; and

tagging each second write with the second tracking session ID.

9 . A system comprising:

a memory; and

processing circuitry configured to execute program instructions out of the memory to:

receive, at a storage system, first data specified in a plurality of first writes directed to a source volume in a first tracking session, the first tracking session having a first tracking session ID and corresponding to a first replication interval of an asynchronous replication process;

process, by a first core of a central processing unit (CPU), a first portion of the plurality of first writes from the first tracking session, the first core being associated with a first hash table containing at least a first key-value pair entry, a key of the first key-value pair entry corresponding to the first tracking session ID, a value of the first key-value pair entry corresponding to a pointer to a first buffer for the first tracking session;

process, by a second core of the CPU, a second portion of the plurality of first writes from the first tracking session, the second core being associated with a second hash table containing at least a second key-value pair entry, a key of the second key-value pair entry corresponding to the first tracking session ID, a value of the second key-value pair entry corresponding to a pointer to a second buffer for the first tracking session;

record, in the first buffer for the first tracking session, offsets and lengths of the first portion of the plurality of first writes processed by the first core;

record, in the second buffer for the first tracking session, offsets and lengths of the second portion of the plurality of first writes processed by the second core;

in response to an end of the first tracking session, obtain a first list of the recorded offsets and lengths of the first portion of the plurality of first writes and the second portion of the plurality of first writes;

identify first areas of the source volume written to in the first tracking session using the obtained first list; and

replicate the first data from the identified first areas of the source volume to a destination volume.

10 . The system of claim 9 wherein the processing circuitry is further configured to execute the program instructions out of the memory to:

receive, at the storage system, second data specified in a plurality of second writes directed to the source volume in a second tracking session, the second tracking session having a second tracking session ID and corresponding to a second replication interval of the asynchronous replication process, the first tracking session and the second tracking session corresponding to two successive tracking sessions;

process, by the first core of the CPU, a first portion of the plurality of second writes from the second tracking session, the first hash table containing a third key-value pair entry, a key of the third key-value pair entry corresponding to the second tracking session ID, a value of the third key-value pair entry corresponding to a pointer to a first buffer for the second tracking session;

process, by the second core of the CPU, a second portion of the plurality of second writes in the second tracking session, the second hash table containing a fourth key-value pair entry, a key of the fourth key-value pair entry corresponding to the second tracking session ID, a value of the fourth key-value pair entry corresponding to a pointer to a second buffer for the second tracking session;

record, in the first buffer for the second tracking session, offsets and lengths of the first portion of the plurality of second writes processed by the first core; and

record, in the second buffer for the second tracking session, offsets and lengths of the second portion of the plurality of second writes processed by the second core.

11 . The system of claim 10 wherein the processing circuitry is further configured to execute the program instructions out of the memory to:

in response to an end of the second tracking session, obtain a second list of the recorded offsets and lengths of the first portion of the plurality of second writes and the second portion of the plurality of second writes;

identify second areas of the source volume written to in the second tracking session using the obtained second list; and

replicate the second data from the identified second areas of the source volume to the destination volume.

12 . The system of claim 10 wherein a user snapshot of the source volume is created during the second tracking session, and wherein the processing circuitry is further configured to execute the program instructions out of the memory to:

record creation of the user snapshot in the first buffer for the second tracking session associated with the first core that processed the first portion of the plurality of second writes in the second tracking session; and

record the creation of the user snapshot in the second buffer for the second tracking session associated with the second core that processed the second portion of the plurality of second writes in the second tracking session.

13 . A computer program product including a set of non-transitory, computer-readable media having instructions that, when executed by processing circuitry, cause the processing circuitry to perform a method comprising:

receiving, at a storage system, first data specified in a plurality of first writes directed to a source volume in a first tracking session, the first tracking session having a first tracking session ID and corresponding to a first replication interval of an asynchronous replication process;

processing, by a first core of a central processing unit (CPU), a first portion of the plurality of first writes from the first tracking session, the first core being associated with a first hash table containing at least a first key-value pair entry, a key of the first key-value pair entry corresponding to the first tracking session ID, a value of the first key-value pair entry corresponding to a pointer to a first buffer for the first tracking session;

processing, by a second core of the CPU, a second portion of the plurality of first writes from the first tracking session, the second core being associated with a second hash table containing at least a second key-value pair entry, a key of the second key-value pair entry corresponding to the first tracking session ID, a value of the second key-value pair entry corresponding to a pointer to a second buffer for the first tracking session;

recording, in the first buffer for the first tracking session, offsets and lengths of the first portion of the plurality of first writes processed by the first core;

recording, in the second buffer for the first tracking session, offsets and lengths of the second portion of the plurality of first writes processed by the second core;

in response to an end of the first tracking session, obtaining a first list of the recorded offsets and lengths of the first portion of the plurality of first writes and the second portion of the plurality of first writes;

identifying first areas of the source volume written to in the first tracking session using the obtained first list; and

replicating the first data from the identified first areas of the source volume to a destination volume.

14 . The computer program product of claim 13 wherein the method further comprises:

receiving, at the storage system, second data specified in a plurality of second writes directed to the source volume in a second tracking session, the second tracking session having a second tracking session ID and corresponding to a second replication interval of the asynchronous replication process, the first tracking session and the second tracking session corresponding to two successive tracking sessions;

processing, by the first core of the CPU, a first portion of the plurality of second writes from the second tracking session, the first hash table containing a third key-value pair entry, a key of the third key-value pair entry corresponding to the second tracking session ID, a value of the third key-value pair entry corresponding to a pointer to a first buffer for the second tracking session;

processing, by the second core of the CPU, a second portion of the plurality of second writes in the second tracking session, the second hash table containing a fourth key-value pair entry, a key of the fourth key-value pair entry corresponding to the second tracking session ID, a value of the fourth key-value pair entry corresponding to a pointer to a second buffer for the second tracking session;

recording, in the first buffer for the second tracking session, offsets and lengths of the first portion of the plurality of second writes processed by the first core; and

recording, in the second buffer for the second tracking session, offsets and lengths of the second portion of the plurality of second writes processed by the second core.

15 . The computer program product of claim 14 wherein the method further comprises:

in response to an end of the second tracking session, obtaining a second list of the recorded offsets and lengths of the first portion of the plurality of second writes and the second portion of the plurality of second writes;

identifying second areas of the source volume written to in the second tracking session using the obtained second list; and

replicating the second data from the identified second areas of the source volume to the destination volume.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 31, 2024
From: BERNARD, DAVID; AJMERA, MAYANK; VANKAMAMIDI, VAMSI K.; PRABHAKAR, VIKRAM
To: DELL PRODUTCS L.P.
Reel/Frame 066307/0241 →
Continuity (1)
Related Publication 20250217234A1 · Jul 3, 2025
References Cited (16)
US 9720619B1 · Shah · 2017 [cited by examiner]
US 10761941B1 · Redko · 2020 [cited by examiner]
US 11269776B2 · Chen et al. · 2022 [cited by applicant]
US 11409455B2 · Chen et al. · 2022 [cited by applicant]
US 11409700B2 · Ren · 2022 [cited by examiner]
US 11514075B2 · Haravu et al. · 2022 [cited by applicant]
US 11809274B2 · Tal et al. · 2023 [cited by applicant]
US 12117989B1 · Taylor · 2024 [cited by examiner]
US 12210763B1 · Taylor · 2025 [cited by examiner]
US 20200133919A1 · Ren · 2020 [cited by examiner]
US 20200348980A1 · Meiri · 2020 [cited by examiner]
US 20210034484A1 · Meiri · 2021 [cited by examiner]
US 20230342064A1 · Haravu · 2023 [cited by examiner]
US 20230409534A1 · Kashi Visvanathan · 2023 [cited by examiner]
US 20240143554A1 · Kaushik · 2024 [cited by examiner]
US 20250061092A1 · Khilko · 2025 [cited by examiner]