LAZY INDEX CONSTRUCTION OF SNAPSHOTS IN A REPLICATION RECEIVER
A lazy index construction technique efficiently and cost effectively manages creation and storage of an index data structure based on characteristics of storage media used by an archival storage system. The index data structure (index) is configured to reference snapshot data of snapshots stored in the archival storage system. The technique is configured to defer creation and storage of the index on the archival storage system in a lazy manner until all snapshot data is received by a replication receiver and stored on the storage media so that updates/changes to the index on the storage media are minimized. The technique may be used with any type or combination of (i) “overwrite” data structure embodied as an index (i.e., an index data structure with overwrite capabilities) stored in an (ii) archival storage system having storage media (e.g., an object store) that is not conducive to overwrite capabilities.
1 . A method comprising:
receiving, at a replication receiver, a replication transaction of data blocks of a snapshot of a logical entity, the snapshot addressable by a client using a first address space;
writing, by the replication receiver, the data blocks to objects of an object store, wherein modifications to the objects include copying unchanged information to the object store;
accumulating, as the data blocks are received at the replication receiver, metadata locating the blocks in the first address space to transitory storage at the replication receiver, wherein the transitory storage supports overwrite without copying of unchanged information;
deferring construction of an index data structure until the replication transaction completes, wherein the index data structure is constructed by (i) reading the accumulated metadata, and (ii) mapping the first address space of the snapshot to a second address space addressable by the client for the objects based on the accumulated metadata; and
writing the constructed index data structure to the object store.
2 . The method of claim 1 , wherein the index data structure is constructed such that a maximum index structure depth is maintained to locate any of the data blocks.
3 . The method of claim 2 , wherein the maximum index structure depth is maintained by controlling a branch factor for internal nodes of the index data structure.
4 . The method of claim 1 , wherein internal nodes of the index data structure include keys as a list of offsets in the first address space, wherein each key is associated with a corresponding reference to a child node.
5 . The method of claim 1 , wherein the logical entity is organized according to extents and wherein the extents are organized in the objects according to a preferred size of the objects for the object store.
6 . The method of claim 1 , wherein the data blocks are included in a first object of the object store different from a second object of the object store having the index data structure.
7 . The method of claim 1 , further comprising organizing the index data structures in a repository according to snapshot.
8 . The method of claim 1 , wherein a first leaf node of the index data structure identifies a first object of the object store having the data blocks of the snapshot, and wherein a second leaf node of another index data structure identifies a second object of the object store having data blocks changed from the snapshot.
9 . The method of claim 1 , wherein accumulating, as the data blocks are received at the replication receiver, the metadata locating the blocks in the first address space to transitory storage further comprises recording the metadata in a database.
10 . A non-transitory computer readable medium including program instructions for execution on a processor, the program instructions configured to:
receive, at a replication receiver, a replication transaction of data blocks of a snapshot of a logical entity, the snapshot addressable by a client using a first address space;
write, by the replication receiver, the data blocks to objects of an object store, wherein modifications to the objects include copying unchanged information to the object store;
accumulate, as the data blocks are received at the replication receiver, metadata locating the blocks in the first address space to transitory storage at the replication receiver, wherein the transitory storage supports overwrite without copying of unchanged information;
defer construction of an index data structure until the replication transaction completes, wherein the index data structure is constructed by (i) reading the accumulated metadata, and (ii) mapping the first address space of the snapshot to a second address space addressable by the client for the objects based on the accumulated metadata; and
write the constructed index data structure to the object store.
11 . The non-transitory computer readable medium of claim 10 wherein the index data structure is constructed such that a maximum index structure depth is maintained to locate any of the data blocks.
12 . The non-transitory computer readable medium of claim 11 wherein the maximum index structure depth is maintained by controlling a branch factor for internal nodes of the index data structure.
13 . The non-transitory computer readable medium of claim 10 wherein internal nodes of the index data structure include keys as a list of offsets in the first address space, wherein each key is associated with a corresponding reference to a child node.
14 . The non-transitory computer readable medium of claim 10 wherein the logical entity is organized according to extents and wherein the extents are organized in the objects according to a preferred size of the objects for the object store.
15 . The non-transitory computer readable medium of claim 10 wherein the data blocks are included in a first object of the object store different from a second object of the object store having the index data structure.
16 . The non-transitory computer readable medium of claim 10 wherein the program instructions are further configured to organize the index data structures in a repository according to snapshot.
17 . The non-transitory computer readable medium of claim 10 wherein a first leaf node of the index data structure identifies a first object of the object store having the data blocks of the snapshot, and wherein a second leaf node of another index data structure identifies a second object of the object store having data blocks changed from the snapshot.
18 . The non-transitory computer readable medium of claim 10 wherein the program instructions configured to accumulate, as the data blocks are received at the replication receiver, the metadata locating the blocks in the first address space to transitory storage are further configured to record the metadata in a database.
19 . An apparatus comprising:
a network interface of a node having a processor executing a replication receiver configured to execute program instructions configured to:
receive a replication transaction of data blocks of a snapshot of a logical entity via the network interface, the snapshot addressable by a client using a first address space;
write the data blocks to objects of an object store, wherein modifications to the objects include copying unchanged information to the object store;
accumulate, as the data blocks are received, metadata locating the blocks in the first address space to transitory storage at the replication receiver, wherein the transitory storage supports overwrite without copying of unchanged information;
defer construction of an index data structure until the replication transaction completes, wherein the index data structure is constructed by (i) reading the accumulated metadata, and (ii) mapping the first address space of the snapshot to a second address space addressable by the client for the objects based on the accumulated metadata; and
write the constructed index data structure to the object store.
20 . The apparatus of claim 19 wherein the index data structure is constructed such that a maximum index structure depth is maintained to locate any of the data blocks.
21 . The apparatus of claim 20 wherein the maximum index structure depth is maintained by controlling a branch factor for internal nodes of the index data structure.
22 . The apparatus of claim 19 wherein internal nodes of the index data structure include keys as a list of offsets in the first address space, wherein each key is associated with a corresponding reference to a child node.
23 . The apparatus of claim 19 wherein the logical entity is organized according to extents and wherein the extents are organized in the objects according to a preferred size of the objects for the object store.
24 . The apparatus of claim 19 wherein the data blocks are included in a first object of the object store different from a second object of the object store having the index data structure.
25 . The apparatus of claim 19 wherein the program instructions are further configured to organize the index data structures in a repository according to snapshot.
26 . The apparatus of claim 19 wherein a first leaf node of the index data structure identifies a first object of the object store having the data blocks of the snapshot, and wherein a second leaf node of another index data structure identifies a second object of the object store having data blocks changed from the snapshot.
27 . The apparatus of claim 19 wherein the program instructions configured to accumulate, as the data blocks are received at the replication receiver, the metadata locating the blocks in the first address space to transitory storage are further configured to record the metadata in a database.