Integrating change tracking of storage objects of a distributed object storage database into a distributed storage system
In one embodiment, distributed data storage systems and methods integrate a change tracking manager with scalable databases. According to one embodiment, a computer implemented method comprises integrating change tracking of storage objects into the distributed object storage database that includes a first database of a first type and one or more chapter databases of a second type with the distributed object storage database supporting a primary lookup index and a secondary lookup index in order to locate a storage object. The method includes recording in a header of a chapter database a network topology for connecting a bucket having the chapter database to a first peer bucket when a new mirror to the first peer bucket is being established, and recording a first directive into the header of the chapter database to express a type of content to be mirrored from the bucket to the first peer bucket.
1 . A computer implemented method performed by one or more processing resources of a distributed object storage database, the method comprising:
integrating change tracking of storage objects into the distributed object storage database that includes a first database of a first type and one or more chapter databases of a second type with the distributed object storage database supporting a primary lookup index that is sorted by name of storage objects and a secondary lookup index for each chapter database that includes pending work items to be sorted by a peer identity in order to locate a storage object;
recording an array of multiple directives into a header of a chapter database to describe pending work items from a bucket to one or more peer buckets to allow continual collective tracking of an overall replication state from the bucket to one or more peer buckets; and
sorting the pending work items in the header of each chapter database based on urgency of performing each pending work item and a recovery point objective (RPO) for each peer bucket to prioritize urgent pending work items.
2 . The computer implemented method of claim 1 , further comprising:
recording a first directive into the header of the chapter database to express a type of content to be mirrored from the bucket to a first peer bucket;
recording a second directive into the header of the chapter database to express a type of content to be cached from the bucket to a second peer bucket; and
recording a third directive into the header of the chapter database to express a type of content to be archived from the bucket to a third peer bucket.
3 . The computer implemented method of claim 1 , wherein the distributed object storage database continuously tracks changes of the storage objects that are stored in the distributed object storage database without using a separate transaction log database.
4 . The computer implemented method of claim 1 , wherein the header is a reserved space of less than 10 kilobits in the chapter database.
5 . The computer implemented method of claim 1 , further comprising:
recording in a header of a chapter database a network topology for connecting a bucket having the chapter database to a first peer bucket when a new mirror to the first peer bucket is being established.
6 . The computer implemented method of claim 1 , further comprising:
updating the header of each chapter database to determine which chapter databases among a large number of chapter databases within the bucket having urgent pending work items based on a recovery point objective (RPO); and
performing the urgent pending work items based on the RPO.
7 . A storage node comprising:
one or more processing resources; and
a non-transitory computer-readable medium coupled to the processing resource, having stored therein instructions, which when executed by the one or more processing resources cause the one or more processing resources to:
manage storage objects and continuously track changes of the storage objects in a distributed object storage database that includes one or more chapter databases with the distributed object storage database supporting a primary lookup index and a secondary lookup index for each chapter database that includes pending work items to be sorted by a peer identity in order to locate a record;
generate a chapter database header for each chapter database to provide a summary of pending work items to be performed;
recording an array of multiple directives into a header of a chapter database to describe pending work items from a bucket to one or more peer buckets to allow continual collective tracking of an overall replication state from the bucket to one or more peer buckets;
sorting the pending work items in the header based on urgency of performing each pending work item and a recovery point objective (RPO) for each peer bucket to prioritize urgent pending work items; and
update the chapter database header when work items are updated for the chapter database.
8 . The storage node of claim 7 , wherein the processing resource is configured to execute instructions to:
record a network topology for a link between a first bucket and a mirrored second bucket of the distributed object storage database in the chapter database header for the first bucket when a new mirror is established to the second bucket.
9 . The storage node of claim 7 , wherein the one or more processing resources is configured to execute instructions to:
add directive information into the chapter database header for a first bucket for each link from the first bucket to one or more peer buckets of the distributed object storage database.
10 . The storage node of claim 7 , wherein each chapter database of the distributed object storage database to independently record its own state regarding which peer buckets are to be updated and to provide semantics for each peer link for caching, for archival, for mirroring, or for migration, wherein the distributed object storage database continuously tracks changes of the storage objects that are stored in the distributed object storage database without using a separate transaction log database.
11 . The storage node of claim 7 , wherein the one or more processing resources is configured to execute instructions to:
update the header of each chapter database to determine which chapter databases among a large number of chapter databases within a first bucket having urgent pending work items based on a recovery point objective (RPO) having an intentional time-delay.
12 . The storage node of claim 11 , wherein the one or more processing resources is configured to execute instructions to:
perform the urgent pending work items based on the RPO having the intentional time-delay; and
delay the non-urgent pending work items based on the RPO to eliminate processing of pending work items that are transient in existing a shorter time period than the intentional time-delay of the RPO.
13 . A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processing resources of a distributed storage system cause the one or more processing resources to:
manage storage objects and continuously track changes of the storage objects in a distributed object storage database that includes one or more chapter databases with the distributed object storage database supporting a primary lookup index and a secondary lookup index for each chapter database that includes pending work items to be sorted by a peer identity in order to locate a record;
recording an array of multiple directives into a header of a chapter database to describe pending work items from a bucket to one or more peer buckets to allow continual collective tracking of an overall replication state from the bucket to one or more peer buckets; and
sorting the pending work items in the header based on urgency of performing each pending work item and a recovery point objective (RPO) for each peer bucket to prioritize urgent pending work items.
14 . The non-transitory computer-readable storage medium of claim 13 , wherein the one or more processing resources are configured to execute the set of instructions to:
update the header of the chapter database when work items are updated for the chapter database.
15 . The non-transitory computer-readable storage medium of claim 13 , wherein the one or more processing resources are configured to execute the set of instructions to:
initially assign a first processing resource to the pending work items of the header of a chapter database of the bucket; and
dynamically assign a second processing resource to the pending work items of the header of the chapter database of the bucket in response to changes in the pending work items of the header of the chapter database of the bucket.
16 . The non-transitory computer-readable storage medium of claim 13 , wherein the one or more processing resources are configured to execute the set of instructions to:
add directive information into the header of the chapter database for the bucket for each link from the bucket to one or more peer buckets.
17 . The non-transitory computer-readable storage medium of claim 13 , wherein each chapter database to independently record its own state regarding which peer buckets are to be updated and to provide semantics for each peer link for caching, for archival, for mirroring, or for migration, wherein the distributed object storage database continuously tracks changes of the storage objects that are stored in the distributed object storage database without using a separate transaction log database.