Asynchronous namespace maintenance
Asynchronous namespace maintenance in a distributed replicated data storage system is disclosed. An access device/program serving as a front end to the distributed replicated data storage system updates a batch of updated meta data about stored data items when data items are stored in the distributed replicated data storage system. When the elapsed time since the last batch of data item meta data was stored exceeds a first threshold value or the current batch size exceeds a second threshold value, the access device/program stores the current batch of updated meta data as an object in the distributed replicated data storage system, receiving a batch object identifier for the stored batch of updated meta data, and distributes the batch object identifier to other access devices and/or access programs which retrieve the batch of updated meta data and update their namespaces.
1. An access device including an access program, the access device coupled with and serving as a front end to a distributed replicated storage system, the access program comprising instructions which when executed cause the access device to perform actions including:
updating a current batch of updated meta data about stored data items when the access device stores data items in the distributed replicated data storage system, the updating including appending meta data about stored data items to an existing batch of updated meta data about earlier stored data items to create the current batch of updated meta data about stored data items;
when an elapsed time since a last batch of data item meta data was stored exceeds a first threshold value or when a current batch size of the current batch of updated meta data exceeds a second threshold value,
storing the current batch of updated meta data as an object in the distributed replicated data storage system, including receiving a batch object identifier for the stored batch of updated meta data,
distributing the batch object identifier to other access devices and/or access programs causing the other access devices and/or access programs to retrieve the batch of updated meta data from the distributed replicated data storage system and update their namespaces.
2. The system of claim 1 wherein the updating comprises:
receiving a plurality of storage requests for data items from clients,
processing the storage requests for the data items including forwarding the storage requests to the distributed replicated data storage system and receiving corresponding stored data item identifiers,
wherein the updated meta data includes the stored data item identifiers.
3. The system of claim 1 wherein the access program includes additional instructions which when executed cause the access device to perform further actions including:
determining whether an elapsed time since a last batch of data item meta data was stored exceeds a first threshold value,
determining whether a current batch size of the current batch of data item meta data exceeds a second threshold value.
4. The system of claim 1 wherein the access devices are server computers and/or gateways.
5. The system of claim 1 wherein the distributing the batch object identifier causes the access programs on the access devices to synchronize the namespaces among the access devices.
6. The system of claim 1 wherein the distributed replicated storage system includes at least two storage zones, each storage zone includes a plurality of storage nodes, and each storage node includes a plurality of storage devices.
7. The system of claim 6 wherein each access device is associated with and acts as a front end to at least one storage zone.
8. The system of claim 6 wherein the storage devices are selected from the group including hard disk drives and solid state storage devices.
9. The system of claim 1 wherein the meta data includes at least three selected from the group including: object identifier, data item name, data item version, data item type, data item size, data item creation date, data item storage date, data item modification date, data item owner, data item author, data item object kind, flags, parent/predecessor data item identifier, storage policy identifier.