Data storage system having mutable objects incorporating time
A data storage system having mutable objects incorporating time is described herein. According to the systems and methods described herein, a data item may be partitioned into parts (data objects) and stored as an index object. As the object storage system provides immutable objects, when a new version of a data item needs to be stored, only those parts (data objects) of the data item that changed need be saved rather than the entire data item. The systems and methods described herein allow for efficient storage, access and manipulation of mutable data items using an underlying immutable object system.
1. A storage system comprising:
a first node of a plurality of storage nodes, the storage nodes including a plurality of storage devices allowing for storing immutable objects including data objects and index objects, the first node having instructions which when executed cause a first processor included in a first controller in the first node to perform actions including:
receiving a storage request from a client of the system to store a data item
partitioning the data item into a plurality of data objects, each data object including an object identifier, a timestamp and a data portion
determining which data objects of the data item have been modified by comparing a hash of the data objects with earlier stored data objects
creating data objects for only the data objects that have been modified
creating a new index object for the data item, the new index object including a new index object identifier, a new timestamp and a list of constituent data objects, the list of constituent data objects including an object identifier and a hash value for each data object and further information from a parent index object about the data objects that have not been modified and information about new data objects that represent the data objects that have been modified, the new index object including a parent index object identifier of an earlier version of the data item
storing the new index object and those data objects that have been modified
providing the new index object identifier for the new index object to the client.
2. The storage system of claim 1 wherein the storage devices are selected from the group including hard disk drives and solid state storage devices.
3. The storage system of claim 1 wherein the storage request includes a link to the data item.
4. The storage system of claim 1 wherein the new timestamp refers to a time the new index object was created.
5. The storage system of claim 1 wherein an earliest index object includes a parent index object identifier set to null or zero or other designation representing no parent.
6. The storage system of claim 1 wherein each index object further includes a policy identifier specifying a replication policy for the index object and its constituent data objects in the data storage system.
7. A storage system comprising:
a first node of a plurality of storage nodes, the storage nodes including a plurality of storage devices allowing for storing immutable objects including data objects and index objects, the first node having instructions which when executed cause a first processor included in a first controller in the first node to perform actions including:
receiving a storage request from a client of the system to store a data item
evaluating whether the data item was earlier stored
when the data item was not earlier stored,
creating a first index object for the data item, the first index object including an index object identifier, a timestamp and a list of constituent data objects, the list of constituent data objects including an object identifier and a hash for each data object
storing the first index object and the plurality of data objects
providing an object identifier for the first index object to the client.
8. The storage system of claim 7 wherein the first node has further instructions stored thereon which when executed cause the first processor to perform further actions including:
when the data item was earlier stored,
determining which data objects of the data item have been modified
creating data objects for only the data objects that have been modified
creating a second index object for the data item including in the listing of constituent data objects information about the data objects from the first index object that have not been modified and information about new data objects that represent the data objects that have been modified, the second index object including a parent index object identifier of an earlier version of the data item
incrementing a reference count value in the index object of the parent of the second index object identified by the parent index object identifier in the second index object
storing the second index object and the new data objects
providing a second object identifier for the second index object to the client.
9. The storage system of claim 7 wherein the storage devices are selected from the group including hard disk drives and solid state storage devices.
10. The storage system of claim 7 wherein the storage request includes a link to the data item.
11. The storage system of claim 7 wherein the timestamp refers to a time the first index object was created.
12. The storage system of claim 7 wherein the first index object further includes a parent index object identifier set to null or zero or other designation representing no parent.
13. The storage system of claim 7 wherein the first index object further includes a policy identifier specifying a replication policy for the first index object and its constituent data objects in the data storage system.
14. The storage system of claim 7 wherein storing the first index object and the plurality of data objects is performed according to the policy identifier.
15. The storage system of claim 7 wherein determining which data objects of the data item have been modified includes comparing the hash values of the constituent data objects of the data item.
16. The storage system of claim 8 wherein creating a second index object for the data item includes obtaining a second timestamp representing a time the second index object was created and including in the second index object the second timestamp.
17. The storage system of claim 8 wherein the second index object further includes a policy identifier specifying a replication policy for the second index object and its constituent data objects in the data storage system.
18. The storage system of claim 8 wherein storing the second index object and the new data objects are performed according to the policy identifier.