Combined asynchronous and synchronous fountain code storage in an object store
View Patent ↗Example apparatus and methods produce a set of rateless erasure codes (e.g., fountain codes) for a file stored in a primary data store (e.g., hard drive) or in an archive system. The archive system may store the file in a redundant array of independent disks (RAID). A first subset of the rateless erasure codes are stored in an object storage using a synchronous protocol. A second subset of rateless erasure codes are stored in the object storage using an asynchronous protocol. The object storage system may inform the archive system when desired redundancy has been achieved or when desired redundancy has been lost. The archive system may buffer rateless erasure codes before providing the codes to the object storage to improve performance. A failure in the archive system or object storage system may be mitigated by retaining the file in the primary data store until the desired redundancy is achieved.
1. A non-transitory computer-readable storage device storing computer executable instructions that when executed by a computer control the computer to perform a method, the method comprising:
receiving a request to increase the redundancy of data stored in a first data store;
receiving the data in a second, different data store located in a computer storage device that provides an object store;
producing a plurality of erasure codes from the data;
selecting a first subset of erasure codes from the plurality of erasure codes, where the first subset provides a first degree of redundancy for the data;
selecting a second subset of erasure codes from the plurality of erasure codes, where the second subset is disjoint from the first subset, where the plurality of erasure codes provides a second, greater degree of redundancy for the data;
storing the first subset of erasure codes in the object store using a synchronous approach;
upon determining that the first subset of erasure codes has been stored in the object store, sending a first signal to the first data store indicating that the first degree of redundancy has been achieved;
storing the second subset of erasure codes in the object store using an asynchronous approach; and
upon determining that the entire plurality of erasure codes has been stored in the object store, sending a second signal to the first data store indicating that the second degree of redundancy has been achieved.
2. The non-transitory computer-readable storage device of claim 1 , where the number of erasure codes in the plurality of erasure codes is controlled by a redundancy policy, where the redundancy policy identifies a minimum number of erasure codes to be stored in the object store and a total number of erasure codes to be stored in the object store.
3. The non-transitory computer-readable storage device of claim 1 , where storing the second subset of erasure codes in the object store using the asynchronous approach includes buffering one or more erasure codes in the second data store before storing the one or more erasure codes in the object store.
4. The non-transitory computer-readable storage device of claim 3 , where storing the second subset of erasure codes in the object store using the asynchronous approach includes forcing a flush of an erasure code from the second data store to the object store.
5. The non-transitory computer-readable storage device of claim 1 , the method comprising:
upon determining that the entire plurality of erasure codes has been stored in the object store, controlling the first data store to release the data.
6. The non-transitory computer-readable storage device of claim 1 , where the data is a file associated with a file system.
7. The non-transitory computer-readable storage device of claim 1 , where the data is an object.
8. The non-transitory computer-readable storage device of claim 7 , the object comprising, data, metadata, and a globally unique identifier.
9. The non-transitory computer-readable storage device of claim 1 , where the erasure codes are rateless erasure codes.
10. An apparatus, comprising:
a processor;
a memory;
a set of logics; and
an interface that connects the processor, the memory, and the set of logics;
the set of logics comprising:
a first logic that produces a set of redundancy blocks for data stored in a primary data storage;
a second logic that causes a first subset of the set of redundancy blocks to be stored in an object store using a synchronous protocol, where the first subset provides a first level of redundancy for the data; and
a third logic that causes a second, disjoint subset of the set of redundancy blocks to be stored in the object store using an asynchronous protocol, where the second subset in combination with the first subset provides a second, greater level of redundancy for the data.
11. The apparatus of claim 10 , where the third logic selectively buffers members of the second subset as part of the asynchronous protocol and selectively flushes members of the second subset to the object store as part of the asynchronous protocol.
12. The apparatus of claim 10 , the apparatus being part of the object store.
13. The apparatus of claim 10 , the redundancy blocks being erasure codes.
14. The apparatus of claim 10 , comprising a fourth logic that causes the data to be released from the primary data storage upon determining that the set of redundancy blocks has been stored in the object store.
15. The apparatus of claim 10 , where the second logic sends a first message to the primary data storage upon determining that the first subset of the set of redundancy blocks has been stored in the object store using the synchronous protocol.
16. The apparatus of claim 15 , where the third logic sends a second message to the primary data storage upon determining that the second subset of the set of redundancy blocks has been stored in the object store using the asynchronous protocol.
17. The apparatus of claim 10 , comprising a fourth logic that, upon determining that the primary data storage has released the data stored in the primary data storage, replicates to one or more other object stores an object associated with the data, the first subset of the set of redundancy blocks, or the second subset of the set of redundancy blocks, where the replication is performed with object-level granularity.
18. The apparatus of claim 10 , comprising a fourth logic that, upon determining that the primary data storage has released the data stored in the primary data storage, distributes to one or more other object stores an object associated with the data, the first subset of the set of redundancy blocks, or the second subset of the set of redundancy blocks, where the distribution is performed with object-level granularity.
19. A client-server computer system, comprising:
a client apparatus configured to store a first data item; and
a server apparatus configured to provide redundancy for the first data item by storing erasure codes in an object store,
where the client apparatus is configured to make a request to the server apparatus that the server apparatus increase the redundancy for the first data item to a desired level of redundancy;
where the server apparatus is configured to, upon receiving the request:
produce a set of erasure codes that are sufficient to provide the desired level of redundancy;
select a subset of the set of erasure codes that is sufficient to provide a minimal level of redundancy;
use a synchronous approach to cause the object store to store the subset of the set of erasure codes;
provide a first message to the client apparatus concerning achieving the minimal level of redundancy;
use an asynchronous approach to cause the object store to store a second, disjoint subset of the set of erasure codes, where the second, disjoint subset includes the erasure codes in the set of erasure codes that were not stored as part of the subset; and
provide a second message to the client apparatus concerning achieving the desired level of redundancy;
where the client apparatus is configured to, upon receiving the first message, selectively manipulate the first data item in a first way, and
where the client apparatus is configured to, upon receiving the second message, selectively manipulate the first data item in a second different way.
20. The client-server computer system of claim 19 , where selectively manipulating the first data item in a first way comprises allowing access to the data item by a file system and where selectively manipulating the first data item in the second way comprises deleting the first data item from the client system.