Combined asynchronous and synchronous fountain code storage in an object store
View Patent ↗Example apparatus and methods produce a set of rateless erasure codes (e.g., fountain codes) for a file stored in a primary data store (e.g., hard drive) or in an archive system. The archive system may store the file in a redundant array of independent disks (RAID). A first subset of the rateless erasure codes are stored in an object storage using a synchronous protocol. A second subset of rateless erasure codes are stored in the object storage using an asynchronous protocol. The object storage system may inform the archive system when desired redundancy has been achieved or when desired redundancy has been lost. The archive system may buffer rateless erasure codes before providing the codes to the object storage to improve performance. A failure in the archive system or object storage system may be mitigated by retaining the file in the primary data store until the desired redundancy is achieved.
1. A non-transitory computer-readable storage medium storing computer-executable instructions that when executed by a computer cause the computer to perform a method, the method comprising:
receiving a request to increase the redundancy level of a file stored in a first data store, where the first data store is a disk drive associated with a file system or memory associated with a file system;
producing a number of rateless erasure codes to be provided from a second data store to a third data store to increase the redundancy level of the file, where the second data store is an archive system and the third data store is an object storage system, where the number of rateless erasure codes is controlled by a redundancy policy, where the redundancy policy identifies a total number of rateless erasure codes to be stored in the first data store, in the second data store, or in the third data store, and a minimum number of rateless erasure codes to be stored in the first data store, in the second data store, or in the third data store, where the minimum number is less than the total number;
providing the minimum number of rateless erasure codes to the third data store using a synchronous approach;
storing, in the third data store, the minimum number of rateless erasure codes;
determining a remaining number of rateless erasure codes equal to the difference between the total number and the minimum number;
upon determining that the minimum number of rateless erasure codes have has been stored in the third data store:
providing the remaining number of rateless erasure codes to the third data store using an asynchronous approach, where providing the remaining number of rateless erasure codes to the third data store using the asynchronous approach includes buffering one or more rateless erasure codes in the second data store before providing the one or more rateless erasure codes to the third data store;
storing, in the third data store, the remaining number of rateless erasure codes; and
upon determining that the total number of rateless erasure codes has been stored by the third data store, sending a signal to the first data store indicating that the total number of rateless erasure codes has been stored in the third data store.
2. The non-transitory computer-readable storage medium of claim 1 , where the minimum number of rateless erasure codes is zero.
3. The non-transitory computer-readable storage medium of claim 1 , the method comprising:
upon determining that the total number of rateless erasure codes has been stored in the third store, controlling the first data store to release the file.
4. The non-transitory computer-readable storage medium of claim 1 , the method comprising producing the rateless erasure codes according to a user configurable code rate.
5. The non-transitory computer-readable storage medium of claim 1 , where providing the remaining number of rateless erasure codes to the third data store using the asynchronous approach includes forcing a flush of a rateless erasure code from the second data store to the third data store.
6. The non-transitory computer-readable storage medium of claim 1 , the method comprising receiving a notification from the third data store that the minimum number of rateless erasure codes has been stored in the third data store.
7. The non-transitory computer-readable storage medium of claim 6 , the method comprising receiving a notification from the third data store that the total number of rateless erasure codes has been stored in the third data store.
8. The non-transitory computer-readable storage medium of claim 1 , the method comprising receiving a notification from the third data store that a rateless erasure code has become unavailable and,
upon determining that a desired redundancy level for the file is not present, selectively re-performing a store operation using data present in the second data store, where the store operation comprises:
producing one or more rateless erasure codes;
selectively providing one or more rateless erasure codes to the third data store using a synchronous approach; or
selectively providing one or more rateless erasure codes to the third data store using an asynchronous approach.
9. The non-transitory computer-readable storage medium of claim 1 , the method comprising receiving a notification from the third data store that a rateless erasure code has become unavailable and,
upon determining that a desired redundancy level for the file is not present, selectively re-performing a store operation using data present in the first data store, where the store operation comprises:
producing one or more rateless erasure codes;
selectively providing one or more rateless erasure codes to the third data store using a synchronous approach; or
selectively providing one or more rateless erasure codes to the third data store using an asynchronous approach.
10. An apparatus, comprising:
a processor;
a memory;
a set of logics; and
an interface that connects the processor, the memory, and the set of logics;
the set of logics comprising:
a first logic that produces a set of fountain codes for a file stored in a primary data storage, where the set of fountain codes are to be stored in an object store;
a second logic that causes a first subset of the set of fountain codes to be provided to the object store using a synchronous protocol, and that stores the first subset in the object store; and
a third logic that causes a second, disjoint subset of the set of fountain codes to be provided to the object store using an asynchronous protocol, and that stores the second, disjoint subset in the object store.
11. The apparatus of claim 10 , where the third logic selectively buffers members of the second subset as part of the asynchronous protocol.
12. The apparatus of claim 11 , where the third logic selectively flushes members of the second subset to the object store as part of the asynchronous protocol.
13. The apparatus of claim 12 , where the first logic, the second logic, or the third logic interact, substantially in parallel, with a plurality of objects associated with two or more files.
14. The apparatus of claim 10 , the apparatus being an archive system positioned in a communication path between the primary data storage and the object store.
15. The apparatus of claim 10 , comprising a fourth logic that causes the file to be released from the primary data storage upon determining that the set of fountain codes has been stored in the object store.
16. The apparatus of claim 15 , where the fourth logic controls the first logic, the second logic and the third logic to store in the object store additional fountain codes for the file upon determining that a threshold number of fountain codes for the file are unavailable in the object store.