Dynamic redundancy in storage systems
A method and system for dynamic redundancy in storage systems is described. The method may include receiving a data fragment from a data stream of user data to be archived. The method may further include splitting the data fragment into a first number of data chunks. The method may also include, in response to determining that the data fragment is not a last data fragment in the data stream, generating a second number of additional data chunks based upon, at least in part, the first number of data chunks. The method may additionally include, in response to determining that the data fragment is the last data fragment in the data stream, generating a third number of additional data chunks based upon, at least in part, the first number of data chunks.
1. A method for dynamic redundancy in storage systems, the method comprising:
receiving a data fragment from a data stream of user data to be archived;
splitting the data fragment into a first number of data chunks;
in response to determining that the data fragment is not a last data fragment in the data stream, generating a second number of redundant data chunks based upon, at least in part, the first number of data chunks; and
in response to determining that the data fragment is the last data fragment in the data stream, generating a third number of redundant data chunks based upon, at least in part, the first number of data chunks, wherein the third number of redundant data chunks is greater than the second number of redundant data chunks.
2. The method of claim 1 , further comprising:
transmitting the first number of data chunks and the third number of redundant data chunks to a fourth number of storage devices, wherein the fourth number of storage devices is equal to the first number plus the third number.
3. The method of claim 2 , further comprising:
receiving confirmation from a fifth number of storage devices that the transmitted data chunks were stored, wherein the fifth number is less than the fourth number.
4. The method of claim 3 , wherein the fifth number of storage devices is sufficient to store the data fragment with a required level of redundancy.
5. The method of claim 3 , further comprising:
in response to receiving confirmation from the fifth number of storage devices that the transmitted data chunks were stored, confirming that the data fragment was stored with a required level of redundancy without receiving confirmation from each of the fourth number of storage devices that the transmitted data chunks were stored.
6. The method of claim 3 wherein the storage devices from the fourth number of storage devices that are not from the fifth number of storage devices are the slowest storage devices to which data chunks were transmitted.
7. The method of claim 3 , further comprising:
receiving a second data fragment from the data stream of user data to be archived; and in response to determining that the data fragment that was determined to be the last data fragment in the data stream is no longer the last data fragment in the data stream, deleting data chunks from the third number of redundant data chunks which are not from the second number of redundant data chunks.
8. The method of claim 1 wherein a front end server receives the data fragment and further comprising using erasure codes to generate one or both of the second number of redundant data chunks and the third number of redundant data chunks.
9. The method of claim 8 further comprising storing one or more of the first number of data chunks, the second number of redundant data chunks and the third number of redundant data chunks in an append-only electronic memory storage.
10. A system for dynamic redundancy in storage systems, the system comprising:
a data fragment from a data stream of user data to be archived;
a first number of data chunks split from the data fragment;
a second number of redundant data chunks generated based upon, at least in part, the first number of data chunks and associated with a data fragment that is not last in the data stream; and
a third number of redundant data chunks generated based upon, at least in part, the first number of data chunks and associated with a data fragment that is last in the data stream, wherein the third number of redundant data chunks is greater than the second number of redundant data chunks.
11. The system of claim 10 , wherein the system transmits the first number of data chunks and the third number of redundant data chunks to a fourth number of storage devices, wherein the fourth number of storage devices is equal to the first number plus the third number.
12. The system of claim 11 , wherein the system receives confirmation from a fifth number of storage devices that the transmitted data chunks were stored, wherein the fifth number is less than the fourth number.
13. The system of claim 12 , wherein the fifth number of storage devices is sufficient to store the data fragment with a required level of redundancy.
14. The system of claim 12 , wherein, in response to receiving confirmation from the fifth number of storage devices that the transmitted data chinks chunks were stored, the system confirms that the data fragment was stored with a required level of redundancy without receiving confirmation from each of the fourth number of storage devices that the transmitted data chunks were stored.
15. The system of claim 12 , wherein the storage devices from the fourth number of storage devices that are not from the fifth number of storage devices are the slowest storage devices to which data chunks were transmitted.
16. The system of claim 12 , further comprising:
a second data fragment from the data stream of user data to be archived;
wherein, in response to determining that the data fragment that was determined to be the last data fragment in the data stream is no longer the last data fragment in the data stream, the system deletes data chunks from the third number of redundant data chunks which are not from the second number of redundant data chunks.
17. The system of claim 10 further comprising:
an archive managing server executing one or more server-side dynamic redundancy applications and
one or more electronic memory storages in communication with the archive managing server,
wherein one or more of the first number of data chunks, the second number of redundant data chunks and the third number of redundant data chunks are stored in the one or more electronic memory storages.
18. The system of claim 17 wherein one or more of the one or more electronic memory storages are an append-only electronic memory storage.
19. The system of claim 18 further comprising erasure codes to generate one or both of the second number of redundant data chunks and the third number of redundant data chunks.
20. The system of claim 19 wherein one or more of the append-only memory storages are connected to the archive managing server.