Methods and systems for storing data sequentially on a media
Systems and methods for storing data are provided herein using a data archive architecture controlled by a head server. The head server can write data in a contiguous manner across multiple storage devices by partitioning objects into multiple portions and spreading the portions across the multiple storage devices. Portions of a first object can be transmitted to a first subset of storage devices and portions of a second object can be transmitted to a second subset of storage devices. The first subset of storage devices can write the portions of the first object to a first write location indicated by a write pointer. The head server can increment the write pointer based on a length of the portions of the first object to a second write location. The second subset of storage devices can write the portions of the second object to the second write location.
1. A method to store data sequentially on a media comprising:
receiving, by a head server in communication with a plurality of storage devices, a first object and a second object;
dividing, by the head server, the first object into a first plurality of portions and the second object into a second plurality of portions, wherein each of the first plurality of portions have a first length and each of the second plurality of portions have a second length;
transmitting, by the head server, each of the first plurality of portions to a respective one of the first subset of the plurality of storage devices;
writing, by each of a first subset of the plurality of storage devices, a respective one of the first plurality of portions to a first write location indicated by a write pointer, wherein the first write location is the same for each of the first subset of the plurality of storage devices;
incrementing, by the head server, the write pointer to a second write location based on the first length;
transmitting, by the head server, each of the second plurality of portions to a respective one of the second subset of the plurality of storage devices; and
writing, by each of a second subset of the plurality of storage devices, a respective one of the second plurality of portions to the second write location, wherein the second write location is the same for each of the second subset of the plurality of storage devices.
2. The method of claim 1 , further comprising storing the first write location corresponding to the first plurality of portions and the second write location corresponding to the second plurality of portions to a storage media independent from the plurality storage devices.
3. The method of claim 1 , further comprising writing each of the first plurality of portions and each of the second plurality of portions with disk media using shingled magnetic recording (SMR) disk drives.
4. The method of claim 1 , further comprising:
randomly assigning the first plurality of portions to the first subset of the plurality of storage devices using a first hash calculation; and
randomly assigning the second plurality of portions to the second subset of the plurality of storage devices using a second hash calculation.
5. The method of claim 1 , further comprising writing an indication of the first length to a header of the first plurality of portions and writing an indication of the second length to a header of the second plurality of portions.
6. The method of claim 1 , further comprising:
duplicating one of the first plurality of portions to one of the plurality of storage devices not in the first subset of the plurality of storage devices.
7. The method of claim 1 , further comprising:
generating an encryption key for the first object;
encrypting the first object with the encryption key; and
recording the encryption key with the first write location.
8. The method of claim 7 , further comprising encrypting the encryption key with an account-specific encryption key prior to recording the encryption key with the first write location.
9. The method of claim 1 , further comprising:
writing, by one of the plurality of storage devices, one of the plurality of the first portions to a first block stored on a temporary storage medium;
writing, by the one of the plurality of storage devices, one of the plurality of the second portions to the first block stored on the temporary storage medium; and
writing, by the one of the plurality of storage devices, the first block to a long-term storage medium.
10. The method of claim 1 , wherein the second subset of the plurality of storage devices is different than the first subset of the plurality of storage devices.
11. A system to store data sequentially on a media, comprising:
a head server in communication with a plurality of storage devices, the head server configured to:
receive a first object and a second object;
divide the first object into a first plurality of portions and the second object into a second plurality of portions, wherein each of the first plurality of portions have a first length and each of the second plurality of portions have a second length;
transmit each of the first plurality of portions to a respective one of a first subset of the plurality of storage devices;
increment a write pointer from a first write location to a second write location based on the first length; and
transmit each of the second plurality of portions to a respective one of a second subset of the plurality of storage devices; and
the first subset of the plurality of storage devices configured to:
write a respective one of the first plurality of portions to the first write location indicated by the write pointer, wherein the first write location is the same for each of the first subset of the plurality of storage devices; and
the second subset of the plurality of storage devices configured to:
write a respective one of the second plurality of portions to the second write location, wherein the second write location is the same for each of the second subset of the plurality of storage devices.
12. The system of claim 11 , further comprising a storage media independent from the plurality of storage devices to store the first write location corresponding to the first plurality of portions and the second write location corresponding to the second plurality of portions.
13. The system of claim 11 , wherein the first subset of the plurality of storage devices configured to write each of the first plurality of portions and the second subset of the plurality of storage devices configured to write each of the second plurality of portions with disk media using shingled magnetic recording (SMR) disk drives.
14. The system of claim 11 , wherein the head server is configured to:
randomly assign the first plurality of portions to the first subset of the plurality of storage devices using a first hash calculation; and
randomly assign the second plurality of portions to the second subset of the plurality of storage devices using a second hash calculation.
15. The system of claim 11 , wherein the first subset of the plurality of storage devices configured to write an indication of the first length to a header of the first plurality of portions; and
the second subset of the plurality of storage devices configured to write an indication of the second length to a header of the second plurality of portions.
16. The system of claim 11 , wherein the first subset of the plurality of storage devices configured to duplicate one of the first plurality of portions to one of the plurality of storage devices not in the first subset of the plurality of storage devices.
17. The system of claim 11 , wherein the head server is configured to:
generate an encryption key for the first object;
encrypt the first object with the encryption key; and
record the encryption key with the first write address.
18. The system of claim 17 , wherein the head server is configured to encrypt the encryption key with an account-specific encryption key prior to recording the encryption key with the first write address.
19. The system of claim 11 , wherein at least one of the plurality of storage devices is configured to:
write one of the plurality of the first portions to a first block stored on a temporary storage medium;
write one of the plurality of the second portions to the first block stored on the temporary storage medium; and
write the first block to a long-term storage medium.
20. The system of claim 11 , wherein the second subset of the plurality of storage devices is different than the first subset of the plurality of storage devices.