Sequential write based durable file system
A durable file system has been designed for storage devices that do not support write in place and/or that are susceptible to errors or failures. The durable file system also facilitates organization and access of large objects (e.g., gigabytes to terabytes in size). Regardless of whether target storage devices are configured with sequential write constraints, the durable file system writes object fragments across a set of sequences or ranges of storage units, such as logical blocks. The durable file system sequentially writes an object fragment into each storage unit sequence along with indexing information for the object fragments. In addition to writing the indexing information for the object fragments into the set of storage unit sequences, the durable file system updates the file system index with the object indexing information.
1 . A method comprising:
generating, by a file system, a plurality of fragments from an object in response to receipt of a request from a client to write the object into a file system instance, the object identified by an object identifier, and each fragment of the plurality of fragments being of equal size to each other;
creating, by the file system, indexing information for the object that identifies the object and a same range of logical block numbers for each of the plurality of fragments;
writing, by the file system, each fragment from the plurality of fragments with the indexing information into a different storage device of a plurality of storage devices that each present a respective zone for storage, wherein each fragment from the plurality of fragments is written into the same range of logical block numbers within each of the different storage devices, and wherein the indexing information in each one of the different storage devices identifies a same last logical block number at the same range of logical block numbers written within each respective zone of each of the different storage devices;
updating, by the file system after the writing, an index log of the file system instance with the indexing information; and
tracking the same last logical block number written within each respective zone for the plurality of fragments.
2 . The method of claim 1 further comprising:
in response to a determination that the index log includes an entry that references a second range of logical block numbers with an older version of the object, removing the entry from the index log.
3 . The method of claim 1 further comprising:
searching the index log for a key prefix that matches the object identifier of the object, wherein keys of the index log comprise a client defined object identifier, a time stamp of when the object arrived, and a fragment identifier.
4 . The method of claim 1 , further comprising:
determining that the index log includes an entry with a key prefix that matches the object identifier, an index key of the entry comprising an object arrival time stamp that is older than a time stamp of when the object arrived.
5 . The method of claim 1 further comprising:
wherein generating the plurality of fragments includes dividing the object equally to generate a first segment corresponding to the same range of logical block numbers and a second segment corresponding to the same range of logical block numbers.
6 . The method of claim 1 , wherein the indexing information includes a size of an individual one of the plurality of fragments.
7 . The method of claim 1 , wherein the writing further comprises:
including, with writing the plurality of fragments, data protection information.
8 . The method of claim 1 , wherein the creating further comprises:
creating information that comprises a client defined identifier of the object, an arrival time stamp for the object, and an identifier of the same range of logical block numbers.
9 . The method of claim 1 further comprising:
reading, in response to a read request that indicates the object identifier, at least a subset of the plurality of fragments in accordance with storage device identifiers corresponding to the same range of logical block numbers.
10 . A non-transitory machine readable medium having stored thereon instructions for managing access and organization of objects, the instructions which when executed by at least one machine, causes the machine to:
select a range of logical block numbers to store an object, wherein the range of logical block numbers is the same on each independently accessible storage device of a plurality of storage devices that each presents a respective zone for storage that the range of logical block numbers is within;
add same indexing information to each of different fragments of the object of substantially equal size to each other to form a plurality of indexed fragments, wherein the same indexing information comprises a client defined object identifier of the object, and the same indexing information in each one of the storage devices identifies a same last logical block number within the range of logical block numbers at each independently accessible storage device;
write each of the plurality of indexed fragments across the range of logical block numbers of each respective independently accessible storage device of the plurality of storage devices;
update a file system index with the same indexing information after writing the indexed fragments; and
tracking the same last logical block number written within each respective zone for the plurality of indexed fragments.
11 . The non-transitory machine readable medium of claim 10 further comprising instructions to:
read, in response to a read request that indicates the client defined object identifier, at least a subset of the plurality of indexed fragments in accordance with storage device identifiers corresponding to the last block written.
12 . A computing device comprising:
a memory containing machine readable medium comprising machine executable code for managing access and organization of objects; and
a processor coupled to the memory, the processor configured to execute the machine executable code to cause the computing device to:
generate a plurality of fragments from an object in response to receipt of a request to write the object, into a file system instance, the object identified by an object identifier, and each fragment of the plurality of fragments being of substantially equal size to each other;
create, by the file system, indexing information for the object that identifies the object and a same range of logical block numbers for each of the plurality of fragments;
write each fragment from the plurality of fragments with the indexing information into a different storage device of a plurality of storage devices that each present a respective zone for storage, wherein each fragment from the plurality of fragments is written into the same range of logical block numbers within each of the different storage devices, and wherein the indexing information in each one of the different storage devices identifies a same last logical block number at the same range of logical block numbers written within each respective zone of each of the different storage devices;
update, after writing the plurality of fragments and the indexing information, an index log of the file system instance with the indexing information; and
tracking the same last logical block number written within each respective zone for the plurality of fragments.
13 . The computing device of claim 12 , further comprising machine executable code to cause the computing device to:
in response to a determination that the index log includes an entry that references a second range of logical block numbers with an older version of the object, remove the entry from the index log.
14 . The computing device of claim 12 , further comprising machine executable code to cause the computing device to:
search the index log for a key prefix that matches the object identifier of the object, wherein keys of the index log comprise a client defined object identifier, a time stamp of when the object arrived, and a fragment identifier.
15 . The computing device of claim 12 , further comprising machine executable code to cause the computing device to:
determine that the index log includes an entry with a key prefix that matches the object identifier, an index key of the entry comprising an object arrival time stamp that is older than a time stamp of when the object arrived.
16 . The computing device of claim 12 , further comprising machine executable code to cause the computing device to:
generate a time stamp that corresponds to when the object arrived at the computing device.
17 . The computing device of claim 12 , further comprising machine executable code to cause the computing device to:
divide the object into a first segment corresponding to the range of logical block numbers and a second segment corresponding to the same range of logical block numbers.
18 . The computing device of claim 12 , wherein the indexing information includes a size of an individual one of the plurality of fragments.
19 . The computing device of claim 12 , further comprising machine executable code to cause the computing device to:
include, with the plurality of fragments, data protection information.