Erasure coding and replication in storage clusters
A cluster receives a request to store an object using replication or erasure coding. The cluster writes the object using erasure coding. A manifest is written that includes an indication of erasure coding and a unique identifier for each segment. The cluster returns a unique identifier of the manifest. The cluster receives a request from a client that includes a unique identifier. The cluster determines whether the object has been stored using replication or erasure coding. If using erasure coding, the method reads a manifest. The method identifies segments within the cluster using unique segment identifiers of the manifest. Using these unique segment identifiers, the method reconstructs the object. A persistent storage area of another disk is scanned to find a unique identifier of a failed disk. If using erasure coding, a missing segment previously stored on the disk is identified. The method locates other segments. Missing segments are regenerated.
1. A method of converting a digital object within a storage cluster, said method comprising:
storing said digital object within said storage cluster in a first erasure coding storage format, said digital object having a unique identifier within said storage cluster;
after said storing, identifying a requirement to convert said digital object to a second erasure coding storage format;
reading said digital object from said storage cluster using a computer node of said storage cluster;
determining a segment size for segments of said second erasure coding storage format;
writing said digital object to said storage cluster using said second erasure coding storage format having segments of said segment size; and
retaining said unique identifier for said digital object written in said second erasure coding storage format, whereby a client application retrieves said digital object written in said second erasure coding storage format using said unique identifier.
2. A method as recited in claim 1 wherein a first erasure code of said first erasure coding storage format is different from a second erasure code of said second erasure coding storage format.
3. A non-transitory computer-readable medium comprising computer code for converting a digital object within a storage cluster, said computer code of said computer-readable medium executable by a machine to perform the following:
storing said digital object within said storage cluster in a first erasure coding storage format, said digital object having a unique identifier within said storage cluster;
after said storing, identifying a requirement to convert said digital object to a second erasure coding storage format;
reading said digital object from said storage cluster using a computer node of said storage cluster;
determining a segment size for segments of said second erasure coding storage format;
writing said digital object to said storage cluster using said second erasure coding storage format having segments of said segment size; and
retaining said unique identifier for said digital object written in said second erasure coding storage format, whereby a client application retrieves said digital object written in said second erasure coding storage format using said unique identifier.
4. A computer-readable medium as recited in claim 3 wherein a first erasure code of said first erasure coding storage format is different from a second erasure code of said second erasure coding storage format.
5. A method of storing a digital object in a storage cluster, said method comprising:
receiving a request at a computer node of said storage cluster to store said digital object;
determining whether to store said digital object in said storage cluster using replication or erasure coding;
when it is determined to store said digital object using erasure coding, writing said digital object to a plurality of computer nodes of said storage cluster using erasure coding, said digital object being written in parallel to said computer nodes as a plurality of segments;
creating a manifest computer file that includes a unique identifier within said storage cluster for each of said segments; and
storing said manifest computer file on a computer node of said storage cluster.
6. A non-transitory computer-readable medium comprising computer code for storing a digital object in a storage cluster, said computer code of said computer-readable medium executable by a machine to perform the following:
receiving a request at a computer node of said storage cluster to store said digital object;
determining whether to store said digital object in said storage cluster using replication or erasure coding;
when it is determined to store said digital object using erasure coding, writing said digital object to a plurality of computer nodes of said storage cluster using erasure coding, said digital object being written in parallel to said computer nodes as a plurality of segments;
creating a manifest computer file that includes a unique identifier within said storage cluster for each of said segments; and
storing said manifest computer file on a computer node of said storage cluster.
7. A method of storing a digital object in a storage cluster, said method comprising:
receiving a request from a client application at a first computer node of said storage cluster to store said digital object;
determining whether to store said digital object in said storage cluster using replication or erasure coding using a setting of said storage cluster;
when it is determined to store said digital object using replication, receiving said digital object at a second computer node of said storage cluster;
writing said digital object to said second computer node of said storage cluster;
calculating a unique identifier for said digital object; and
returning said unique identifier identifying said digital object to said client application.
8. A non-transitory computer-readable medium comprising computer code for storing a digital object in a storage cluster, said computer code of said computer-readable medium executable by a machine to perform the following:
receiving a request from a client application at a first computer node of said storage cluster to store said digital object;
determining whether to store said digital object in said storage cluster using replication or erasure coding using a setting of said storage cluster;
when it is determined to store said digital object using replication, receiving said digital object at a second computer node of said storage cluster;
writing said digital object to said second computer node of said storage cluster;
calculating a unique identifier for said digital object; and
returning said unique identifier identifying said digital object to said client application.
9. A method of converting a digital object within a storage cluster, said method comprising:
storing said digital object within said storage cluster in a first erasure coding storage format, said digital object having a unique identifier within said storage cluster;
after said storing, determining to convert said digital object to a second erasure coding storage format using a setting of said storage cluster, said second erasure coding format using fewer segments than said first erasure coding storage format;
reading said digital object from said storage cluster using a computer node of said storage cluster;
writing said digital object to said storage cluster using said second erasure coding storage format; and
retaining said unique identifier for said digital object written in said second erasure coding storage format, whereby a client application may retrieve said digital object written in said second erasure coding storage format using said unique identifier.
10. A non-transitory computer-readable medium comprising computer code for converting a digital object within a storage cluster, said computer code of said computer-readable medium executable by a machine to perform the following:
storing said digital object within said storage cluster in a first erasure coding storage format, said digital object having a unique identifier within said storage cluster;
after said storing, determining to convert said digital object to a second erasure coding storage format using a setting of said storage cluster, said second erasure coding format using fewer segments than said first erasure coding storage format;
reading said digital object from said storage cluster using a computer node of said storage cluster;
writing said digital object to said storage cluster using said second erasure coding storage format; and
retaining said unique identifier for said digital object written in said second erasure coding storage format, whereby a client application may retrieve said digital object written in said second erasure coding storage format using said unique identifier.
11. A method of converting a digital object within a storage cluster, said method comprising:
storing said digital object within said storage cluster in an erasure coding storage format, said digital object having a unique identifier within said storage cluster;
after said storing, determining to convert said digital object to a storage format using replication using a setting of said storage cluster;
reading said digital object from said storage cluster using a computer node of said storage cluster;
writing said digital object to a single disk of a computer node of said storage cluster as a stream of bits; and
retaining said unique identifier for said digital object written as said stream of bits, whereby a client application may retrieve said digital object written as said stream of bits using said unique identifier.
12. A non-transitory computer-readable medium comprising computer code for converting a digital object within a storage cluster, said computer code of said computer-readable medium executable by a machine to perform the following:
storing said digital object within said storage cluster in an erasure coding storage format, said digital object having a unique identifier within said storage cluster;
after said storing, determining to convert said digital object to a storage format using replication using a setting of said storage cluster;
reading said digital object from said storage cluster using a computer node of said storage cluster;
writing said digital object to a single disk of a computer node of said storage cluster as a stream of bits; and
retaining said unique identifier for said digital object written as said stream of bits, whereby a client application may retrieve said digital object written as said stream of bits using said unique identifier.
13. A method as recited in claim 1 wherein said unique identifier identifies a manifest computer file within said storage cluster that includes a segment unique identifier for each of said segments of said second erasure coding storage format.
14. A non-transitory computer-readable medium as recited in claim 3 wherein said unique identifier identifies a manifest computer file within said storage cluster that includes a unique identifier for each of said segments of said second erasure coding storage format.
15. A method as recited in claim 5 wherein said request is received from a client application, said method further comprising:
calculating a unique identifier for said manifest computer file that uniquely identifies said manifest computer file within said storage cluster; and
returning said unique identifier identifying said manifest computer file to said client application.
16. A non-transitory computer-readable medium as recited in claim 6 wherein said request is received from a client application, said computer code also performing the following:
calculating a unique identifier for said manifest computer file that uniquely identifies said manifest computer file within said storage cluster; and
returning said unique identifier identifying said manifest computer file to said client application.
17. A method as recited in claim 9 wherein said unique identifier identifies a manifest computer file within said storage cluster that includes a unique identifier for each segment of said second erasure coding storage format.
18. A non-transitory computer-readable medium as recited in claim 10 wherein said unique identifier identifies a manifest computer file within said storage cluster that includes a unique identifier for each segment of said second erasure coding storage format.
19. A method as recited in claim 11 wherein said unique identifier of said digital object stored within said storage cluster in an erasure coding storage format identifies a manifest computer file within said storage cluster that includes a unique identifier for each segment of said erasure coding storage format.
20. A non-transitory computer-readable medium as recited in claim 12 wherein said unique identifier of said digital object stored within said storage cluster in an erasure coding storage format identifies a manifest computer file within said storage cluster that includes a unique identifier for each segment of said erasure coding storage format.