Datacenter maintenance
Systems and methods for maintaining a datacenter are provided. A repair component includes modules that are configured to scan the data objects based on the identifiers of the data objects, identify potential issues or concerns with the data objects, and then repair the issues as they are discovered or over time. The repair component can be run in a non-centralized and highly parallelized manner.
1. A computer-implemented method for performing a decentralized maintenance operation in a datacenter, the method comprising:
instantiating multiple instances of a repair component on multiple nodes in the datacenter, wherein each instance is associated with a portion of data objects stored in the data center, wherein the datacenter stores manifests that include first identifiers associated with the data objects and file lists include second identifiers that are associated with the data objects
wherein each instance is configured to:
determine whether data objects believed to be stored in the data center are actually stored in the datacenter based on an evaluation of a manifest and a file list;
evaluating a portion of the first identifiers and a corresponding portion of the second identifiers to determine actions to perform on the data objects, the manifests, or the file lists based on the evaluation of the portion of the first identifiers from manifests and the portion of the second identifiers from the file lists, wherein the evaluation identifies deficiencies in the data objects represented by the manifests and the file lists and wherein the actions remedy the identified deficiencies; and
perform the actions on the data objects, the manifest, or the file list.
2. The computer-implemented method of claim 1 , wherein the datacenter includes a plurality of storage devices, the method further comprising identifying the manifests and the file lists, wherein the manifests identify the data objects believed to be stored in the datacenter and the file lists identify the data objects actually stored in the datacenter.
3. The computer-implemented method of claim 2 , wherein identifying the manifests and file lists includes scanning the datacenter.
4. The computer-implemented method of claim 3 , wherein scanning the datacenter includes requesting the manifests from metadata nodes, requesting the file lists from data nodes, and storing at least a part of the manifests and at least a part of the file lists in a memory, wherein each instance scans a different portion of the datacenter.
5. The computer-implemented method of claim 1 , wherein each instance tracks specific manifests associated with the portion of the first identifiers and the portion of the second identifiers, wherein the portion of the first identifiers corresponds to a range of the first identifiers and the portion of the second identifiers corresponds to a range of the second identifiers.
6. The computer-implemented method of claim 1 , further comprising, by each corresponding instance for a particular range of data objects:
identifying common identifiers that are present in both the manifests and the file lists;
identifying file identifiers that are only present in the file lists; and
identifying file identifiers that are only present in the manifests.
7. The computer-implemented method of claim 6 , further comprising deleting the data objects associated with the file indentifiers only present in the file lists or only present in the manifests.
8. The computer-implemented method of claim 6 , further comprising determining whether the file lists include a full set of object fragments for each of the common identifiers.
9. The computer-implemented method of claim 8 , wherein data objects that have less than a full set of object fragments are scheduled for repair as long as a number or remaining object fragments is greater than or equal to a threshold number of object fragments.
10. The computer-implemented method of claim 8 , wherein when the number of remaining object fragments is below the threshold, the data object is deleted by removing an entry corresponding to the data object from the manifests.
11. The computer-implemented method of claim 8 , wherein data objects associated with more than a full set of object fragments are scheduled for deduplication.
12. The computer-implemented method of claim 1 , wherein the actions identified by the evaluation are entered into a queue and wherein the actions include one or more of a repair operation, a scrub operation, a deduplication operations, or a delete operation.
13. The computer-implemented method of claim 12 , further comprising prioritizing the queue and providing age protections to entries in the queue such that actions are not inadvertently performed on the data objects.
14. A computer-implemented method for performing a maintenance operation in a datacenter, the method comprising:
instantiating multiple instances of a repair component on multiple nodes in the datacenter, wherein each instance is associated with a portion of data objects stored in the data center, wherein the multiple instances cooperate to:
scan a database stored in storage devices of the datacenter to retrieve a range of first identifiers stored in manifests and a corresponding range of second identifiers from file lists stored in the datacenter, wherein the first identifiers identify the data objects believed to be stored in the datacenter and the file lists identify the data objects actually stored in the datacenter;
determine, based on an evaluation of the first identifiers and the second identifiers, whether data objects believed to be stored in the data center are actually stored in the datacenter;
determine actions to perform on the data objects based on whether the data objects are actually stored in the datacenter or are only believed to be stored in the data center;
placing an entry in a queue for each identifier that requires one of the actions; and
performing the actions in the queue.
15. The computer implemented method of claim 14 , further comprising scheduling the actions, wherein each of the actions is at least one a repair operation, a scrub operation, a deduplication operation, or a delete operation, wherein the actions remedy the deficiencies identified by the evaluation.
16. The computer implemented method of claim 15 ,
wherein the action is the repair operation when a particular identifier of a particular data object is included in both the first identifiers and the second identifiers and a number of object fragments for the particular data object is less than a full set of object fragments and greater than or equal to a threshold number of object fragments, wherein the repair operation reconstructs the data object from the number of object fragments available for the data object;
wherein the action is the scrub operation when the number of object fragments is less than the threshold number of object fragments, wherein the scrub operation removes the identifier from the manifests,
wherein the action is the deduplication operation when the number of object fragments is greater than the full set of object fragments, wherein the deduplication operation deletes duplicate object fragments, and
wherein the action is the delete operation for each second identifier that is not included in the first identifiers.
17. The computer implemented method of claim 14 , wherein the multiple instances operate in parallel.
18. The computer implemented method of claim 17 , wherein the multiple instances operate on different portions of the manifests and different portions of the file lists.
19. The computer implemented method of claim 18 , wherein each of the multiple instances is associated with a specific range of identifiers.