Consistent ring namespaces facilitating data storage and organization in network infrastructures
Computer-implemented methods and systems for managing data objects within a computer network infrastructure that facilitate more efficient and reliable data storage and access are provided. An exemplary method may comprise establishing a physical identifier for each storage resource. A plurality of unique virtual identifiers for each storage resource is generated based on the physical identifier. The plurality of unique virtual identifiers is stored in a consistent ring namespace accessible to every node. An object identifier associated with a location of a data object to be stored in a storage resource is generated. The object identifier is mapped to the consistent ring namespace. The method may also include enabling traversing the consistent ring namespace from any node in the computer network to locate and access the data object.
1. A method for managing data objects within a computer network infrastructure comprising a plurality of nodes, each node comprising tangible storage resources, the method comprising:
establishing a physical identifier for each tangible storage resource, each of the physical identifiers uniquely identifying one of the tangible storage resources;
generating a plurality of unique virtual identifiers for each tangible storage resource, wherein a first unique virtual identifier is a hash value calculated from the physical identifier of the tangible storage resource associated with the physical identifier, a second unique virtual identifier is a hash value calculated from the first unique virtual identifier, and each subsequent unique virtual identifier is a hash value calculated from a preceding unique virtual identifier, wherein the plurality of unique virtual identifiers to be stored in a consistent ring namespace accessible for every node, wherein the plurality of unique virtual identifiers are to be evenly distributed over the consistent ring namespace;
generating an object identifier associated with a location of a data object to be stored in a tangible storage resource;
mapping the object identifier to the consistent ring namespace; and
enabling traversing of the consistent ring namespace from any node in the computer network infrastructure in order to locate and access the data object.
2. The method of claim 1 , wherein the plurality of virtual identifiers is generated to achieve a specific distribution either manually or programmatically.
3. The method of claim 2 , wherein generating the plurality of unique virtual identifiers comprises applying hash chaining to the physical identifier associated with the storage resource to generate an arbitrary number of virtual identifiers associated with an individual tangible storage resource.
4. The method of claim 2 , wherein generating the plurality of unique virtual identifiers comprises applying an offset to the physical identifier to generate a relative distance between the virtual identifiers around the consistent ring namespace.
5. The method of claim 2 , wherein generating the plurality of unique virtual identifiers comprises a relative division among the other virtual identifiers on the consistent ring namespace to achieve a specific proportional ownership of the consistent ring namespace.
6. The method of claim 2 , wherein generating the plurality of unique virtual identifiers is based on applying hash chaining to the network address associated with the tangible storage resource.
7. The method of claim 2 , wherein generating the plurality of unique virtual identifiers is based on applying hash chaining to the combination of the network address and bus location of the tangible storage resource.
8. The method of claim 2 , wherein generating the plurality of unique virtual identifiers is based on a pseudo random distribution that is seeded with a well defined starting value.
9. The method of claim 1 , wherein the plurality of unique virtual identifiers is generated in such a way that an arbitrary number of unique virtual identifiers is mapped over the consistent ring namespace.
10. The method of claim 1 , wherein a number of generated unique virtual identifiers is the same or greater than a number of tangible storage resources in the computer network infrastructure.
11. The method of claim 1 , wherein a number of mapped unique virtual identifiers on the consistent ring namespace related to a single tangible storage resource is associated with a relative capacity of the tangible storage resources in the computer network infrastructure.
12. The method of claim 1 , wherein the number of mapped unique virtual identifiers on the consistent ring namespace is related to a performance of a tangible storage resource in the computer network infrastructure.
13. The method of claim 1 , wherein the number of mapped unique virtual identifiers on the consistent ring namespace is changed over time based on a relative useable capacity of the tangible storage resource for storing data objects.
14. The method of claim 1 , wherein a data object identifier is a hash value generated from the data.
15. The method of claim 1 , wherein the tangible storage resource of a data object replica is written based upon the location of the mapped virtual identifiers in the consistent ring namespace.
16. The method of claim 15 , further comprising generating a plurality of data object replicas and storing them in different tangible storage resources by walking the ring to find different virtual identifiers.
17. The method of claim 16 , wherein the tangible storage resources are selected based upon a preference to use different nodes in the computer network infrastructure for the data object replicas.
18. The method of claim 16 , wherein the tangible storage resources are selected based upon a preference to use different storage resources in the same node for the data object replicas.
19. The method of claim 16 , wherein the tangible storage resources are skipped in the order of the ring walk when a data object replica cannot successfully be stored.
20. The method of claim 15 , further comprising generating a plurality of data object replicas and storing them in different tangible storage resources.
21. The method of claim 20 , wherein a number of data object replicas corresponds to a replication factor based on the number of tangible storage resources or policy requirements of the computer network infrastructure.
22. The method of claim 20 , wherein the number of data object replicas corresponds to a replication factor that can be dynamically adapted during operation based upon configuration or policy of the computer network.
23. The method of claim 1 , wherein enabling traversing of the consistent ring namespace comprises navigating, through a consistent ring namespace, to tangible storage resource identifiers in order to determine a tangible storage resource to retrieve the data object.
24. The method of claim 1 , further comprising re-mapping unique virtual identifiers within the consistent ring namespace when a tangible storage resource is added to the computer network infrastructure.
25. The method of claim 1 , further comprising re-mapping unique virtual identifiers within the consistent ring namespace when available capacity of any tangible storage resource is changed.
26. The method of claim 1 , further comprising removal of unique virtual identifiers within the consistent ring namespace when a tangible storage resource is removed from the computer network infrastructure.
27. The method of claim 1 , further comprising re-distributing data objects to different tangible storage resources after the consistent ring namespace has been changed in view of addition or removal of at least one node or at least one tangible storage resource.
28. The method of claim 27 , wherein re-distribution of data objects from a current tangible storage resource to a new tangible storage resource is based upon the consistent ring namespace that is re-mapped.
29. The method of claim 27 , wherein after a data object has been copied to newly available tangible storage resources, another data object replica can be removed from a tangible storage resource and a replication factor of the data object is still preserved.
30. The method of claim 27 , wherein when a tangible storage resource is removed from the computer network infrastructure, the data objects are copied in ring walk order to subsequent tangible storage resources still available in the computer network infrastructure.
31. The method of claim 27 , wherein re-distribution of data objects can exceed the replication factor to use spare capacity of tangible storage resources in the computer network infrastructure for performance and reconstruction improvements.
32. A system for managing data objects within a computer network infrastructure comprising a plurality of nodes, each node comprising tangible storage resources, the system comprising:
a physical identifier module configured to establish a physical identifier for each tangible storage resource, each of the physical identifiers uniquely identifying one of the tangible storage resources;
a virtual identifier module configured to generate a plurality of unique virtual identifiers for each tangible storage resource, wherein a first unique virtual identifier is a hash value calculated from the physical identifier of the tangible storage resource associated with the physical identifier, a second unique virtual identifier is a hash value calculated from the first unique virtual identifier, and each subsequent unique virtual identifier is a hash value calculated from a preceding unique virtual identifier, wherein the plurality of unique virtual identifiers to be stored in a consistent ring namespace accessible for every node, wherein the plurality of unique virtual identifiers are to be evenly distributed over the consistent ring namespace;
an object identifier module configured to generate an object identifier associated with a location of a data object to be stored in a tangible storage resource;
a mapping module configured to map the object identifier to the consistent ring namespace; and
a traversing module configured to enable traversing the consistent ring namespace from any node in the computer network infrastructure in order to locate and access the data object.
33. The system of claim 32 , wherein the plurality of virtual identifiers is generated to achieve a specific distribution either manually or programmatically.
34. The system of claim 33 , wherein generating the plurality of unique virtual identifiers comprises applying hash chaining to the physical identifier associated with the storage resource to generate an arbitrary number of virtual identifiers associated with an individual tangible storage resource.
35. The system of claim 33 , wherein generating the plurality of unique virtual identifiers comprises applying an offset to the physical identifier to generate a relative distance between the virtual identifiers around the consistent ring namespace.
36. The system of claim 33 , wherein generating the plurality of unique virtual identifiers comprises a relative division among the other virtual identifiers on the consistent ring namespace to achieve a specific proportional ownership of the consistent ring namespace.
37. The system of claim 33 , wherein generating the plurality of unique virtual identifiers is based on applying hash chaining to a network address associated with the tangible storage resource.
38. The system of claim 33 , wherein generating the plurality of unique virtual identifiers is based on applying hash chaining to the combination of the network address and bus location of the tangible storage resource.
39. The system of claim 33 , wherein generating the plurality of unique virtual identifiers is based on a pseudo random distribution that is seeded with a well defined starting value.
40. The system of claim 32 , wherein the plurality of unique virtual identifiers is generated in such a way that an arbitrary number of unique virtual identifiers is mapped over the consistent ring namespace.
41. The system of claim 32 , wherein a number of generated unique virtual identifiers is the same or greater than a number of tangible storage resources in the computer network infrastructure.
42. The system of claim 32 , wherein a number of mapped unique virtual identifiers on the consistent ring namespace related to a single tangible storage resource is associated with a relative capacity of the tangible storage resources in the computer network infrastructure.
43. The system of claim 32 , wherein the number of mapped unique virtual identifiers on the consistent ring namespace is related to a performance of a tangible storage resource in the computer network infrastructure.
44. The system of claim 32 , wherein the number of mapped unique virtual identifiers on the consistent ring namespace is changed over time based on a relative useable capacity of a tangible storage resource for storing data objects.
45. The system of claim 32 , wherein a data object identifier is a hash value generated from the data.
46. The system of claim 32 , wherein the tangible storage resource of a data object replica is written based upon the location of the mapped virtual identifiers in the consistent ring namespace.
47. The system of claim 46 , further comprising a replica generator configured to generate a plurality of data object replicas and store them in different tangible storage resources by walking the ring to find different virtual identifiers.
48. The system of claim 47 , wherein the tangible storage resources are selected based upon a preference to use different nodes in the computer network infrastructure for the data object replicas.
49. The system of claim 47 , wherein the tangible storage resources are selected based upon a preference to use different tangible storage resources in the same node for the data object replicas.
50. The system of claim 47 , wherein the tangible storage resources are skipped in the order of the ring walk when a data object replica cannot successfully be stored.
51. The system of claim 47 , wherein a number of data object replicas corresponds to a replication factor based on the number of tangible storage resources or policy requirements of the computer network infrastructure.
52. The system of claim 47 , wherein the number of data object replicas corresponds to a replication factor that can be dynamically adapted during operation based upon configuration or a policy of the computer network.
53. The system of claim 32 , wherein enabling traversing of the consistent ring namespace comprises navigating, through a consistent ring namespace, to tangible storage resource identifiers in order to determine a tangible storage resource to retrieve the data object.
54. The system of claim 32 , further comprising a re-mapping module configured to re-map the unique virtual identifiers within the consistent ring namespace when the tangible storage resource is added to the computer network infrastructure.
55. The system of claim 54 , wherein the re-mapping module is configured to re-distribute the unique virtual identifiers within the consistent ring namespace when available capacity of any tangible storage resource is changed.
56. The system of claim 54 , wherein the re-mapping module is configured to remove the unique virtual identifiers within the consistent ring namespace when the tangible storage resource is removed from the computer network infrastructure.
57. The system of claim 56 , wherein the re-mapping module is configured to re-distribute data objects to different tangible storage resources after the consistent ring namespace has been changed in view of addition or removal of at least one node or at least one tangible storage resource.
58. The system of claim 57 , wherein re-distribution of data objects from a current tangible storage resource to a new tangible storage resource is based upon the consistent ring namespace that is re-mapped.
59. The system of claim 57 , wherein after a data object has been copied to newly available tangible storage resources, another data object replica can be removed from a tangible storage resource and a replication factor of the data object is still preserved.
60. The system of claim 57 , wherein when a tangible storage resource is removed from the computer network infrastructure, the data objects are copied in ring walk order to subsequent tangible storage resources still available in the computer network infrastructure.
61. The system of claim 57 , wherein re-distribution of data objects can exceed the replication factor to use spare capacity of tangible storage resources in the computer network infrastructure for performance and reconstruction improvements.
62. A non-transitory processor-readable medium having instructions stored thereon, which when executed by one or more processors, causes the one or more processors to implement a method for managing data objects within a computer network infrastructure comprising a plurality of nodes, each node comprising tangible storage resources, the method comprising:
establishing a physical identifier for each tangible storage resource, each of the physical identifiers uniquely identifying one of the tangible storage resources;
generating a plurality of unique virtual identifiers for each tangible storage resource, wherein a first unique virtual identifier is a hash value calculated from the physical identifier of the tangible storage resource associated with the physical identifier, a second unique virtual identifier is a hash value calculated from the first unique virtual identifier, and each subsequent unique virtual identifier is a hash value calculated from a preceding unique virtual identifier, wherein the plurality of unique virtual identifiers to be stored in a consistent ring namespace accessible for every node, wherein the plurality of unique virtual identifiers are to be evenly distributed over the consistent ring namespace;
generating an object identifier associated with a location of a data object to be stored in a tangible storage resource;
mapping the object identifier to the consistent ring namespace; and
enabling traversing of the consistent ring namespace from any node in the computer network infrastructure in order to locate and access the data object.