Distributed storage system with web services client interface
A distributed, web-services based storage system. A system may include a web services interface configured to receive, according to a web services protocol, a given client request for access to a given data object, the request including a key value corresponding to the object. The system may also include storage nodes configured to store replicas of the objects, where each replica is accessible via a respective unique locator value, and a keymap instance configured to store a respective keymap entry for each object. For the given object, the respective keymap entry includes the key value and each locator value corresponding to replicas of the object. A coordinator may receive the given client request from the web services interface, responsively access the keymap instance to identify locator values corresponding to the key value and, for a particular locator value, retrieve a corresponding replica from a corresponding storage node.
1. A system for providing arbitrary amounts of reliable storage accessible from any location, the system comprising:
at least one processor; and
at least one memory, wherein the at least one memory stores program instructions, and wherein executing the program instructions causes the system to:
receive web services calls from at least one client;
cause a data storage web service to store a data object in response to a web services call, wherein the web services call references a namespace corresponding to a user of the data storage web service and a key corresponding to the data object, and wherein the stored data object is addressable using the namespace and the key;
cause the data storage web service to store one or more replicas of the data object in one or more of a plurality of distributed storage nodes; and
cause an operation metrics service to store information characteristic of an actual usage of the data storage web service, and wherein the actual usage is based, at least in part, upon the web services call.
2. The system of claim 1 , wherein the web services calls comprise Representational State Transfer (REST)-style requests.
3. The system of claim 1 , wherein the web services calls are implemented according to an application programing interface (API).
4. The system of claim 1 , wherein the data object is stored in a bucket.
5. The system of claim 1 , wherein executing the program instructions causes the system to cause the operation metrics service to store information characteristic of a size of the data object or a storage utilization of the data storage web service.
6. The system of claim 1 , wherein executing the program instructions causes the system to cause the data storage web service to store the data object according to an encoding scheme.
7. The system of claim 6 , wherein the encoding scheme is a parity encoding scheme.
8. The system of claim 1 , wherein a keymap stores a relationship between the key and locators corresponding to locations of the data object and the one or more replicas of the data object.
9. A computer program product embodied on a non-transitory computer readable storage medium comprising:
computer code for receiving web services calls from at least one client;
computer code for causing a data storage web service to store a data object in response to a web services call, wherein the web services call references a namespace corresponding to a user of the data storage web service and a key corresponding to the data object, and wherein the stored data object is addressable using the namespace and the key;
computer code for causing the data storage web service to store one or more replicas of the data object in one or more of a plurality of distributed storage nodes; and
computer code for storing information characteristic of an actual usage of the data storage web service, wherein the actual usage is based, at least in part, upon the web services call.
10. The computer program product of claim 9 , wherein the web services calls comprise Representational State Transfer (REST)-style requests.
11. The computer program product of claim 9 , wherein the web services calls are implemented according to an application programing interface (API), and wherein an operation metrics service stores the information characteristic of the actual usage of the data storage web service.
12. The computer program product of claim 9 , wherein the data object is stored in a bucket.
13. The computer program product of claim 9 , wherein an operation metrics service stores information characteristic of a size of the data object or a storage utilization of the data storage web service.
14. The computer program product of claim 9 , wherein the data object is stored according to an encoding scheme.
15. The computer program product of claim 14 , wherein the encoding scheme is a parity encoding scheme.
16. The computer program product of claim 9 , wherein a keymap stores a relationship between the key and locators corresponding to locations of the data object and the one or more replicas of the data object.
17. A system for providing arbitrary amounts of reliable storage accessible from any location, the system comprising:
means for receiving web services calls from at least one client;
means for storing a data object in response to a web services call, wherein the web services call references a namespace corresponding to a user of the data storage web service and a key corresponding to the data object, and wherein the stored data object is addressable using the namespace and the key;
means for storing one or more replicas of the data object in one or more of a plurality of distributed storage nodes; and
means for storing information characteristic of an actual usage of the data storage web service, wherein the actual usage is based, at least in part, upon the web services call.
18. The system of claim 17 , wherein the web services calls comprise Representational State Transfer (REST)-style requests.
19. The system of claim 17 , wherein the web services calls are implemented according to an application programing interface (API), and wherein an operation metrics service stores the information characteristic of the actual usage of the data storage web service.
20. The system of claim 17 , wherein the data object is stored in a bucket.