Removable media based object store
Embodiments disclosed herein provide systems, methods, and computer-readable media to implement an object store with removable storage media. In a particular embodiment, a method provides identifying first data for storage on a first removable storage medium and designating at least a portion of the first data to a first data object. The method further provides determining a first location where to store the first data object in a first value store partition of the first removable storage medium and writing the first data object to the first location. Also, the method provides writing a first key that identifies the first data object and indicates the first location to a first key store partition of the first removable storage medium.
1. A method for implementing an object store using removable storage media, the method comprising:
receiving a request to retrieve first data;
determining a first data object in which at least a portion of the first data is stored:
determining a removable storage medium, of the removable storage media, on which the first data object is stored, the removable storage medium including a value store partition into which one or more data objects including at least the first data object are stored and a key store partition into which one or more keys identifying the one or more data objects are stored;
retrieving the first data object from the value store partition; and
retrieving the at least a portion of the first data from the first data object comprises:
receiving a notification from a data manager that the first data object has been placed in a cache; and retrieving the at least a portion of the first data from the cache.
2. The method of claim 1 , further comprising:
providing the at least a portion of the first data in response to the request.
3. The method of claim 1 , further comprising:
receiving a second request to retrieve second data;
determining the second data is stored in the first data object;
retrieving the second data from the first data object; and
providing the second data in response to the second request.
4. The method of claim 1 , wherein the at least a portion of the first data comprises a first portion of the first data, and the method further comprising:
determining a second data object in which a second portion of the first data is stored;
retrieving the second data object from the value store partition; and
retrieving the second portion of the first data from the second data object.
5. The method of claim 1 , further comprising:
before retrieving the first data object, determining that the first data object is not located in a cache.
6. The method of claim 1 , wherein determining the removable storage medium comprises:
retrieving metadata indicating the at least a portion of the first data is stored in the first data object and that the first data object is stored on the removable storage medium.
7. The method of claim 1 , further comprising: requesting bids for retrieving the first data object from two or more data managers; and selecting the data manager of the two or more data managers based on the bids.
8. The method of claim 7 , wherein retrieving the first data object comprises: notifying the data manager that the data manager had a winning bid, wherein the data manager handles placement of the first data object into the cache.
9. The method of claim 1 , wherein the removable storage medium comprises a magnetic tape.
10. A data management system to implement an object store using removable storage media, the data management system comprising: one or more computer-readable storage media; a processing system operatively coupled with the one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media that, when read and executed by the processing system, direct the processing system to:
receive a request to retrieve first data;
determine a first data object in which at least a portion of the first data is stored;
determine a removable storage medium, of the removable storage media, on which the first data object is stored, the removable storage medium including a value store partition into which one or more data objects including at least the first data object are stored and a key store partition into which one or more keys identifying the one or more data objects are stored;
retrieve the first data object from the value store partition; and
retrieve the at least a portion of the first data from the first data object comprises:
receive a notification from a data manager that the first data object has been placed in a cache; and retrieve the at least a portion of the first data from the cache.
11. The data management system of claim 10 , wherein the program instructions further direct the processing system to:
provide the at least a portion of the first data in response to the request.
12. The data management system of claim 10 , wherein the program instructions further direct the processing system to:
receive a second request to retrieve second data;
determine the second data is stored in the first data object;
retrieve the second data from the first data object; and
provide the second data in response to the second request.
13. The data management system of claim 10 , wherein the at least a portion of the first data comprises a first portion of the first data, and wherein the program instructions further direct the processing system to:
determine a second data object in which a second portion of the first data is stored;
retrieve the second data object from the value store partition; and
retrieve the second portion of the first data from the second data object.
14. The data management system of claim 10 , wherein the program instructions further direct the processing system to:
before retrieving the first data object, determine that the first data object is not located in a cache.
15. The data management system of claim 10 , wherein to determine the removable storage medium, the program instructions direct the processing system to:
retrieve metadata indicating the at least a portion of the first data is stored in the first data object and that the first data object is stored on the removable storage medium.
16. The data management system of claim 10 , wherein the program instructions further direct the processing system to: request bids for retrieving the first data object from the two or more data managers; and select the data manager of the two or more data managers based on the bids.
17. The data management system of claim 16 , wherein to retrieve the data object, the program instructions direct the processing system to: notify the data manager that the data manager had a winning bid, wherein the data manager handles placement of the data object into the cache.
18. One or more non-transitory computer-readable storage media having program instructions stored thereon to implement an object store using removable storage media, the program instructions, when read and executed by a processing system, direct the processing system to:
receive a request to retrieve first data;
determine a first data object in which at least a portion of the first data is stored:
determine a removable storage medium, of the removable storage media, on which the first data object is stored, the removable storage medium including a value store partition into which one or more data objects including at least the first data object are stored and a key store partition into which one or more keys identifying the one or more data objects are stored;
retrieve the first data object from the value store partition; and
retrieve the at least a portion of the first data from the first data object comprises:
receive a notification from a data manager that the first data object has been placed in a cache; and retrieve the at least a portion of the first data from the cache.