Lifecycle transitions in log-coordinated data stores
Based at least in part on an optimization goal determined for a storage group comprising a plurality of data stores for which writes requests are coordinated by a log-based transaction manager, one or more lifecycle transitions are identified for data objects stored in a first data store. In accordance with a first lifecycle transition, a data object is to be transferred from the first data store to a second data store based at least in part on a duration for which the data object has been resident at the first data store. A transition agent is assigned by a configuration manager to implement the first lifecycle transition.
1. A system, comprising:
one or more processors to perform instructions of a configuration manager of a heterogeneous storage service to:
determine one or more lifecycle transitions to be performed for at least a portion of a data set to be stored at a first data store of a plurality of data stores of a log-coordinated storage group established on behalf of a client, wherein, in accordance with a first lifecycle transition of the one or more lifecycle transitions, a data object that has been accepted for inclusion in the first data store is to be transferred to a second data store of the plurality of data stores based at least in part on one or more time-based transition criteria, wherein at least one of the time-based transition criteria comprises a delay since the object was accepted for inclusion in the first data store, and wherein the second data store is designated as a destination of the first lifecycle transition based at least in part on a difference in service capability of the first data store and the second data store; and
assign a particular transition agent of one or more transition agents to implement the first lifecycle transition; and
one or more processors to perform instructions of the particular transition agent, responsive to assignment of the particular transition agent to implement the first lifecycle transition, to:
responsive to detection that a particular data object of the first data store meets a particular time-based transfer criterion of the one or more time-based transition criteria, transfer the particular data object to the second data store in accordance with the first lifecycle transition to provide a change of service capability for the particular data object; and
store a transition record indicating that the particular data object has been transferred.
2. The system as recited in claim 1 , wherein the configuration manager is configured to:
receive a lifecycle flow request from the client via a programmatic interface, wherein the lifecycle flow request indicates the first data store, the second data store, and the one or more time-based transfer criterion.
3. The system as recited in claim 2 , wherein the configuration manager is configured to:
receive an indication of a optimization goal of the client; and
identify the first lifecycle transition as a candidate lifecycle transition based at least in part on the optimization goal.
4. The system as recited in claim 1 , wherein the difference in service capability comprises a difference in one or more of: (a) a data model, (b) a data isolation level, (c) availability, (d) data durability, or (e) performance.
5. The system as recited in claim 1 , further comprising a transaction manager of the log-coordinated storage group, wherein the transaction manager is configured to:
receive a transaction request from the client, indicating (a) a particular write operation directed to the second data store and (b) a read descriptor indicating one or more reads on which the particular write depends;
determine, based at least in part on the read descriptor and at least in part on one or more write records stored in a persistent log, that the particular write is to be accepted for commit; and
initiate a propagation of the particular write to the second data store, without propagating the write to the first data store.
6. A method, comprising:
determining, at a configuration manager of a heterogeneous storage service, one or more lifecycle transitions to be implemented for at least a portion of a first data store of a plurality of data stores for which write requests are coordinated by a log-based transaction manager on behalf of a client, wherein, in accordance with a first lifecycle transition of the one or more lifecycle transitions, a data object is to be transferred to a second data store of the plurality of data stores based at least in part on a transfer criterion, wherein the transition criterion comprises a duration for which the data object has resided in the first data store, and wherein the second data store is selected as a destination of the first lifecycle transition based at least in part on a difference in a service capability between the first data store and the second data store;
assigning, by the configuration manager, a particular transition agent of one or more transition agents of the heterogeneous storage service to perform the first lifecycle transition; and
in response to detecting, by the particular transition agent and in accordance with the first lifecycle transition, that a particular data object meets the transfer criterion, transferring, by the particular transition agent, the particular data object to the second data store and storing, by the particular transition agent, a transition record indicating that the particular data object has been transferred in accordance with the first lifecycle transition to provide a transition of service capability on behalf of the client.
7. The method as recited in claim 6 , wherein the difference in service capability comprises a difference in one or more of: (a) a data model, (b) a data isolation level, (c) availability, (d) data durability, or (e) performance.
8. The method as recited in claim 6 , wherein in accordance with the first lifecycle transition, the data object is to be transferred to the second data store based at least in part on a difference in a first pricing policy associated with the first data store, and a second pricing policy associated with the second data store.
9. The method as recited in claim 6 , wherein the first data store includes a plurality of storage devices utilizing a first medium type of a set of medium types comprising (a) main memory-resident storage, (b) solid-state-drive storage, (c) rotating-disk-based storage and (d) tape-based storage, and wherein the second data store includes at least one storage device utilizing the first medium type.
10. The method as recited in claim 6 , wherein the particular data object is stored at a first storage device of the first data store at a first data center of a provider network, and wherein the particular data object is transferred by the particular transition agent to a different storage device of the second data store at a different data center.
11. The method as recited in claim 6 , further comprising:
receiving, by the configuration manager, a lifecycle flow request from the client via a programmatic interface, wherein the lifecycle flow request indicates the first data store, the second data store, and the transfer criterion.
12. The method as recited in claim 11 , further comprising:
receiving, by the configuration manager, an indication of a optimization goal of the client; and
identifying, by the configuration manager, the first lifecycle transition as a candidate lifecycle transition based at least in part on the optimization goal.
13. The method as recited in claim 6 , further comprising:
generating, by the particular transition agent to transfer the particular data object to the second data store, a representation of the particular data object based at least in part on an examination of a commit record associated with the particular data object, wherein the commit record is stored in a persistent write log of the log-based transaction manager, wherein the persistent write log is implemented as a replication graph.
14. The method as recited in claim 6 , wherein the first data store comprises one of: a non-relational database system, a relational database system, a storage service that implements a web services interface allowing access to unstructured data objects, an in-memory database, or an instance of a distributed cache.
15. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors implements a configuration manager of a storage service, wherein the configuration manager is configured to:
determine an optimization goal for a storage group established on behalf of a client, wherein the storage group comprises a plurality of data stores for which write requests are coordinated by a log-based transaction manager;
identify, in accordance with the optimization goal, one or more data lifecycle transitions to be implemented for at least a portion of a first data store of the plurality of data stores, wherein, in accordance with a first data lifecycle transition of the one or more data lifecycle transitions, a data object is to be transferred to a second data store of the plurality of data stores based at least in part on a duration for which the data object has resided in the first data store; and
assign, by the configuration manager and responsive to the identification, at least a first transition agent to implement the first lifecycle transition.
16. The non-transitory computer-accessible storage medium claim as recited in claim 15 , wherein the configuration manager is configured to:
designate the second data store as a destination of the first lifecycle transition based at least in part on a difference in service capability between the first data store and the second data store.
17. The non-transitory computer-accessible storage medium claim as recited in claim 16 , wherein the difference in service capability comprises a difference in one or more of: (a) a data model, (b) a data isolation level, (c) availability, (d) data durability, or (e) performance.
18. The non-transitory computer-accessible storage medium claim as recited in claim 15 , wherein the optimization goal comprises a budget goal.
19. The non-transitory computer-accessible storage medium claim as recited in claim 15 , wherein the configuration manager is configured to:
determine, based at least in part on monitoring one or more metrics associated with operations performed by the first transition agent, that an additional transition agent is to be configured; and
instantiate a second transition agent to implement the first lifecycle transition.
20. The non-transitory computer-accessible storage medium as recited in claim 15 , wherein the optimization goal is indicated in a request received via a programmatic interface of the storage service, wherein the programmatic interface comprises one or more of: (a) an application programming interface (API), (b) a web-based console, (c) a graphical user interface, (d) a menu of pre-defined service levels, or (e) a command-line tool.