Efficient flash memory-based object store
Approaches for an object store implemented, at least in part, on one or more solid state devices. The object store may store objects on a plurality of solid state devices. The object store may include a transaction model means for ensuring that the object store performs transactions in compliance with atomicity, concurrency, isolation, and durability (ACID) properties. The object store may include means for providing parallel flushing in a write cache maintained on each of the solid state devices. The object store may include means for maintaining one or more double-write buffers, for the object store, at a location other than the solid state devices. The object store may optionally comprise means for maintaining one or more circular transaction logs, for the object store, at a location other than the solid state devices. The object store may operate to minimize write operations performed on the solid state devices.
1. A method for durably storing data blocks maintained in a volatile cache on a solid state device, comprising:
instantiating a plurality of threads that are each configured to perform work according to one of a plurality of stages,
wherein the plurality of stages include:
a first stage in which a thread identifies, in the volatile cache of a server that issues write requests to an object store maintained on one or more solid state devices, a first set of data blocks to write to a double-write buffer cache in non-volatile dynamic random access memory (DRAM),
a second stage in which a thread writes a second set of data blocks from the volatile cache to the double-write buffer,
a third stage in which a thread writes data from the double-write buffer to the one or more solid state devices, and
wherein the plurality of threads include at least two threads that are both operating in different stages of the plurality of stages.
2. The method of claim 1 , wherein the plurality of threads includes at least three threads that are each operating in a different stage of the plurality of stages.
3. The method of claim 1 , wherein the plurality of threads include two or more threads that are operating in the same stage of the plurality of stages.
4. The method of claim 1 , wherein the non-volatile dynamic random access memory (DRAM) is maintained in a controller of a hard-disk drive (HDD).
5. A distributed object store, comprising:
a plurality of solid state devices;
one or more processors on one or more machines; and
a machine readable medium storing one or more sequences of instructions, which when executed by the one or more processors, cause:
maintaining the distributed object store on the plurality of solid state devices, wherein the distributed object store stores objects using a transaction model that ensures compliance with atomicity, concurrency, isolation, and durability (ACID) properties;
the distributed object store providing parallel flushing in a write cache maintained on each of the plurality of solid state devices; and
the distributed object store maintaining one or more double-write buffers, for the distributed object store, at a location other than the plurality of solid state devices.
6. The distributed object store of claim 5 , wherein execution of the one or more sequences of instructions by the one or more processors further cause:
the distributed object store maintaining one or more circular transaction logs, for the distributed object store, at a location other than the plurality of solid state devices.
7. The distributed object store of claim 5 , wherein execution of the one or more sequences of instructions by the one or more processors further cause:
the distributed object store supporting an interface which allows an administrator to reduce a block size for the distributed object store for purposes of minimizing wear on each of the plurality of solid state devices.
8. The distributed object store of claim 5 , wherein the distributed object store providing parallel flushing comprises identifying, in parallel, dirty data blocks stored within the write cache, to be replaced.
9. The distributed object store of claim 5 , wherein the distributed object store is a MySQL database management system.
10. The distributed object store of claim 5 , wherein the distributed object store is a memcached caching system or a non-relational data store.
11. A non-transitory machine readable storage medium storing one or more sequences of instructions for durably storing data blocks maintained in a volatile cache on a solid state device, wherein execution of the one or more sequences of instructions cause:
instantiating a plurality of threads that are each configured to perform work according to one of a plurality of stages,
wherein the plurality of stages include:
a first stage in which a thread identifies, in the volatile cache of a server that issues write requests to an object store maintained on one or more solid state devices, a first set of data blocks to write to a double-write buffer cache in non-volatile dynamic random access memory (DRAM),
a second stage in which a thread writes a second set of data blocks from the volatile cache to the double-write buffer,
a third stage in which a thread writes data from the double-write buffer to the one or more solid state devices, and
wherein the plurality of threads include at least two threads that are both operating in different stages of the plurality of stages.
12. The machine readable storage medium of claim 11 , wherein the plurality of threads includes at least three threads that are each operating in a different stage of the plurality of stages.
13. The machine readable storage medium of claim 11 , wherein the plurality of threads include two or more threads that are operating in the same stage of the plurality of stages.
14. The machine readable storage medium of claim 11 , wherein the non-volatile dynamic random access memory (DRAM) is maintained in a controller of a hard-disk drive (HDD).
15. The machine readable storage medium of claim 11 , wherein instantiating the plurality of threads includes at least instantiating a first thread, and instantiating a second thread.
16. The method of claim 1 , wherein instantiating the plurality of threads includes at least instantiating a first thread, and instantiating a second thread.
17. A non-transitory computer readable storage medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors cause:
maintaining a distributed object store on a plurality of solid state devices, wherein the distributed object store stores objects using a transaction model that ensures compliance with atomicity, concurrency, isolation, and durability (ACID) properties;
the distributed object store providing parallel flushing in a write cache maintained on each of the plurality of solid state devices; and
the distributed object store maintaining one or more double-write buffers, for the distributed object store, at a location other than the plurality of solid state devices.
18. The non-transitory computer readable storage medium of claim 17 , wherein execution of the one or more sequences of instructions by the one or more processors further cause:
the distributed object store maintaining one or more circular transaction logs, for the distributed object store, at a location other than the plurality of solid state devices.
19. The non-transitory computer readable storage medium of claim 17 , wherein execution of the one or more sequences of instructions by the one or more processors further cause:
the distributed object store supporting an interface which allows an administrator to reduce a block size for the distributed object store for purposes of minimizing wear on each of the plurality of solid state devices.
20. The non-transitory computer readable storage medium of claim 17 , wherein the distributed object store providing parallel flushing comprises identifying, in parallel, dirty data blocks stored within the write cache, to be replaced.
21. The non-transitory computer readable storage medium of claim 17 , wherein the distributed object store is a MySQL database management system.
22. The non-transitory computer readable storage medium of claim 17 , wherein the distributed object store is a memcached caching system or a non-relational data store.