Implementing a hybrid storage node in a distributed storage system
A new snapshot of a storage volume is created by instructing computing nodes to suppress write requests. Once pending write requests from the computing nodes are completed, storage nodes create a new snapshot for the storage volume by allocating a new segment to the new snapshot and finalizes and performs garbage collection with respect to segments allocated to the previous snapshot. The snapshots may be represented by a storage manager in a hierarchy. Deleted snapshots may be flagged as such in the hierarchy and deletion may be implemented only in memory on a storage node, which is then restored from the hierarchy in the event of a crash. A hybrid storage node may perform both computing and storage services. An IO module determines whether IOPs reference a locally-mounted storage device or a remote storage device. The IO module issues a library call for local IOPs using a shared memory in kernel space.
1. A method for processing input/output operations (IOPs) in a distributed storage system, the method comprising:
receiving, by an input/output (TO) module executing on a storage node comprising one or more processing devices and one or more storage devices, a first IOP from a first application;
determining, by the IO module, that (a) the first application is executing on the storage node; and
in response to determining (a), issuing, by the IO module, a first library function call to disk virtualization manager (DVM) executing on the storage node to execute the first IOP;
in response to the first library function call, performing, by the DVM:
executing the first IOP, the first IOP being a read IOP, to obtain payload data;
writing the payload data to a shared memory in kernel space of an operating system executing on the storage node, the DVM and IO module executing in a same process in user space of the operating system and both having access to the shared memory; and
issuing a second library function call to the IO module that does not include the payload data; and
in response to the second library function call, reading, by the IO module, the payload data from the shared memory and returning the payload data to the first application.
2. The method of claim 1 , further comprising:
returning, by the DVM, a result of the first IOP to the IO module; and
returning, by the TO module, the result to the first application.
3. The method of claim 2 , wherein the first IOP is a write IOP, the method further comprising:
writing, by the IO module, payload data from the first IOP to a shared memory in kernel space of an operating system executing on the storage node, the DVM and IO module executing in a user space of the operating system and both having access to the shared memory;
issuing, by the IO module, the first library function call to the DVM for the first IOP without including the payload data in the first library function call; and
in response to the first library function call, reading, by the DVM, the payload data from the shared memory and writing the payload data to a location referenced by the first IOP.
4. The method of claim 1 , further comprising:
receiving, by the IO module, a second IOP from a second application;
determining, by the IO module, that (b) the second application is not executing on the storage node; and
in response to determining (a), issuing a remote procedure call (RPC), by the IO module, to a DVM executing on a remote node corresponding to the second IOP.
5. The method of claim 1 , wherein the IO module and DVM execute within a same process on the storage node.
6. The method of claim 5 , wherein the same process is a daemon.
7. The method of claim 6 , wherein the daemon executes in a same user application space in an operation system of the storage node as the first application.
8. A system comprising one or more processing devices, one or more memory devices, and one or more storage devices, the one or more memory devices storing executable code effective to cause the one or more processing devices to:
receive, by an input/output (TO) module, a first input/output operation (TOP) from a first application;
determine, using the IO module, that (a) the first application is executing on the system; and
in response to determining (a), issue, by the IO module, a first library function call to disk virtualization manager (DVM) executing on the system to execute the first IOP;
in response to the first library function call:
execute, by the DVM, the first TOP, the first TOP being a read TOP, to obtain payload data;
write, by the DVM, the payload data to a shared memory in kernel space of an operating system executing on the storage node, the DVM and IO module executing in a user space of the operating system and both having access to the shared memory; and
issue, by the DVM, a second library function call to the IO module that does not include the payload data; and
in response to the second library function call, read, by the IO module, the payload data from the shared memory and return the payload data to the first application.
9. The system of claim 8 , wherein the executable code is further effective to cause the one or more processing devices to:
return, by the DVM, a result of the first TOP to the IO module; and
return, by the IO module, the result to the first application.
10. The system of claim 9 , wherein the first TOP is a write TOP, the executable code is further effective to cause the one or more processing devices to:
write, by the TO module, payload data from the first TOP to a shared memory in kernel space of an operating system executing on the storage node, the DVM and TO module executing in a user space of the operating system and both having access to the shared memory;
issue, by the TO module, the first library function call to the DVM for the first TOP without including the payload data in the first library function call; and
read, by the DVM, the payload data from the shared memory and write the payload data to a location referenced by the first TOP.
11. The system of claim 8 , the executable code is further effective to cause the one or more processing devices to:
receive, by the TO module, a second TOP from a second application;
determine, by the TO module, that (b) the second application is not executing on the storage node; and
in response to determining (a), issue, by the TO module, a remote procedure call (RPC) to a DVM executing on a remote node corresponding to the second TOP.
12. The system of claim 8 , wherein the TO module and DVM execute within a same process on the storage node.
13. The system of claim 12 , wherein the same process is a daemon.
14. The system of claim 13 , wherein the daemon executes in a same user application space in an operation system of the storage node as the first application.