Resolving failed or hanging mount points in a clustered storage solution for containers
A clustered storage system may include potentially many different nodes, each including a storage driver and a kernel module. A node may mount a virtual storage volume for the use of a container application at the node. The storage driver and kernel module may receive a request from a storage driver at a different node and respond by indicating whether the virtual storage volume is in active use. In this way, the clustered storage system may safely but forcibly unmount a virtual storage volume having a failed or hanging mount point so that the volume may be mounted on a different node.
1. A method to be performed by one or more processors, the method comprising:
receiving a virtual storage volume activity request message at a storage driver on a first compute node from a second compute node via a communications interface, the virtual storage volume activity request message identifying a virtual storage volume mounted at the first compute node;
determining, at a processor on the first compute node, whether the virtual storage volume is associated with one or more block device references in an operating system running at the first compute node; and
transmitting a virtual storage volume activity response message from the storage driver to the second compute node via the communications interface, the virtual storage volume activity response message indicating whether the virtual storage volume is associated with one or more block device references in the operating system, wherein the virtual storage volume is configured as a data storage service within a containerized application system.
2. The method recited in claim 1 , wherein the containerized application system configured to run a plurality of distributed application instances within the operating system, each application instance corresponding with a designated application.
3. The method recited in claim 2 , wherein the plurality of distributed applications includes the storage driver.
4. The method recited in claim 1 , wherein the determination is made based on communications between the storage driver and a kernel module associated with the operating system.
5. The method recited in claim 1 , wherein the request message is sent in response to a mount request to mount the virtual storage volume at the remote computing device, and wherein the virtual storage volume is in a non-responsive state in which the virtual storage volume is not responding to storage requests.
6. The method recited in claim 5 , wherein the virtual storage volume is not associated with one or more block device references, and wherein a selected one of the one or more block device references comprises a respective identifier associated with a respective open file accessible via the virtual storage volume.
7. The method recited in claim 6 , wherein the virtual storage volume activity response message indicates that the virtual storage volume may be safely mounted at the remote computing device.
8. A computing device comprising:
a communications interface operable to receive from a remote computing node a virtual storage volume activity request message addressed to a storage driver implemented at the computing device, the virtual storage volume activity request message identifying a virtual storage volume mounted at the computing device;
a processor operable to determine whether the virtual storage volume is associated with one or more block device references in an operating system running at the computing device; and
memory operable to store one or more block device references, wherein the processor is further operable to instruct the communications interface to transmit a virtual storage volume activity response message from the storage driver to the remote computing node, the virtual storage volume activity response message indicating whether the virtual storage volume is associated with one or more block device references in the operating system, wherein the virtual storage volume is configured as a data storage service within a containerized application system.
9. The computing device recited in claim 8 , wherein the containerized application system configured to run a plurality of distributed application instances within the operating system, each application instance corresponding with a designated application.
10. The computing device recited in claim 9 , wherein the plurality of distributed applications includes the storage driver.
11. The computing device recited in claim 8 , wherein the determination is made based on communications between the storage driver and a kernel module associated with the operating system.
12. The computing device recited in claim 8 , wherein the request message is sent in response to a mount request to mount the virtual storage volume at the remote computing device.
13. The computing device recited in claim 12 , wherein the virtual storage volume is in a non-responsive state in which the virtual storage volume is not responding to storage requests.
14. The computing device recited in claim 13 , wherein the virtual storage volume is not associated with one or more block device references.
15. The computing device recited in claim 14 , wherein the virtual storage volume activity response message indicates that the virtual storage volume may be safely mounted at the remote computing device.
16. The computing device recited in claim 8 , wherein a selected one of the one or more block device references comprises a respective identifier associated with a respective open file accessible via the virtual storage volume.
17. One or more machine readable media having instructions stored thereon for performing a method, the method comprising:
receiving a virtual storage volume activity request message at a storage driver on a first compute node from a second compute node via a communications interface, the virtual storage volume activity request message identifying a virtual storage volume mounted at the first compute node;
determining, at a processor on the first compute node, whether the virtual storage volume is associated with one or more block device references in an operating system running at the first compute node; and
transmitting a virtual storage volume activity response message from the storage driver to the second compute node via the communications interface, the virtual storage volume activity response message indicating whether the virtual storage volume is associated with one or more block device references in the operating system, wherein the virtual storage volume is configured as a data storage service within a containerized application system.
18. The one or more machine readable media recited in claim 17 , wherein the containerized application system configured to run a plurality of distributed application instances within the operating system, each application instance corresponding with a designated application.
19. The one or more machine readable media recited in claim 17 , wherein the determination is made based on communications between the storage driver and a kernel module associated with the operating system.
20. The one or more machine readable media recited in claim 17 , wherein the virtual storage volume is in a non-responsive state in which the virtual storage volume is not responding to storage requests.