Resolving failed or hanging mount points in a clustered storage solution for containers
A clustered storage system may include potentially many different nodes. A node may mount a virtual storage volume for the use of a container application at the node. The node may receive a request from a different node and respond by indicating whether the virtual storage volume is in active use. In this way, the clustered storage system may safely but forcibly unmount a virtual storage volume having a failed or hanging mount point so that the volume may be mounted on a different node.
1. A method comprising:
receiving a virtual storage volume activity request message at a first compute node from a second compute node via a communications interface, the virtual storage volume activity request message identifying a virtual storage volume mounted at the first compute node, the virtual storage volume configured as a data storage service within a containerized application system;
determining, at the first compute node and in response to the virtual storage volume activity request message, whether the virtual storage volume is active on the first compute node; and
transmitting a virtual storage volume activity response message from the first compute node to the second compute node via the communications interface, the virtual storage volume activity response message indicating whether the virtual storage volume is active on the first compute node.
2. The method of claim 1 , wherein the containerized application system is configured to run a plurality of distributed application instances on the first compute node and the second compute node.
3. The method of claim 2 , wherein the plurality of distributed application instances includes a storage driver that receives the virtual storage volume activity request message at the first compute node and transmits the virtual storage volume activity response message from the first compute node to the second compute node.
4. The method of claim 1 , wherein the determination is made based on communications between a storage driver on the first compute node and a kernel module associated with an operating system on the first compute node.
5. The method of claim 1 , wherein:
the virtual storage volume activity request message is sent to the first compute node in response to a mount request to mount the virtual storage volume at a remote computing device; and
the virtual storage volume is in a non-responsive state in which the virtual storage volume is not responding to storage requests.
6. The method of claim 5 , wherein:
the virtual storage volume is not active on the first compute node; and
the virtual storage volume activity response message indicates that the virtual storage volume may be safely mounted at the remote computing device.
7. The method of claim 1 , wherein the determination comprises determining that the virtual storage volume is not active on the first compute node based on a lack of association of the virtual storage volume with a block device reference on the first compute node.
8. A computing device comprising:
a communications interface operable to receive, from a remote computing node, a virtual storage volume activity request message, the virtual storage volume activity request message identifying a virtual storage volume mounted at the computing device, the virtual storage volume configured as a data storage service within a containerized application system; and
a processor communicatively coupled to the communications interface and operable to:
determine, in response to the virtual storage volume activity request message, whether the virtual storage volume is active on the computing device; and
instruct the communications interface to transmit a virtual storage volume activity response message to the remote computing node, the virtual storage volume activity response message indicating whether the virtual storage volume is active on the computing device.
9. The computing device of claim 8 , wherein the containerized application system is configured to run a plurality of distributed application instances on the computing device and the remote computing node.
10. The computing device of claim 9 , wherein the plurality of distributed application instances includes a storage driver that receives the virtual storage volume activity request message via the communications interface and transmits the virtual storage volume activity response message to the remote computing node via the communications interface.
11. The computing device of claim 10 , wherein the determination is made based on communications between the storage driver and a kernel module associated with an operating system on the computing device.
12. The computing device of claim 8 , wherein the virtual storage volume activity request message is sent from the remote computing node to the computing device in response to a mount request to mount the virtual storage volume at the remote computing node.
13. The computing device of claim 12 , wherein the virtual storage volume is in a non-responsive state in which the virtual storage volume is not responding to storage requests.
14. The computing device of claim 13 , wherein:
the virtual storage volume is not active on the computing device; and
the virtual storage volume activity response message indicates that the virtual storage volume may be safely mounted at the remote computing node.
15. The computing device of claim 8 , wherein the determination comprises determining that the virtual storage volume is not active on the computing device based on a lack of association of the virtual storage volume with a block device reference on the computing device.
16. One or more non-transitory machine-readable media having instructions stored thereon for directing one or more processors to perform a method, the method comprising:
receiving a virtual storage volume activity request message at a first compute node from a second compute node via a communications interface, the virtual storage volume activity request message identifying a virtual storage volume mounted at the first compute node, the virtual storage volume configured as a data storage service within a containerized application system;
determining, at the first compute node and in response to the virtual storage volume activity request message, whether the virtual storage volume is active on the first compute node; and
transmitting a virtual storage volume activity response message from the first compute node to the second compute node via the communications interface, the virtual storage volume activity response message indicating whether the virtual storage volume is active on the first compute node.
17. The one or more non-transitory machine-readable media of claim 16 , wherein the containerized application system is configured to run a plurality of distributed application instances on the first compute node and the second compute node.
18. The one or more non-transitory machine-readable media of claim 16 , wherein the determination is made based on communications between a storage driver on the first compute node and a kernel module associated with an operating system on the first compute node.
19. The one or more non-transitory machine-readable media of claim 16 , wherein the virtual storage volume is in a non-responsive state in which the virtual storage volume is not responding to storage requests.
20. The one or more non-transitory machine-readable media of claim 19 , wherein:
the virtual storage volume is not active on the first compute node; and
the virtual storage volume activity response message indicates that the virtual storage volume may be safely mounted at the second compute node.