Monitoring containers in a distributed computing system
A new snapshot of a storage volume is created by instructing computing nodes to suppress write requests. Once pending write requests from the computing nodes are completed, storage nodes create a new snapshot for the storage volume by allocating a new segment to the new snapshot and finalizes and performs garbage collection with respect to segments allocated to the previous snapshot. An orchestration layer implements a multi-role application that is provisioned with virtualized storage and computation resources. A snapshot of the application may be created and used to rollback or clone the application. Containers may be loaded with data structures for encrypted and/or authenticated communication. Failed storage volumes may be detected and repaired. Repopulating repaired volumes may be managed by an application or a storage manager implementing a virtualized storage system. An autopilot invokes repair of failed containers and storage nodes in response to detected failures.
1. A method comprising:
deploying, by a first computing device, a bundled application on a plurality of computing devices of a distributed computing system, the bundled application including a plurality of containers executing on the plurality of computing devices, each container of the plurality of containers executing an application instance forming part of the bundled application;
monitoring, by the first computing device, a state of each container of the plurality of containers; and
invoking at least one of restarting, redeploying, and moving containers of the plurality of containers, by the first computing device, effective to maintain a state of the bundled application in response to any faults indicated by the monitoring;
(a) detecting, by the first computing device, failure of a first container of the plurality of containers executing on a second computing device of the plurality of computing devices;
(b) in response to (a), invoking restarting of the first container on the second computing device;
(c) determining that (b) did not resolve the failure of the first container; and
(d) in response to (c), invoking redeploying of the first container on the second computing device.
2. The method of claim 1 , further comprising:
(e) determining that (d) did not resolve failure of the first container;
(f) in response to (e), invoking deployment of a second container on a third computing device of the plurality of computing devices and restoring a state of the first container in the second container.
3. The method of claim 1 , further comprising:
deploying a plurality of first containers of the plurality of containers on a second computing device of the plurality of computing devices;
associating each container of the plurality of first containers with a port on the second computing device; and
periodically querying, by an agent, the port associated with each container of the plurality of first containers to determine a state of the each container, the agent being software executing on one of the second computing device and another computing device.
4. The method of claim 1 , wherein a plurality of storage volumes are mounted to the plurality of containers.
5. The method of claim 4 , further comprising:
monitoring, by the first computing device, a state of the plurality of storage volumes;
in response to failure of a first storage volume of the plurality of storage volumes:
invoking re-provisioning and remounting of a new storage volume to replace the first storage volume; and
instructing the bundled application to re-sync the new storage volume from one or more replicas of the first storage volume.
6. The method of claim 4 , wherein the plurality of storage volumes each include one or more replicas, the method further comprising:
in response to failure of a first storage volume of the plurality of storage volumes:
invoking re-provisioning and remounting of a new storage volume to replace the first storage volume; and
restoring the new storage volume to include contents of the first storage volume using the one or more replicas of the first storage volume.
7. A method comprising:
deploying, by a first computing device, a bundled application on a plurality of computing devices of a distributed computing system, the bundled application including a plurality of containers executing on the plurality of computing devices, each container of the plurality of containers executing an application instance forming part of the bundled application;
monitoring, by the first computing device, a state of each container of the plurality of containers;
invoking at least one of restarting, redeploying, and moving containers of the plurality of containers, by the first computing device, effective to maintain a state of the bundled application in response to any faults indicated by the monitoring;
deploying a plurality of first containers of the plurality of containers on a second computing device of the plurality of computing devices;
associating each container of the plurality of first containers with a port on the second computing device;
periodically querying, by an agent, the port associated with each container of the plurality of first containers to determine a state of the each container, the agent being software executing on one of the second computing device and another computing device;
initiating a parent process on the second computing device;
starting, by the parent process a container daemon;
initiating, by the container daemon the plurality of first containers; and
implementing a socket on the second computing device by the container daemon;
wherein associating each container of the plurality of first containers with the port on the second computing device comprises associating, by the container daemon, each container of the plurality of first containers with a port of the socket.
8. The method of claim 7 , further comprising:
obtaining, by the agent, a mapping of the port of the socket with the plurality of first containers from the container daemon; and
querying, by the agent, the port associated with each container of the plurality of first containers according to the mapping.
9. A system comprising:
a first computing device comprising one or more processing devices and one or more memory devices operably coupled to the one or more processing devices, the one or more memory devices storing executable code effective to cause the one or more processing devices to:
deploy a bundled application on a plurality of computing devices of a distributed computing system, the bundled application including a plurality of containers executing on the plurality of computing devices, each container of the plurality of containers executing an application instance forming part of the bundled application;
monitor a state of each container of the plurality of containers; and
invoke at least one of restarting, redeploying, and moving containers of the plurality of containers effective to maintain a state of the bundled application in response to any faults indicated by the monitoring;
wherein the executable code is further effective to cause the one or more processing devices to:
(a) detect failure of a first container of the plurality of containers executing on a second computing device of the plurality of computing devices;
(b) in response to (a), invoke restarting of the first container on the second computing device;
(c) determine that (b) did not resolve the failure of the first container; and
(d) in response to (c), invoke redeploying of the first container on the second computing device.
10. The system of claim 9 , wherein the executable code is further effective to cause the one or more processing devices to:
(e) determine that (d) did not resolve failure of the first container;
(f) in response to (e), invoke deployment of a second container on a third computing device of the plurality of computing devices and restoring a state of the first container in the second container.
11. The system of claim 9 , wherein the executable code is further effective to cause the one or more processing devices to:
deploy a plurality of first containers of the plurality of containers on a second computing device of the plurality of computing devices;
associate each container of the plurality of first containers with a port on the second computing device; and
invoke periodically querying, by an agent, the port associated with each container of the plurality of first containers to determine a state of the each container, the agent being software executing on one of the second computing device and another computing device.
12. The system of claim 9 , further comprising a plurality of storage volumes mounted to the plurality of containers.
13. The system of claim 12 , wherein the executable code is further effective to cause the one or more processing devices to:
monitor a state of the plurality of storage volumes;
in response to failure of a first storage volume of the plurality of storage volumes:
invoke re-provisioning and remounting of a new storage volume to replace the first storage volume; and
instruct the bundled application to re-sync the new storage volume from one or more replicas of the first storage volume.
14. The system of claim 12 , wherein the plurality of storage volumes each include one or more replicas;
wherein the executable code is further effective to cause the one or more processing devices to:
in response to failure of a first storage volume of the plurality of storage volumes:
invoke re-provisioning and remounting of a new storage volume to replace the first storage volume; and
restore the new storage volume to include contents of the first storage volume using the one or more replicas of the first storage volume.
15. A system comprising:
a first computing device comprising one or more processing devices and one or more memory devices operably coupled to the one or more processing devices, the one or more memory devices storing executable code effective to cause the one or more processing devices to:
deploy a bundled application on a plurality of computing devices of a distributed computing system, the bundled application including a plurality of containers executing on the plurality of computing devices, each container of the plurality of containers executing an application instance forming part of the bundled application;
monitor a state of each container of the plurality of containers; and
invoke at least one of restarting, redeploying, and moving containers of the plurality of containers effective to maintain a state of the bundled application in response to any faults indicated by the monitoring;
deploy a plurality of first containers of the plurality of containers on a second computing device of the plurality of computing devices;
associate each container of the plurality of first containers with a port on the second computing device; and
invoke periodically querying, by an agent, the port associated with each container of the plurality of first containers to determine a state of the each container, the agent being software executing on one of the second computing device and another computing device; and
the second computing device, the second computing device further programmed to:
initiate a parent process on the second computing device;
start, by the parent process a container daemon;
initiate, by the container daemon the plurality of first containers; and
implement a socket on the second computing device by the container daemon;
associate each container of the plurality of first containers with the port on the second computing device by associating, by the container daemon, each container of the plurality of first containers with a port of the socket.
16. The system of claim 15 , wherein the agent is further programmed to:
obtain a mapping of the port of the socket with the plurality of first containers from the container daemon; and
query the port associated with each container of the plurality of first containers according to the mapping.