Fault tolerance for a distributed computing system

1. A method comprising:

detecting a failure of a container, of a set of containers, in a controller node, the container executing a service being performed and isolated from at least one other service being performed in at least one other container on the controller node;

terminating, by the controller node, the container executing the service;

determining, by the controller node, a particular known state for the service, wherein the particular known state is known to be operational without including one or more changes that caused the failure, and wherein the service saves the changes to the particular known state during operation separately from the particular known state;

restarting, by the controller node, the service in a new container that replaces the terminated container, wherein the restarted service starts from the particular known state without using the changes;

wherein an orchestration service, configured to manage the set of containers, detects the failure;

wherein the orchestration service detects the failure via monitoring a communication service in which a status of the service is input; and

wherein the method is performed by at least one device including a hardware processor.

2. The method of claim 1 , wherein the particular known state comprises a full image of the service required to start the service from the new container.

3. The method of claim 1 , wherein the failure of the container is not resolved.

4. The method of claim 1 , wherein detecting comprises:

determining from a communication service that a presence of the service has been lost; and

determining the failure based on the presence being lost.

5. The method of claim 1 , wherein restarting comprises:

restarting an orchestration service instance that manages the service in the new container.

6. A method comprising:

detecting a failure of a container, of a set of containers, in a controller node, the container executing a service being performed and isolated from at least one other service being performed in at least one other container on the controller node;

terminating, by the controller node, the container executing the service;

determining, by the controller node, a particular known state for the service, wherein the particular known state is known to be operational without including one or more changes that caused the failure, and wherein the service saves the changes to the particular known state during operation separately from the particular known state;

restarting, by the controller node, the service in a new container that replaces the terminated container, wherein the restarted service starts from the particular known state without using the changes;

upon restarting with the particular known state, determining, by the service, configuration data or state data for the service from storage; and

wherein the method is performed by at least one device including a hardware processor.

7. A method comprising:

detecting a failure of a container, of a set of containers, in a controller node, the container executing a service being performed and isolated from at least one other service being performed in at least one other container on the controller node;

terminating, by the controller node, the container executing the service;

determining, by the controller node, a particular known state for the service, wherein the particular known state is known to be operational without including one or more changes that caused the failure, and wherein the service saves the changes to the particular known state during operation separately from the particular known state;

restarting, by the controller node, the service in a new container that replaces the terminated container, wherein the restarted service starts from the particular known state without using the changes;

wherein:

the particular known state is included in a file system,

the service with the failure records differences to the file system without changing the file system,

the changes are not used in restarting the service in the new container, and

the method is performed by at least one device including a hardware processor.

8. A system comprising:

at least one device including a hardware processor;

the system being configured to perform operations comprising:

detecting a failure of a container, in a set of containers, in a controller node, the container executing a service being performed and isolated from at least one other service being performed in at least one other container on the controller node;

terminating, by the controller node, the container executing the service;

determining, by the controller node, a particular known state for the service, wherein the particular known state is known to be operational without including one or more changes that caused the failure, and wherein the service saves the changes to the particular known state during operation separately from the particular known state;

restarting, by the controller node, the service in a new container that replaces the terminated container, wherein the restarted service starts from the known state without using the changes; and

wherein an orchestration service, configured to manage the set of containers, detects the failure;

wherein the orchestration service detects the failure via monitoring a communication service in which a status of the service is input.

9. The system of claim 8 , wherein the particular known state comprises a full image of the service required to start the service from the new container.

10. The system of claim 8 , wherein the failure of the container is not resolved.

11. The system of claim 8 , wherein detecting comprises:

determining from a communication service that a presence of the service has been lost; and

determining the failure based on the presence being lost.

12. The system of claim 8 , wherein restarting comprises:

restarting an orchestration service instance that manages the service in the new container.

13. A system comprising:

at least one device including a hardware processor;

The system being configured to perform operations comprising:

detecting a failure of a container, in a set of containers, in a controller node, the container executing a service being performed and isolated from at least one other service being performed in at least one other container on the controller node;

terminating, by the controller node, the container executing the service;

determining, by the controller node, a particular known state for the service, wherein the particular known state is known to be operational without including one or more changes that caused the failure, and wherein the service saves the changes to the particular known state during operation separately from the particular known state;

restarting, by the controller node, the service in a new container that replaces the terminated container, wherein the restarted service starts from the particular known state without using the changes; and

upon restarting with the particular known state, determining, by the service, configuration data or state data for the service from storage.

14. A system comprising:

at least one device including a hardware processor;

the system being configured to perform operations comprising:

detecting a failure of a container, in a set of containers, in a controller node, the container executing a service being performed and isolated from at least one other service being performed in at least one other container on the controller node;

terminating, by the controller node, the container executing the service;

determining, by the controller node, a particular known state for the service, wherein the particular known state is known to be operational without including one or more changes that caused the failure, and wherein the service saves the changes to the particular known state during operation separately from the particular known state;

restarting, by the controller node, the service in a new container that replaces the terminated container, wherein the restarted service starts from the particular known state without using the changes;

wherein:

the particular known state is included in a file system,

the service with the failure records differences to the file system without changing the file system, and

the changes are not used in restarting the service in the new container.

15. A non-transitory computer-readable storage medium containing instructions, that when executed, control a computer system to be configured for:

detecting a failure of a container, in a set of containers, in a controller node, the container executing a service being performed and isolated from at least one other service being performed in at least one other container on the controller node;

terminating, by the controller node, the container executing the service;

determining, by the controller node, a particular known state for the service, wherein the particular known state is known to be operational without including one or more changes that caused the failure, and wherein the service saves changes to the particular known state during operation separately from the particular known state;

restarting, by the controller node, the service in a new container that replaces the terminated container, wherein the restarted service starts from the particular known state without using the changes; and

wherein an orchestration service, configured to manage the set of containers, detects the failure;

wherein the orchestration service detects the failure via monitoring a communication service in which a status of the service is input.

16. The non-transitory computer-readable storage medium of claim 15 , wherein the particular known state comprises a full image of the service required to start the service from the new container.